JP7231113B2 - Visualization control device, visualization system, visualization control method, and computer program - Google Patents

Description

The present invention relates to technology for notifying an operator of an aircraft such as a helicopter of the existence of flying objects flying around.

A wide variety of flying objects such as drones, UAVs (unmanned aerial vehicles), RPASs (remote piloted aircraft systems), UASs (unmanned aircraft systems), etc. A flying society is about to be realized. When a large number of flying objects fly in the same airspace in this way, there is concern that the danger of abnormal approach of flying objects and collision between flying objects increases.

Japanese Patent Publication No. 2015-533109

In Patent Document 1, when a bird strike, in which an object such as a bird collides with an aircraft, has occurred or is likely to occur, a video image obtained by a video camera mounted on the aircraft is used. A technique for reporting information related to is disclosed. Japanese Patent Application Laid-Open No. 2002-200003 discloses that a report reporting information related to a collision includes the collision position, size, and characteristics of an object on an aircraft.

The report in Patent Literature 1 merely reports the collision position, size, and characteristics of an object in the aircraft, and does not prompt the flight crew (pilot) of the aircraft to take action to avoid the collision.

A main object of the present invention is to provide a technology capable of notifying an aircraft operator of information prompting an action to avoid a collision with a flying object.

In order to achieve the above object, the visualization control device according to the present invention, as one form thereof,
The flying object is detected by using reference data representing the flying object to be detected from a photographed image of the surroundings of the aircraft taken by a photographing device mounted on the aircraft, and the flying object and the aircraft are detected. and a detection unit that detects the moving direction of the flying object with respect to the aircraft;
a determination unit that determines whether there is a risk of the flying object approaching or colliding with the aircraft, using the detected moving direction and distance of the flying object;
a display control unit that controls the operation of a display device that is mounted on the aircraft and that visualizes and warns the presence of the flying object determined to be at risk of approaching or colliding with the aircraft;
During the period in which the aircraft is on standby on the ground, the image captured by the image capturing device is directed to a learning device that updates the reference data by machine learning using training data based on the image captured by the image capturing device. and an updating unit that outputs an image and acquires the updated reference data from the learning device.

As one form of the visualization system according to the present invention,
the visualization control device described above;
a photographing device mounted on the aircraft and configured to provide the visualization control device with a photographed image in which the surroundings of the aircraft are photographed;
a display device that visualizes and warns the presence of the flying object determined by the visualization control device to be in danger of approaching or colliding with the aircraft;
and a learning device that updates the reference data used by the visualization control device by machine learning using teacher data based on the image captured by the imaging device.

As one form of the visualization control method according to the present invention,
The flying object is detected by using reference data representing the flying object to be detected from a photographed image of the surroundings of the aircraft taken by a photographing device mounted on the aircraft, and the flying object and the aircraft are detected. and the direction of movement of the flying object relative to the aircraft;
determining whether there is a risk of the flying object approaching or colliding with the aircraft using the detected moving direction and distance of the flying object;
controlling the operation of a display device that visualizes and warns the presence of the flying object that is mounted on the aircraft and is determined to be in danger of approaching or colliding with the aircraft;
During the period in which the aircraft is on standby on the ground, the image captured by the image capturing device is directed to a learning device that updates the reference data by machine learning using training data based on the image captured by the image capturing device. An image is output, and the updated reference data is acquired from the learning device.

A program storage medium according to the present invention has, as one form thereof,
The flying object is detected by using reference data representing the flying object to be detected from a photographed image of the surroundings of the aircraft taken by a photographing device mounted on the aircraft, and the flying object and the aircraft are detected. and detecting the direction of movement of the flying object relative to the aircraft;
a process of determining whether there is a risk of the flying object approaching or colliding with the aircraft, using the detected moving direction and distance of the flying object;
a process of controlling the operation of a display device that visualizes and warns the presence of the flying object that is mounted on the aircraft and that has been determined to be at risk of approaching or colliding with the aircraft;
During the period in which the aircraft is on standby on the ground, the image captured by the image capturing device is directed to a learning device that updates the reference data by machine learning using training data based on the image captured by the image capturing device. A computer program is stored that causes a computer to execute a process of outputting an image and acquiring the updated reference data from the learning device.

According to the present invention, it is possible to notify an aircraft operator of information prompting an action to avoid a collision with a flying object.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure explaining the structure of the visualization system of 1st Embodiment which concerns on this invention. FIG. 2 is a diagram illustrating a functional configuration of a visualization control device that configures the visualization system of the first embodiment; FIG. FIG. 4 is a diagram showing a specific example of a flying object set as a detection target; It is a figure showing an example of the data for determination. It is a figure showing an example of the data for information control. It is a figure showing the example of alerting|reporting by a display apparatus. It is a figure showing another example of alerting|reporting by a display apparatus. It is a flow chart showing an example of operation of a visualization control device. It is a figure explaining other embodiments. It is a figure explaining the structure of the visualization system of 2nd Embodiment which concerns on this invention. It is a figure explaining functional composition of a visualization control device which constitutes a visualization system of a 2nd embodiment. 9 is a flow chart showing an operation example of the visualization control device of the second embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments according to the present invention will be described below with reference to the drawings.

<First embodiment>
The visualization system of the first embodiment according to the present invention aims at a pilot 44 of a helicopter 40, which is an aircraft such as that shown in FIG. It is a system capable of notifying alert information that prompts action. FIG. 2 is a block diagram showing the configuration of the visualization system 1 of the first embodiment. The visualization system 1 of the first embodiment has a function of generating and notifying alert information using a photographed image in which the periphery of the helicopter 40 is photographed. , a display device 5 and a base device 6 .

The photographing device 4 is, for example, a video camera, and is mounted on a helicopter 40 to photograph the surroundings of the helicopter 40 and output moving images as photographed images to the visualization control device 3 . Considering that the surroundings of the helicopter 40 are captured, the imaging device 4 may employ a 360-degree video camera, or may employ a video camera having a pan function and a tilt function. In addition, if a situation in which a flying object collides with the helicopter 40 from behind the helicopter 40 is not assumed, there may be no photographed image of the rear of the helicopter 40 . In other words, the photographing device 4 may employ a photographing device whose photographing range does not include the area behind the helicopter 40 . In this way, the imaging range required for the imaging device 4 is appropriately set.

Information indicating a preset reference direction is associated with the captured image output from the imaging device 4 . The reference direction is, for example, the north direction. Information representing a reference direction based on a sensor output from an orientation sensor built in the imaging device 4 or mounted on the helicopter 40 is associated with the captured image. The captured image output from the image capturing device 4 is also associated with information representing the time of capturing and information identifying the image capturing device 4 that captured the image.

The base device 6 is a device installed in a facility provided at a base where the helicopter 40 waits, and is configured here with a learning device 30 and a database 32 as a storage device.

The learning device 30 is a computer device having a function of generating reference data used by the visualization control device 3 by machine learning. That is, the learning device 30 includes an arithmetic device 31 including a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The computing device 31 has a function of generating reference data by machine learning using teacher data based on the captured image captured by the imaging device 4 . Here, the reference data is data used in the process of detecting a flying object that is expected to fly in the airspace of the helicopter 40 . Examples of flying objects include unmanned aerial vehicles (UAVs), flying vehicles, airships, helicopters, birds, small planes, commercial aircraft, military aircraft, missiles, skydivers, balloons and gliders (hang gliders). and paragliders). In the example of FIG. 1, an unmanned aerial vehicle 41 and a flying car 42 are illustrated as flying objects.

Further, the teacher data used by the arithmetic unit 31 for machine learning may be captured image data to which information for identifying a flying object has been added by a person, or a flying object can be identified by pattern recognition using AI (Artificial Intelligence) technology. It may be photographed image data to which identification information is added.

Furthermore, the method by which the learning device 30 generates the reference data by machine learning is a generation method determined according to the processing method executed by the visualization control device 3 using the reference data, and the description thereof is omitted here. Note that machine learning also includes deep learning.

The database 32 is a storage medium that stores data including reference data generated by the learning device 30 (arithmetic device 31), teacher data used to generate the reference data, and data of images captured by the imaging device 4. .

The display device 5 is a device that visually notifies information. Here, the information notified (displayed) by the display device 5 is provided from the visualization control device 3 . Further, the target of information to be notified by the display device 5 is the operator 44 of the helicopter 40 . Thereby, the display device 5 has a mode in which the operator 44 can easily recognize the displayed information. For example, in the example of FIG. 1, the display device 5 is in the form of smart glasses. That is, the display device 5 is a glasses-like wearable terminal as shown in FIG. 6, and is, for example, a retinal scanning display device (display).

The visualization control device 3 is a computer device that uses the display device 5 to provide the operator 44 of the helicopter 40 with information prompting an action to avoid a collision with the flying object. That is, the visualization control device 3 has an arithmetic device 10 and a storage device 11 . The storage device 11 is a storage medium that stores data and computer programs (hereinafter also referred to as programs) 12 . There are various types of storage media, and the storage device 11 may be configured with any storage media. Also, the visualization control device 3 may be provided with a plurality of types of storage media. A description of the configuration and operation of the storage device 11 is omitted.

The computing device 10 includes, for example, a processor such as a CPU or GPU. By reading and executing a program stored in the storage device 11, the processor can have various functions based on the program. For example, in the first embodiment, the computing device 10 has functional units such as a detection unit 15 , a determination unit 16 , a display control unit 17 , and an update unit 18 .

The detection unit 15 detects the flying object from the captured image captured by the imaging device 4, and further detects the distance between the flying object and the helicopter, the movement direction of the detected flying object, and the speed of the flying object. It has a function to Specifically, the detection unit 15 includes an object detection unit 20 , a distance detection unit 21 , a direction detection unit 22 and a speed detection unit 23 . Here, a flying object that is assumed to fly in the flight airspace of the helicopter 40 is preset as a detection target. For example, a flying object as shown in FIG. 3 is set as a detection target. In the example of FIG. 3, an unknown flying object whose type is not specified (in other words, an object that can be detected as flying but cannot be identified) is also set as an unknown object as a detection target. .

The object detection unit 20 has a function of detecting a flying object to be detected from the captured image captured by the imaging device 4 using the reference data stored in the storage device 11 . Reference data is data for detecting a flying object set as a detection target, and is generated by the learning device 30 and stored in the storage device 11 . The object detection unit 20 may perform detection processing for detecting a flying object to be detected for all frame images that make up the captured image. may be executed to When selecting the frame images for which the detection process is to be performed, for example, when the frame images are arranged in chronological order, frame images are selected for each number determined in consideration of the processing capability of the arithmetic unit 10 and the like. .

As a method for detecting a flying object from a captured image, for example, there is a method using AI (Artificial Intelligence) technology. There are a plurality of detection methods that can be employed as a method for detecting a flying object from a photographed image using AI technology. A detection method appropriately selected in consideration of the matter is adopted. The description of the detection method is omitted. As described above, when an unknown flying object is also set as a detection target, the learning device 30 generates reference data using captured images of unknown flying objects as teacher data. have a function. Using the reference data generated in this way, the object detection unit 20 detects an unknown flying object. Alternatively, for example, the learning device 30 has a function of generating reference data of a background image by machine learning using teacher data using captured images of various sky conditions. Then, the object detection unit 20 uses the background image reference data and the data for detecting an identifiable flying object to detect an object that is neither a background nor an identifiable flying object in the captured image. In some cases, it may be detected that there is an unknown flying object.

The distance detection unit 21 has a function of detecting the distance between the detected flying object (hereinafter also referred to as the detected flying object) and the helicopter 40 from the captured image. That is, the storage device 11 is provided with distance calculation data in advance. The distance calculation data is data for calculating the distance between the detected flying object and the helicopter 40 using the size (for example, the number of pixels) of the detected flying object in the captured image. The distance calculation data is, for example, relationship data between the size of the detected flying object in the captured image and the distance between the detected flying object and the helicopter 40 . This relational data is generated using the results of experiments and simulations that take into consideration the actual size of the detected flying object and the settings of the photographing device 4 such as the lens magnification when actually photographing.

The distance detection unit 21 detects the size (for example, the number of pixels) of the detected flying object in the captured image, and uses the detected size of the detected flying object and data for distance calculation to determine the distance between the detected flying object and the helicopter 40 . Calculate the distance between

The direction detection unit 22 has a function of detecting the movement direction of the detected flying object with respect to the helicopter 40 when the detection target flying object is detected from the captured image. For example, the direction detection unit 22 detects the displacement direction of the detected flying object in a plurality of frame images that are time-series data in which the detected flying object is detected, and the distance between the detected flying object and the helicopter 40 calculated by the distance detection unit 21. to detect changes in the distance of Furthermore, the direction detection unit 22 detects the moving direction of the detected flying object relative to the helicopter 40 by using the displacement direction of the detected flying object and the change in distance. Note that the photographing device 4 has a pan function and a tilt function, and photographing may be performed using the pan function and the tilt function. In this case, for example, the position of the detected flying object in the captured image is specified using a reference two-dimensional coordinate system determined based on the reference direction information included in the captured image, and this specified position is used. to detect the displacement direction of the detected flying object. As a result, the displacement direction of the detected flying object can be detected without being adversely affected by changes in the imaging range due to the pan function and tilt function.

The speed detection unit 23 has a function of detecting the relative speed of the detected flying object with respect to the helicopter 40 (hereinafter simply referred to as flying object speed). To detect the flying object speed, the movement direction of the detected flying object by the direction detection unit 22, the change in the distance between the detected flying object by the distance detection unit 21 and the helicopter 40, and the time required for the change in the distance. is used. The time required for the change in distance is acquired using information about the shooting time attached to the frame image.

The determination unit 16 has a function of determining whether or not there is a risk of a flying object approaching or colliding with the helicopter 40 . The determination includes the distance between the flying object detected by the distance detection unit 21 and the helicopter 40, the moving direction of the flying object detected by the direction detection unit 22, the speed of the flying object detected by the speed detection unit 23, and the previously set and stored Judgment data stored in the device 11 is used.

Judgment data is data representing judgment criteria for judging the risk of a flying object approaching or colliding with the helicopter 40 based on the above-described combination of moving direction, distance, and speed.

FIG. 4 shows a specific example of determination data. The determination data shown in FIG. 4 is data that can determine not only the presence or absence of danger, but also the level representing the degree of danger. According to the determination data in FIG. 4, the flying object has a high speed, the distance between the detected flying object and the helicopter 40 is short, the detected flying object exists near the helicopter 40, and the detected flying object is close to the helicopter 40. If the helicopter 40 is moving towards it, the danger is there. Also, in this case, the risk level is "6". The risk levels here are represented by integers from 1 to 6, and the risk increases from level 1 to level 6.

When the determination unit 16 determines whether there is a risk of the flying object approaching or colliding with the helicopter 40 using the determination data shown in FIG. A speed classification value is given in advance to determine whether the speed is slow or slow. The speed segment value is preset, for example, based on the average speed of the helicopter 40 in stable flight conditions in which the range of variation in the speed of the helicopter 40 is within a preset allowable range. In other words, when determining whether or not there is a risk of the flying object approaching or colliding with the helicopter 40, the relative speed of the flying object with respect to the helicopter 40 is considered to be an important criterion. For this reason, it is conceivable to use the average speed of the helicopter 40 in stable flight conditions to set the speed classification value for judging whether the flying object speed is fast or slow. Note that the method of setting the speed category value is not limited to this example.

The determination unit 16 has a function of determining whether the flying object speed is fast or slow based on the speed classification value set as described above. Furthermore, the determination unit 16 uses the distance determination information to determine whether the distance between the detected flying object and the helicopter 40 is "long," "medium," or "close." Prepare. The distance determination information is information indicating whether the distance between the detected flying object and the helicopter 40 is "long," "medium," or "close," and is set and given in advance. The distance determination information includes range information representing "distant", range information representing "middle", and range information representing "near". The distance between the detected flying object and the helicopter 40 here is one of the criteria for determining whether there is a risk of the flying object approaching or colliding with the helicopter 40 . That is, when the distance between the detected flying object and the helicopter 40 is short, there is a high risk that the flying object will approach or collide with the helicopter 40, and as the distance increases, the flying object will approach or collide with the helicopter 40. becoming less dangerous. It is believed that the average speed of the helicopter 40 under stable flight conditions is involved in the relationship between distance and risk. As a specific example of the setting of the distance determination information, for example, the range representing "distant", the range representing "middle", and the range representing "near" correspond to the helicopter 40 in a stable flight state. is determined based on the average speed of Alternatively, these ranges may be ranges defined by a combination of the average velocity of the helicopter 40 in stable flight conditions and the classification of the flying object velocity (fast or slow) as shown in FIG.

The determination unit 16 determines whether there is a risk that the flying object detected by the object detection unit 20 will approach or collide with the helicopter 40 by using the determination data and information on the movement direction, distance, and speed of the detected flying object. determine whether or not Furthermore, in the first embodiment, the determination unit 16 further determines the level of risk.

The display control unit 17 has a function of controlling the operation of the display device 5 that notifies the operator 44 of information. For example, it is assumed that the storage device 11 stores notification control data as shown in FIG. The notification control data is data that the display control unit 17 refers to when controlling the display operation of the display device 5. The type of information to be displayed according to the level of danger determined by the determination unit 16 and its display are determined. Data representing the method. For example, in the standard setting shown in FIG. 5, when the danger is level 1, the mark and name of the flying object detected by the detection unit 15 are displayed in green and small, and other information is not displayed. Information for display control is set. When the danger is level 6, the mark and name of the flying object detected by the detection unit 15, the distance, the speed, the direction of movement, and the information on the danger level and the warning display by the determination unit 16 are red and maximum. Information for display control to be displayed is set. As the flying object mark, for example, a mark as shown in FIG. 3 is displayed. Also, the display sizes “small”, “medium”, “large” and “maximum” are set in advance based on the size of the display area of the display device 5 . In addition, the color of the display, the wording of the warning display, the symbol, the character decoration, etc. are appropriately set in consideration of the clarity of the display.

Here, it is assumed that selection information is associated with the flying object mark and name, distance, speed, moving direction, and risk level information that can be notified by the display device 5 . The selection information is information indicating whether notification is necessary or not, and when an input device (not shown) for inputting information to the visualization control device 3 is operated by, for example, the operator 44, either the necessity or the unnecessary of notification is determined. It is selected to form selection information. It is assumed that this selection information can be set for each risk level.

When the notification control data as described above is stored in the storage device 11, the flying object is detected by the detection operation of the detection unit 15, and the distance, speed, and moving direction of the detected flying object are determined. Suppose it is detected. Furthermore, it is assumed that the judgment unit 16 has judged the presence or absence of danger and the danger level. The display control unit 17 controls the detection flight based on the information about the type of flying object detected by the detection unit 15 and the determination unit 16, the presence or absence of danger, and the level of danger, and the information control data including the information selected by the operator 44. The operation of the display device 5 is controlled so that the information regarding the object is notified to the operator 44 .

That is, the display control unit 17 has a function of visualizing and warning the presence of a flying object that may approach or collide with the helicopter 40 by controlling the operation of the display device 5 .

In the example of the standard settings shown in FIG. 5, even when it is determined that there is no danger (level 1), the mark and name information of the flying object detected by the detection unit 15 and the determination unit 16 are not displayed by the operator. 44 by the display device 5. However, in the case of level 1, if the information selected by the operator 44 as described above is set so that notification of the detected flying object mark and name information is unnecessary, the detected flying object mark and name information (ie nothing) is displayed by the display device 5 .

The updating unit 18 learns the data of the captured images captured by the imaging device 4 and stored in the storage device 11 while the helicopter 40 is on standby on the ground (hereinafter also referred to as the helicopter 40 on standby on the ground). It has a function of outputting to the device 30 . That is, the fact that the helicopter 40 is on standby on the ground means that the propeller of the helicopter 40 is stopped by detecting, for example, a sensor output from a rotation speed sensor that detects the rotation speed of the propeller of the helicopter 40. can be detected by As a result, the updating unit 18 detects that the helicopter 40 is waiting on the ground based on the sensor output of the propeller rotation speed sensor (for example, when detecting landing), the learning device 30 , and outputs the captured image data in the storage device 11 toward the learning device 30 . As described above, the data of the captured image to be output is associated with the information of the capturing time and the information for identifying the capturing device 4 that captured the image.

In the learning device 30 that has received the photographed image from the helicopter 40, the data of the received photographed image is written in the database 32, and the data of the photographed image that has been received and added is used for reference by machine learning using teacher data. Data is updated. The updated reference data overwrites the reference data in the database 32 .

The updating unit 18 further has a function of synchronizing the reference data stored in the storage device 11 of the helicopter 40 with the reference data in the database 32 via the learning device 30 while the helicopter 40 is on standby on the ground. The timing for synchronizing the reference data can be, for example, the timing when the helicopter 40 transitions from the non-operating state to the operating state and is about to take off. As a result, while the helicopter 40 is in flight, the detection unit 15 in the calculation device 10 can perform a detection operation using the latest reference data.

The configurations of the visualization system 1 and the visualization control device 3 of the first embodiment are configured as described above. Next, an operation example of the visualization control device 3 will be described using the flowchart of FIG.

For example, when takeoff preparations are started to take off the helicopter 40 and power is turned on, the updating unit 18 of the visualization control device 3 connects to the learning device 30 of the base device 6 . Then, the update unit 18 synchronizes the reference data in the storage device 11 with the reference data in the database 32 of the base device 6 via the learning device 30 (step S101). In addition, the display control unit 17 notifies the operator 44 through the display device 5 of information of options for allowing the operator 44 to select the information notified by the display device 5 . Based on the received operation information of the input device by the operator 44, the display control unit 17 sets notification conditions (that is, selection information of notification control data) to the operator 44 by the display device 5 (step S102).

After that, during the flight after the helicopter 40 takes off, the detection unit 15 analyzes the image captured by the imaging device 4 by AI technology using reference data, and detects flying objects around the helicopter 40 . Further, when detecting a flying object, the detection unit 15 detects the moving direction of the detected flying object and the distance between the helicopter 40 and the detected flying object. The detection unit 15 executes detection processing for detecting the above-described flying object and detecting the movement direction and distance of the detected flying object (step S103). Furthermore, when a flying object is detected by the detection unit 15, the determination unit 16 determines the danger of the detected flying object approaching or colliding with the helicopter 40 based on the information detected by the detection processing and the determination data. A judgment process for judging whether or not there is a property is executed (step S104). Furthermore, the display control unit 17 controls the operation of the display device 5 with reference to the notification control data, so that the information obtained by the detection processing by the detection unit 15 and the information obtained by the determination processing by the determination unit 16 are transmitted to the operator 44. (step S105).

The detection process, determination process, and notification process described above are repeatedly executed while the helicopter 40 is in flight.

After that, when the update unit 18 detects that the helicopter 40 has landed (while the helicopter 40 is on standby on the ground), the update unit 18 connects to the learning device 30 and updates the captured image data stored in the storage device 11 to the learning device. 30 (step S106). The learning device 30 that has received the captured image data stores the received captured image data in the database 32 . Further, the learning device 30 updates the reference data used by the detection unit 15 in the visualization control device 3 by machine learning using the teacher data based on the data of the added captured image, and stores the updated reference data in the database. Overwrite the reference data of 32.

The visualization system 1 of the first embodiment and the visualization control device 3 constituting it detect a flying object that has a risk of colliding with a helicopter 40, which is an aircraft, from a captured image, and visualize information about the detected flying object. It has a function of notifying the operator 44 by pressing the button. As a result, the visualization system 1 of the first embodiment and the visualization control device 3 that constitutes it can notify the operator 44 of the helicopter 40 of information prompting an action to avoid a collision with a flying object. Obtainable.

Further, in the first embodiment, the display device 5 is in the form of smart glasses and is a device that can be worn by the operator 44. Therefore, there is no need to fix the display device 5 inside the helicopter 40. In addition, the space inside the helicopter 40 can be expanded.

Furthermore, in the first embodiment, the operation of the display device 5 is controlled so as to clearly notify that there is a danger when the danger is notified. As a result, the visualization system 1 of the first embodiment and the visualization control device 3 constituting it can effectively notify the operator 44 of the helicopter 40 that there is a danger.

Furthermore, in the first embodiment, the visualization control device 3 is provided with the updating unit 18, and by the function of the updating unit 18, the base device 6 is provided with the photographed images taken during flight, and the learning device 30 provides the photographed images It has a configuration that can be used for machine learning of reference data. Furthermore, the visualization control device 3 can synchronize the reference data in the storage device 11 with the reference data updated by machine learning by the learning device 30 by the function of the updating unit 18 . With such a configuration, the visualization control device 3 can detect the flying object from the captured image using the updated reference data, thereby increasing the reliability of the detection of the flying object. .

Furthermore, in the first embodiment, the visualization control device 3 is configured to connect to the base device 6 and perform data communication with the base device 6 while the helicopter 40 is on standby on the ground. As a result, the visualization control device 3 can perform data communication with the base device 6 in a situation in which poor communication is less likely to occur than during flight.

In the first embodiment, the display device 5 is in the form of smart glasses and is a device that can be worn by the operator 44 . Alternatively, the display device 5 may be, for example, a projector that displays information on a glass window in front of the operator 44 as shown in FIG. Further, the display device 5 may be smart glasses or a display device other than that shown in FIG.

In addition, the base device 6 can be connected to a plurality of helicopters 40, and the learning device 30 utilizes teacher data based on the captured images of the imaging device 4 received from each of the plurality of helicopters 40, and the visualization control device 3 may be configured to machine-learn the reference data used by . In this case, the number of teacher data used for machine learning of reference data can be increased compared to the case of using images captured by one helicopter 40, and the situation in which the flying object is captured can be increased. It becomes easy to obtain captured images in which the number is variously different. As a result, reference data that can improve the detection accuracy of the detection unit 15 of the visualization control device 3 can be generated, so the reliability of the information that the visualization system and the visualization control device 3 notify the operator 44 can be increased.

Further, the reference data not only includes machine-learned data of flying objects to be detected, but also flying objects that do not need to be detected (for example, small birds that are assumed to have a small impact on the helicopter 40) are excluded from detection targets. , which includes machine-learned data.

Furthermore, in the examples of FIGS. 6 and 7, the distance and speed information are displayed as character information such as "close" and "fast", but more specific numerical information is displayed. may Furthermore, the direction detection unit 22 shows an example having a function of detecting the moving direction of the detected flying object relative to the helicopter 40 . Alternatively, for example, the direction detection unit 22 may detect direction information as the movement direction of the detected flying object by also using information from a direction sensor mounted on the helicopter 40 . In such a case, for example, the direction information displayed by the display device 5 may be azimuth (for example, “northward” or “southeastward”).

Furthermore, the visualization control device 3 may further include a reaction detector 19 as shown in FIG. 9, the storage device 11 in the visualization control device 3 and the detection unit 15, the determination unit 16, and the update unit 18 in the calculation device 10 are omitted.

The reaction detection unit 19 determines that the operator 44 does not react even though the display device 5 has notified the operator 44 of the information indicating that the danger is high by the operation of the display control unit 17 . It has a function to detect For example, the reaction detection unit 19 determines whether or not the operator 44 reacts when the display device 5 informs the operator 44 of information indicating that the danger is high. Here, the response of the operator 44 means that the operator 44 makes a maneuver to change the traveling direction and flight altitude of the helicopter 40 in response to the information indicating that the danger is high by the display device 5. is to do Such a reaction of the operator 44 is detected based on operation information of a control lever or the like.

When the reaction detection unit 19 detects that the operator 44 does not react, the display control unit 17 changes the notification method of notifying that the risk is high to the direction of more clearly notifying that the risk is high. It further has a function to change to With such a configuration, the visualization control device 3 can more effectively notify the operator 44 of the high risk. Furthermore, although the reaction detection unit 19 has changed the notification method to more clearly notify the high risk by the display control unit 17, even if a predetermined time has passed since the change, A function of detecting that the operator 44 does not react may be provided. When the response detection unit 19 detects that the operator 44 does not respond despite the change in the notification method by the display control unit 17, the visualization control device has a function of reporting to a predetermined reporting destination. 3 may be provided. The report destination is, for example, a management center that manages the operation of the helicopter 40 . By providing such a notification structure, it becomes possible for the management center to alert the operator 44 by wireless communication. Further, for example, the control center can detect that a serious situation has occurred in which the operator 44 cannot respond to the notification that the danger is high due to poor physical condition.

<Second embodiment>
FIG. 10 is a block diagram showing the configuration of a visualization system according to the second embodiment of the invention. Also, FIG. 11 is a block diagram showing the functional configuration of a visualization control device that constitutes the visualization system of the second embodiment.

A visualization system 50 of the second embodiment includes an imaging device 51 , a visualization control device 52 , a display device 53 and a learning device 54 .

The imaging device 51 is a device that is mounted on an aircraft and provides the visualization control device 52 with a captured image in which the surroundings of the aircraft are captured.

The display device 53 is a device that visually warns the presence of a flying object that has been determined by the visualization control device 52 to approach or collide with an aircraft.

The learning device 54 has a function of updating reference data used by the visualization control device 52 by machine learning using teacher data based on images captured by the imaging device 51 .

The visualization control device 52 includes a detection unit 55, a determination unit 56, a display control unit 57, and an update unit 58, as shown in FIG.

The detection unit 55 has a function of detecting a flying object from an image captured by the imaging device 51 using reference data generated by the learning device 54 and representing the flying object to be detected. The detection unit 55 also has a function of detecting the distance between the flying object and the aircraft, and the moving direction of the flying object with respect to the aircraft.

The determination unit 56 has a function of determining whether or not there is a risk of the flying object approaching or colliding with the aircraft, using the detected moving direction and distance of the flying object.

The display control unit 57 has a function of visualizing and warning the presence of a flying object determined to be in danger of approaching or colliding with an aircraft by controlling the operation of the display device 53 .

The updating unit 58 outputs an image captured by the imaging device 51 to the learning device 54 while the aircraft is on standby on the ground, and acquires updated reference data from the learning device 54. Prepare.

Next, an operation example of the visualization control device 52 will be described with reference to FIG. For example, the detection unit 55 of the visualization control device 52 detects the flying object to be detected from the image captured by the imaging device 51 during flight of the aircraft, and also detects the distance between the flying object and the aircraft, and the distance between the aircraft and the aircraft. The moving direction of the flying object relative to is detected (step S301). Then, the determining unit 56 determines whether there is a risk of the flying object approaching or colliding with the aircraft, using the detected moving direction and distance of the flying object (step S302). Thereafter, when it is determined that there is a risk of approaching or colliding with an aircraft, the display control unit 57 visually warns of the danger by controlling the operation of the display device 53 (step S303). After that, after the aircraft has landed, the update unit 58 outputs the image captured by the image capturing device 51 to the learning device 54 (step S304).

The visualization system 50 and the visualization control device 52 of the second embodiment are configured as described above, so that, similarly to the first embodiment, the visualization system 50 and the visualization control device 52 are directed toward the aircraft operator to avoid a collision with a flying object. It is possible to obtain the effect of being able to notify the information prompting the

The present invention has been described above using the above-described embodiments as exemplary examples. However, the invention is not limited to the embodiments described above. That is, within the scope of the present invention, various aspects that can be understood by those skilled in the art can be applied to the present invention.

Reference Signs List 1, 50 visualization system 3, 52 visualization control device 4, 51 imaging device 15, 55 detection unit 16, 56 determination unit 17, 57 display control unit

Claims (8)

The flying object is detected by using reference data representing the flying object to be detected from a photographed image of the surroundings of the aircraft taken by a photographing device mounted on the aircraft, and the flying object and the aircraft are detected. and sensing means for sensing the distance between and the direction of movement of the flying object relative to the aircraft;
determining means for determining whether there is a risk that the flying object will approach or collide with the aircraft, using the detected moving direction and distance of the flying object;
display control means for controlling the operation of a display device mounted on the aircraft and visualizing and warning the presence of the flying object determined to be at risk of approaching or colliding with the aircraft;
During the period in which the aircraft is on standby on the ground, the image captured by the image capturing device is directed to a learning device that updates the reference data by machine learning using training data based on the image captured by the image capturing device. A visualization control device, comprising: updating means for outputting an image and acquiring the updated reference data from the learning device. The determination means has a function to determine a level representing the degree of risk,
2. The visualization control device according to claim 1, wherein said display control means changes a warning notification method by said display device according to said level. The visualization control according to claim 1 or 2, wherein the updating means acquires the updated reference data while the aircraft is waiting on the ground and preparing for takeoff. Device. A visualization control device according to any one of claims 1 to 3;
a photographing device mounted on the aircraft and configured to provide the visualization control device with a photographed image in which the surroundings of the aircraft are photographed;
a display device that visualizes and warns the presence of the flying object determined by the visualization control device to be in danger of approaching or colliding with the aircraft;
and a learning device that updates the reference data used by the visualization control device by machine learning, using training data based on the image captured by the imaging device. 5. A visualization system according to claim 4, wherein said display device is adapted to be worn by an operator of said aircraft. 5. The visualization system of claim 4, wherein the display device comprises aspects for projecting information onto a window in front of an operator of the aircraft. by computer,
The flying object is detected by using reference data representing the flying object to be detected from a photographed image of the surroundings of the aircraft taken by a photographing device mounted on the aircraft, and the flying object and the aircraft are detected. and the direction of movement of the flying object relative to the aircraft;
determining whether there is a risk of the flying object approaching or colliding with the aircraft using the detected moving direction and distance of the flying object;
controlling the operation of a display device that visualizes and warns the presence of the flying object that is mounted on the aircraft and is determined to be in danger of approaching or colliding with the aircraft;
During the period in which the aircraft is on standby on the ground, the image captured by the image capturing device is directed to a learning device that updates the reference data by machine learning using training data based on the image captured by the image capturing device. A visualization control method for outputting an image and acquiring the updated reference data from the learning device. The flying object is detected by using reference data representing the flying object to be detected from a photographed image of the surroundings of the aircraft taken by a photographing device mounted on the aircraft, and the flying object and the aircraft are detected. and detecting the direction of movement of the flying object relative to the aircraft;
a process of determining whether there is a risk of the flying object approaching or colliding with the aircraft, using the detected moving direction and distance of the flying object;
a process of controlling the operation of a display device that visualizes and warns the presence of the flying object that is mounted on the aircraft and that has been determined to be at risk of approaching or colliding with the aircraft;
During the period in which the aircraft is on standby on the ground, the image captured by the image capturing device is directed to a learning device that updates the reference data by machine learning using training data based on the image captured by the image capturing device. A computer program for causing a computer to execute a process of outputting an image and acquiring the updated reference data from the learning device.

Images