JP2021030806A - Steering-assistance system - Google Patents

Abstract

To assist an operator to enhance ability to recognize the situation in the outside even under low-visibility condition, by displaying a high-visibility image in a manner of superimposing the image on the outside.SOLUTION: A steering-assistance system comprises: a viewing range identification part (20, 1) that identifies a viewing range that a user in a moving body is viewing; a real image acquisition part (30, 21, 1) that acquires a real image of an outside that can be seen from the moving body; a three-dimensional map creation part (1) that creates a virtual image of the outside that is supposed to be visible from the moving body on the basis of three-dimensional map information near a current position of the moving body; and a display processing part (1, 22) that matches respective parts of the real image and the virtual image corresponding to at least the viewing range with each other, and display a post-matching image in the corresponding-parts-matched state, in a manner of superimposing the image on a view field of the user.SELECTED DRAWING: Figure 1

Description

The present invention relates to a maneuvering support system that assists maneuvering a moving body including an aircraft, a vehicle, and a ship.

For aircraft such as helicopters and airplanes, when the visibility is reduced due to nighttime or weather and it is difficult for the operator to visually recognize the situation outside the aircraft, flight using a system to assist the maneuvering is performed. Will be done.

Such a maneuvering support system includes an image taken outside the aircraft by an infrared camera or the like, such as a head up display (HUD) or a head (or helmet) mount display (HMD). Some are displayed on a display device (see Patent Document 1 and the like). In addition, instead of such a captured image, a three-dimensional terrain image is generated based on the position information of the own aircraft by GPS or an inertial navigation system (INS) and the terrain data stored in advance, and this is displayed as a display device. It is also displayed in. Even if it is difficult for the operator to visually recognize the situation of the outside world, the operator can grasp the situation of the outside world and operate by using various information displayed on the display device as described above. It can be performed.

Japanese Unexamined Patent Publication No. 2012-224210

When it is difficult for the operator to confirm the situation of the outside world with the naked eye, an image taken by an infrared camera or the like is used to grasp the situation of the outside world. However, depending on the conditions of the outside world, even if the image is taken by an infrared camera, obstacles such as surrounding buildings and mountains may be unclear and difficult to recognize, or only a part of them may be visible.

For example, if the weather suddenly changes while you are maneuvering while looking at an image from an infrared camera and the visibility becomes even worse, you will not be able to grasp the situation of the outside world temporarily, which will reduce flight safety. To do. Therefore, the operator needs to take action to leave or avoid the area as soon as possible.
It should be noted that the same problem may occur not only in an aircraft but also in the case of operating (driving) a vehicle such as an automobile or in the case of operating a ship.

The present invention has been made to solve the above problems, and its main purpose is to improve the ability of the operator to recognize the situation of the outside world at night or under low visibility conditions, thereby flying, operating, running, etc. It is to provide a maneuvering support system that can improve the safety of the aircraft.

The maneuvering support system according to one aspect of the present invention made to solve the above problems is
A visual field range specifying part that specifies the visual field range that the user in the moving body is viewing, and
A real image acquisition unit that acquires a real image of the outside world seen from the moving body,
A 3D map image creation unit that creates a virtual image of the outside world that is assumed to be visible from the moving body based on the 3D map information near the current position of the moving body.
A display processing unit that displays the matched image superimposed on the user's field of view in a state where at least a portion corresponding to the visual field range is matched between the real image and the virtual image.
Is provided.

In the maneuvering support system according to the above aspect, the "moving body" includes an aircraft, a ship, a vehicle, and the like. In addition, the "user in the moving body" is not only the operator who controls the moving body, but also the co-pilot who assists or assists the maneuvering, and whether or not he / she is directly involved in the maneuvering. It also includes passengers who carry out tasks and tasks such as monitoring and searching for the situation of the outside world from moving objects.

In the maneuvering support system according to one aspect of the present invention, the real image acquisition unit acquires a real image of a wide range of the outside world that can be seen from a moving body. However, depending on the conditions outside the moving body such as the weather, a part or all of the actual image may be blurred. On the other hand, the three-dimensional map image creation unit creates a virtual image of the outside world that is assumed to be visible from the moving body based on the three-dimensional map information near the current position of the moving body at that time. Further, the visual field range specifying unit identifies the visual field range visually recognized by the user in the moving body, for example, the operator, and the display processing unit corresponds to at least the visual field range specified in the real image and the virtual image. In the state where the parts are matched, the image after the matching is superimposed on the user's field of view and displayed.

Therefore, according to the maneuvering support system according to one aspect of the present invention, even in a situation where the visibility is low due to nighttime, bad weather, etc., and the whole or a part of the actual photographed image of the outside world is unclear. The three-dimensional map information can be used to supplement all or part of the captured image in the viewing range viewed by the user, and a highly visible image can be displayed on the display screen. As a result, the ability of the user to recognize the situation in the outside world can be improved, and the safety of the user's flight, running, and operation of the moving body can be improved.

The schematic block block diagram of the maneuvering support system which is one Embodiment of this invention. A detailed block configuration diagram of a control / processing unit in the maneuvering support system of the present embodiment. The schematic diagram for demonstrating the process at the time of creating the display image for the main pilot in the maneuvering support system of this embodiment. The schematic diagram for demonstrating the process at the time of creating the display image for the co-pilot in the maneuvering support system of this embodiment. The schematic diagram for demonstrating the matching process of the virtual image and the real image by 3D map information in the maneuvering support system of this embodiment.

An embodiment of the maneuvering support system according to the present invention will be described with reference to the accompanying drawings. The maneuvering support system of this embodiment assists the main pilot and the co-pilot on board the helicopter in maneuvering the helicopter. That is, here, the "moving body" is a helicopter, and the "user in the moving body" is the main pilot and the co-pilot.

[Overall configuration of the maneuvering support system of this embodiment]
FIG. 1 is a schematic block configuration diagram of the maneuvering support system of the present embodiment.
This maneuvering support system includes a control / processing unit 1, two head-mounted display devices 2A and 2B, a sensor pod 3, a flight information collecting unit 4, and a GNSS / INS device 5. Further, the maneuvering support system can be connected to an external digital map database 6 not included in this system, or a storage device in which a part of the map data recorded in the digital map database 6 is stored. including. In the former case, the system includes a transmitter / receiver that accesses the external digital map database 6 and receives only necessary data.

The sensor pod 3 is provided outside the helicopter 100 to be operated (usually below the aircraft). The head-mounted display device 2A is attached to the head of the main operator, and the head-mounted display device 2B is attached to the head of the co-pilot. In this example, the number of head-mounted display devices is 2, but the number is not limited to this and may be 1 or 3 or more.

[Detailed configuration of sensor pot]
The sensor pod 3 includes a multi-sensor unit 30 and a sensor pod control unit 35. The multi-sensor unit 30 includes an infrared camera 31, a visible camera 32, a distance measuring sensor 33, and an illumination unit 34.

The infrared camera 31 acquires an infrared image of the outside world, and is configured to include a sensor capable of detecting one of the near infrared, the middle infrared, the far infrared, or a plurality of different wavelength bands thereof. Can be done. The visible camera 32 is a video camera capable of acquiring an image with high sensitivity even under low illuminance. The distance measuring sensor 33 is a distance measuring device using LiDAR (Light Detection And Ranging) or millimeter wave radar, and obtains the direction and distance where the surrounding terrain, obstacles, etc. are located, and creates a distance image based on the direction and distance. is there. Each of these captures an image of the outside world as seen from the helicopter 100, and each is an example of the "real image acquisition unit" in the above aspect of the present invention.

The illumination unit 34 irradiates an object outside the machine with visible light or infrared light at a low visibility or the like where illumination is required to improve the imaging conditions by the infrared camera 31 and the visible camera 32, resulting in higher visibility. It is possible to acquire a high-quality image. The illumination unit 34 may also be used for a laser range finder or the like.

The sensor pod control unit 35 has a stabilizer function that reduces the influence of vibration of the helicopter 100, and responds to the control signal from the control / processing unit 1 in the azimuth (AZ) direction and elevator of the sensor pod 3. It adjusts the posture in the vibration (EL) direction. In the maneuvering support system of the present embodiment, the sensor pod 3 is oriented in the direction presumed to be seen by the main pilot under the control of the sensor pod control unit 35 that receives the control signal from the control / processing unit 1. A head direction tracking mode that operates so as to, a specific direction fixed mode that operates so as to point in a specified specific direction, and a scan mode that operates so that the direction in which the direction is directed repeatedly changes over a predetermined range. Used in one of the modes. However, in the case of the characteristic processing operation described later, the mode of the sensor pod 3 is the head direction tracking mode.

The cameras and sensors included in the multi-sensor unit 30 are not limited to those described above, and some cameras and sensors may be deleted or other types of cameras and sensors may be added. Further, by mounting a plurality of cameras and sensors of the same type, a wide range can be photographed at one time even if the directivity direction of the sensor pod 3 is constant.

[Detailed configuration of head-mounted display device]
The head-mounted display devices 2A and 2B are, for example, devices integrated with a helmet worn by the main operator and the co-pilot on the head, and are electrically connected to the control / processing unit 1 via the connector 9. Has been done. The head-mounted display devices 2A and 2B include a head angle / position / angular velocity and line-of-sight direction detection unit 20, a high-sensitivity sensor unit 21, a display unit 22, an image selection / composition unit 23, and a battery 24. including.

The head angle / position / angular velocity and line-of-sight direction detection unit 20 of the head-mounted display device 2A were obtained by, for example, a plurality of cameras for photographing the surroundings, a 6-axis gyro sensor, and the plurality of cameras and the gyro sensor. Includes a signal processing unit that processes signals. Each of the plurality of cameras captures the scenery around the main pilot or a predetermined marker in the aircraft (control room), and the signal processing unit positions and tilts the main pilot's head based on the captured image. Calculate the angle.

The position of the head is, for example, position information on a predetermined three-dimensional reference coordinate XYZ inside the aircraft. Further, the inclination angle of the head is, for example, an angle formed by each axis of the three-dimensional reference coordinates XYZ in the machine body and the central axis of the head. On the other hand, the 6-axis gyro sensor detects the angular velocity of the head of the main operator as it moves. Then, the head angle / position / angular velocity and line-of-sight direction detection unit 20 transmits the information on the head position, angle, and angular velocity of the main operator obtained as described above to the control / processing unit 1. This information is used to identify the field of view that the pilot is currently looking at. Therefore, in the present embodiment, the head angle / position / angular velocity and the line-of-sight direction detecting unit 20 are elements constituting the "visual field range specifying unit" in the above aspect of the present invention.

The high-sensitivity sensor unit 21 of the head-mounted display device 2A is a camera capable of acquiring an actual image in a predetermined range in a low illuminance state centering on the direction in which the main pilot is facing his / her face. Therefore, the high-sensitivity sensor unit 21 is also an example of the “real image acquisition unit” in the above aspect of the present invention. The image selection / composition unit 23 selects or combines one of the display image sent from the control / processing unit 1 and the high-sensitivity sensor image obtained by the high-sensitivity sensor unit 21 to display unit 22. Enter in.

The display unit 22 of the head-mounted display device 2A is arranged between a display element such as a transmissive liquid crystal display element, a visor (or combiner) arranged in front of the main operator's eyes, and the display element to the visor. Including the irradiation optical system. The image input from the image selection / compositing unit 23 is displayed on, for example, a transmissive liquid crystal display element. Then, the image light emitted from the display element passes through the irradiation optical system, is reflected by the visor, and reaches the eyes of the main pilot as parallel light. The visor allows outside light coming from the outside world in the direction the main pilot is facing. Therefore, in front of the main pilot's eyes, the background image due to the external light, which is generally parallel light, and the display image due to the reflected light from the visor are superimposed and displayed. As is clear from the description below, in the system of the present embodiment, the image selection / synthesis unit 23 and the display unit 22 together with a part of the control / processing unit 1 form a "display processing unit" in the above aspect of the present invention. It is a constituent element.

The head-mounted display device 2B used by the co-pilot also has the same components as the head-mounted display device 2A described above. Drive power is supplied from the control / processing unit 1 to these head-mounted display devices 2A and 2B via the connector 9. For example, when the operator leaves the aircraft due to an emergency or the like, the connector 9 is supplied. Even when the head-mounted display devices 2A and 2B are separated from the control / processing unit 1, the high-sensitivity sensor unit 21 uses the power supplied from the built-in battery 24. It is possible to display the high-sensitivity sensor image according to the above on the display unit 22.

[Detailed configuration of control / processing unit]
FIG. 2 is a detailed block configuration diagram of the control / processing unit 1 in the system of the present embodiment.
As shown in FIG. 2, the control / processing unit 1 includes a symbol image creation unit 10, a 3D (three-dimensional) map image creation unit 11, an image matching unit 12, a visual field range identification processing unit 13, a feature portion extraction unit 14, and visual recognition. It includes a sex index calculation unit 15, a parallax correction unit 16, a scaling unit 17, and a display image creation processing unit 18. The display image creation processing unit 18 includes functional blocks such as an image selection unit 180, an image complement unit 181, a field range image extraction unit 182, and an image superimposition unit 183.

Flight information data is input from the flight information collecting unit 4 to the control / processing unit 1, and GNSS / INS data is input from the GNSS / INS device 5. The flight information data is data indicating the flight specifications of the aircraft in flight (helicopter 100 in this example), for example, flight speed, flight altitude, attitude of the aircraft (pitch angle, bank angle), orientation, engine torque, and so on. It is data including the remaining amount of fuel. These data are information obtained based on various devices generally mounted on an aircraft, and the content of the information differs depending on the type of aircraft and the like. Further, the GNSS / INS data is data indicating the flight position in real time with high accuracy.

Further, head angle / position / angular velocity and line-of-sight direction data A and high-sensitivity sensor image data A are input from the head-mounted display device 2A to the control / processing unit 1, and the head is head-mounted from the head-mounted display device 2B. Part angle / position / angular velocity and line-of-sight direction data B and high-sensitivity sensor image data B are input. Furthermore, infrared image data, visible image data, and distance image data are input to the control / processing unit 1 from the multi-sensor unit 30 of the sensor pod 3. Further, digital map data is input to the control / processing unit 1 from the digital map database 6 or a recording device in which map data is recorded in advance.

The control / processing unit 1 executes data processing using various data input as described above, so that the display image A and the display image B used for display in the head-mounted display devices 2A and 2B are displayed. Are created and output to the head-mounted display devices 2A and 2B, respectively.

[Processing operation in the control / processing section]
The data processing operation in the control / processing unit 1 during flight will be described.
The symbol image creating unit 10 uses the flight information data, the head angle / position / angular velocity and line-of-sight direction data A of the main pilot, and the head angle / position / angular velocity and line-of-sight direction data B of the sub-pilot. A symbol image A for the pilot and a symbol image B for the assistant pilot are created. The symbol image is an image centered on, for example, numerical values such as flight speed, text information such as characters, figures, symbols, etc., and is standard so that high visibility is ensured even when superimposed on other images. For example, it is displayed in green, and especially when it is necessary to attract the attention of the operator (in case of emergency, etc.), it is displayed in red.

Information related to flight specifications such as flight speed and flight altitude is common to symbol image A and symbol image B, but not all the information that can be obtained is always displayed, and the information to be displayed is displayed by the FO. It is also possible to change between and the co-pilot. In addition, it is possible to display information on items different between the main operator and the co-pilot, such as the direction the operator is looking at at that time, in the symbol image instead of the information on the flight specifications. Therefore, the symbol image A for the main pilot and the symbol image B for the co-pilot are usually different.

The parallax correction unit 16 receives the data constituting the infrared image, the visible image, and the distance image obtained by the multi-sensor unit 30, and in each of these images, the infrared camera 31, the visible camera 32, and the measurement in the sensor pod 3. The disparity (angle of view) caused by the difference between the mounting position of the distance sensor 33 and the viewpoint position (position of the cockpit) of the main operator and the sub-operator is corrected. Further, the scaling unit 17 performs scaling (enlargement / reduction) processing on each of the above images in order to correct a difference in the angle of view due to the size of the lens or the sensor. The infrared image and the visible image after the parallax correction and the scaling process are input to the feature portion extraction unit 14 and the visibility index calculation unit 15 together with the display image creation processing unit 18. Further, the distance image after the parallax correction and the scaling processing is also input to the display image creation processing unit 18.

Since the high-sensitivity sensor image data A and B are image data obtained from the head-mounted display devices 2A and 2B worn by the main pilot and the sub-pilot, respectively, the image data obtained by the sensor pod 3 Unlike, there is no need to correct the angle of view. Therefore, the high-sensitivity sensor image data A and B are introduced into the feature portion extraction unit 14 and the display image creation processing unit 18 without undergoing parallax correction and scaling processing. Even when parallax correction or scaling processing is performed, it may be difficult to completely correct the parallax or angle of view, and the image quality may deteriorate due to such processing. For this reason, in general, it is easier to obtain a better real image by using the high-sensitivity sensor image data A and B that do not undergo these processes.

The feature site extraction unit 14 includes high-sensitivity sensor image data A, high-sensitivity sensor image data B, and parallax correction and scaling sent from the head-mounted display devices 2A and 2B of the main pilot and the sub-pilot, respectively. The contour and feature parts of the object are extracted from the processed infrared image data, visible image data, and distance image data. The characteristic part is, for example, a characteristic topography, a sign, a boundary between the ground surface and a river, a lake, a coast, etc., or a place where a characteristic boundary, a pattern, etc. due to a vegetation difference or the like appears. On the other hand, the visibility index calculation unit 15 calculates a visibility index value that is a predetermined index for determining the degree of visibility of the image, such as the contrast of the entire image, for each of the images. This visibility index value is not limited to the image unit, and the visibility index value is obtained for each small area in which one image is divided into a plurality of areas, and the visibility index value of each of the plurality of small areas is used. The visibility index value of the entire image may be calculated.

The 3D map image creation unit 11 grasps the position and the direction of travel of the aircraft at that time from GNSS / INS data and the like, and digital map data over a predetermined range of the direction of travel in the vicinity of the position is stored in the digital map database. Collect from 6 to create a 3D map image. The images based on the high-sensitivity sensor image data A and B, the infrared image data, and the visible image data described above are real images of the outside world that are actually visible from the helicopter 100. On the other hand, the 3D map image is a virtual image of the outside world estimated to be visible from the helicopter 100.

Due to the limit of accuracy of GNSS / INS data, time delay, etc., or due to the mobility, flight speed, etc. of the helicopter 100, there is a deviation in position between the 3D map image and the actual image of the outside world. .. Therefore, the image matching unit 12 uses the information of the feature portion obtained from any one or more of the two high-sensitivity sensor images, the infrared image, and the visible image, and uses the information of the feature portion as the actual image of the outside world at that time. Match with 3D map image. Matching here means moving or rotating the entire 3D map image so that an object in the outside world comes to the corresponding position on the image, or enlarging or reducing the whole or part of the 3D map image. It is a process to do.

On the other hand, the visual field range specifying processing unit 13 is mainly based on the head angle / position / angular velocity and line-of-sight direction data A, the head angle / position / angular velocity and line-of-sight direction data B, and the displayable range in the display unit of the display device. Determine the field of view that is visible to the pilot and assistant pilot, respectively. That is, since the direction and angle at which the main pilot is facing the face and the degree of change over time can be obtained from the head angle / position / angular velocity and the line-of-sight direction data A, the center line of the line of sight determined by that can be obtained. The displayable range of the display device centered on is determined as the field of view range that the main pilot is looking at at that time.

Further, in reality, a human is not always looking at the front in the direction in which the face is facing, and the face can move the eyes while facing the front to direct the line of sight in an oblique direction. The eye tracker is known as a device capable of accurately grasping the direction of such a line of sight. The direction in which the main operator is directing the line of sight by using the eye tracker in place of the head angle / position / angular velocity and line-of-sight direction detection unit 20, or in addition to the head angle / position / angular velocity and line-of-sight direction detection unit 20. Can be detected more accurately, and the viewing range seen by the main pilot can be determined more accurately. The same applies to the field of view seen by the co-pilot. The eye tracker can be incorporated into a helmet or visor.

The display image creation processing unit 18 obtained parallax-corrected and scaled infrared images, visible images, and distance images as real images of the outside world, and head-mounted display devices 2A and 2B of each pilot. High-sensitivity sensor images A and B are input. Further, a 3D map image matched with the real image is input as a virtual image based on the map data instead of the real image. Further, the symbol images A and B are also input. In addition, the distance image generally displays each object (obstacle, etc.) outside the aircraft in different colors according to the distance from the helicopter 100, and the direction corresponds to the display position on the image. There is. When the distance measurement data by the distance measurement sensor 33 is point cloud data, that is, data indicating a direction and a distance, this data may be converted into an image form, that is, imaged and used.

Further, information indicating the visual field ranges of the main pilot and the co-pilot is input to the display image creation processing unit 18. The field-of-view range image extraction unit 182 extracts only the images included in the field-of-view range of the main pilot with respect to the infrared image, the visible image, the distance image, and the high-sensitivity sensor image A, which are real images. Similarly, the field-of-view range image extraction unit 182 extracts only the images included in the field-of-view range of the assistant pilot with respect to the infrared image, the visible image, the distance image, and the high-sensitivity sensor image B, which are real images. Further, the field-of-view range image extraction unit 182 also extracts an image included in the field-of-view range of the main pilot and an image included in the field-of-view range of the sub-pilot with respect to the 3D map image matched with the real image.

Further, the visibility index value of each actual image is input to the display image creation processing unit 18 in substantially real time. For example, when the visibility index value of the entire visible image is low, the visible image at that time indicates that the entire visible image is unclear. Further, when the visibility index value for a part of the small area in the visible image is low, it indicates that the partial image of only the small area is unclear. Therefore, in the display image creation processing unit 18, the image selection unit 180 compares the overall visibility index values of the infrared image, the visible image, the distance image, and the two high-sensitivity sensor images A, and among them, the visibility index value. When the value of the visibility index value of the image having the highest value is equal to or higher than a predetermined value, the image is selected.

Further, when the visibility index value of a part of the visible image is low and the visibility index value of the corresponding region in the infrared image, the distance image, or the high-sensitivity sensor images A and B is high, the image complementing unit 181 Complementary processing is performed to replace an image in a region having a low visibility index value among visible images with an infrared image, a distance image, or an image in high-sensitivity sensor images A and B. Furthermore, when the visibility index value is less than a predetermined value for all of the visible image, the infrared image, the distance image, and the high-sensitivity sensor image A, the image selection unit 180 replaces these actual images with the 3D map image A. Select. Further, when the visibility index value of only a part of the visible image, the infrared image, the distance image, and the high-sensitivity sensor image A is low, the 3D map image A is replaced with the actual image only for that area. May be complemented with. In this way, as the image of the viewing range of the main pilot, one of the real images, an image obtained by joining a plurality of different types of real images, an image obtained by joining a real image and a virtual image, or a virtual image ( 3D map image) is obtained. The same applies to the image of the field of view of the co-pilot.

The image superimposition unit 183 superimposes the symbol images A and B on the images for the main pilot and the co-pilot after being selected or complemented as described above to create the display images A and B. It is preferable that the operator can select whether or not to superimpose such images.

Further, as described above, since the distance image is a display of each object (obstacle, etc.) outside the aircraft in different colors according to the distance from the helicopter 100, each object is viewed in different colors when looking at the colors on the distance image. It is easy to grasp the distance to. Therefore, the distance image is not treated as a real image, and the image superimposition unit 183 superimposes the distance image together with the symbol images A and B on the image after being selected or complemented as described above, thereby displaying the display image A. B may be generated.

In the control / processing unit 1, the display image A displayed on the display unit 22 of the head-mounted display device 2A for the main pilot and the head-mounted display device for the co-pilot are displayed by the data processing as described above. A display image B to be displayed on the display unit 22 of 2B is created. The display image thus created is sent to the head-mounted display devices 2A and 2B, respectively, and displayed by the display unit 22. Therefore, even if the outside world is almost invisible in the visible image and the infrared image, the 3D map image is superimposed and displayed on the image by the outside light coming from the outside world in front of the main pilot and the co-pilot. Will be done.

[Specific example of display image creation process]
FIG. 3 is a diagram showing an example of a signal flow in the process of creating a display image for the main pilot. This is the case where the sensor pod 3 follows the direction of the head of the main pilot.
As described above, the 3D map image creation unit 11 creates a 3D map image based on the position information of the aircraft based on the GNSS / INS data, but depending on the accuracy and delay of the GNSS / INS data, the maneuverability of the aircraft, the speed, etc. There may be a discrepancy between the generated 3D map image and the real world. Therefore, the image matching unit 12 uses the feature information obtained from any one or more of the high-sensitivity sensor image A, the infrared image, the visible image, and the distance image, and matches with the real image of the outside world at that time. Generate a 3D map image. A specific example of matching between a real image of the outside world and a 3D map image will be described later.

As described above, the display image creation processing unit 18 extracts an image of a region corresponding to the visual field range of the main pilot from each image. Then, referring to the visibility index value, the image having the highest visibility index value among the visible image, the infrared image, the distance image, and the high-sensitivity sensor image A is selected, and the unclear part in the image is 3D. Complement with map image A. Alternatively, when the visibility of the entire image is low, the 3D map image A is selected instead of the image. The symbol image A is superimposed on the complemented or selected image and output as the display image A.

FIG. 4 is a diagram showing an example of the signal flow of the display image generation processing for the co-pilot.
In this case, since a 3D map image matched with the actual image is obtained as shown in FIG. 3, the display image creation processing unit 18 can perform a visible image, an infrared image, a distance image, and a high-sensitivity sensor image. The image having the highest visibility index value is selected from B, and the unclear portion in the image is complemented with the 3D map image B corresponding to the viewing range of the assistant pilot. Alternatively, when the visibility of the entire image is low, the 3D map image B is selected instead of the image. The symbol image B is superimposed on the complemented or selected image and output as a display image B.

[Specific example of matching between the real outside world and 3D map image]
FIG. 5 is a conceptual diagram showing a specific example of a matching method between a 3D map image and the real outside world.
Now, as shown in FIG. 5A, it is assumed that the actual image in a predetermined range is acquired by the camera included in the multi-sensor unit 30 of the sensor pod 3. Matching between the 3D map image and the real outside world is performed using the contours and feature parts of the object extracted from this image. Here, it is possible to extract a plurality of characteristic parts such as the shape of the mountain ridge line (the part indicated by the thick line in the figure) appearing in the actual image.

The 3D map image creation unit 11 generates a map image of a wide range of the surrounding external world (hereinafter referred to as 3D map image space) based on the digital map data. The image matching unit 12 searches for a portion of the entire 3D map image space that matches the feature portion obtained from the actual image. Then, when a matching portion is found, the map image in the 3D map image space is enlarged / reduced and the position is adjusted accordingly (FIG. 5 (b)). Then, from the map image of the 3D map image space after the matching, as shown in FIG. 5 (c), each image (partial 3D map image) corresponding to the viewing range of each of the main pilot and the sub pilot is obtained. It is cut out and used to replace or complement the displayed images A and B. By creating a map image of a wide range of 3D map image space in this way, even if the main pilot or sub-pilot suddenly changes the field of view, the partial 3D map image that quickly follows the change. Can be obtained.

As described above, in the maneuvering support system of the present embodiment, the infrared image captured by the camera via the visors of the head-mounted display devices 2A and 2B worn by the main pilot and the co-pilot, respectively. A visible image can be displayed superimposed on a background image by the outside world, and if the visibility of the infrared image or visible image deteriorates in whole or in part, it is virtual instead or partially complemented. 3D map image can be displayed. Then, the display image superimposed on the background image can be quickly made to follow the movement of the heads of the main pilot and the co-pilot. As a result, even when the visibility of the outside world with the naked eye is lowered, the ability of the operator to recognize the situation of the outside world can be improved, and the flight safety of the helicopter 100 can be improved.

Of course, not only such pilots and co-pilots, but also those who are on board a moving body such as a helicopter 100 often need to check the situation of the outside world. Therefore, such a system can be used not only by the person who actually operates the system and the person who assists / assists it.

Further, in the system of the above embodiment, after matching the real image and the virtual image (3D map image), the images of the viewing ranges of the main pilot and the sub pilot are extracted. After extracting the image of the viewing range from the real image and the virtual image (3D map image), the extracted images may be matched.

Furthermore, the above-described embodiment is merely an example of the present invention, and is not limited to the above-described modification, and is included in the claims of the present application even if it is appropriately changed, modified, or added within the scope of the purpose of the present invention. It is natural that.

[Various aspects]
It will be apparent to those skilled in the art that the exemplary embodiments described above are specific examples of the following embodiments.

(Section 1) The maneuvering support system according to one aspect of the present invention is
A visual field range specifying part that specifies the visual field range that the user in the moving body is viewing, and
A real image acquisition unit that acquires a real image of the outside world seen from the moving body,
A 3D map image creation unit that creates a virtual image of the outside world that is assumed to be visible from the moving body based on the 3D map information near the current position of the moving body.
A display processing unit that displays the matched image superimposed on the user's field of view in a state where at least a portion corresponding to the visual field range is matched between the real image and the virtual image.
Is provided.

According to the maneuvering support system described in paragraph 1, even in a situation where the visibility is low due to nighttime or bad weather, and the whole or a part of the actual photographed image of the outside world is unclear, 3 The three-dimensional map information can be used to supplement all or part of the captured image in the visual field range viewed by the user, and a highly visible image can be displayed in front of the user's eyes. As a result, the ability of the user to recognize the situation in the outside world can be improved, and the safety of the user's flight, running, and operation of the moving body can be improved.

(Clause 2) In the maneuvering support system according to the first paragraph, the visual field range specifying portion is provided in a wearable object worn by the user on the head or in a moving body space in which the user is present, and the user. It can include a state detection unit that detects the static and / or dynamic state of the head of the head, and can specify the visual field range of the user based on the information obtained by the state detection unit. ..

Here, the wearable object worn by the user on the head is typically a helmet or goggles. In addition, the space inside the moving body where the user is located is typically the cockpit.
According to the maneuvering support system described in Section 2, the position, tilt angle, direction, angular velocity, etc. of the user's head can be grasped accurately and without delay, and it is appropriate according to the movement of the head. The image can be displayed in front of the user's eyes.

(Clause 3) In the maneuvering support system according to the first or second paragraph, the visual field range specifying unit includes a line-of-sight measurement unit that measures the user's line-of-sight, and includes information obtained by the line-of-sight measurement unit. Based on this, the visual field range of the user can be specified.

According to the maneuvering support system described in Section 3, the direction of the user's actual line of sight is measured. Therefore, for example, when the face is facing the front and the line of sight is directed diagonally to the left or right. However, it is possible to show the user an actual image of the outside world that is actually visible to the user and a virtual image that matches the actual image.

(Clause 4) In the maneuvering support system according to any one of paragraphs 1 to 3, the visual field range specifying unit is based on the displayable range of the display device worn by the user. The viewing range of the user can be determined.

According to the maneuvering support system described in paragraph 4, it is possible to acquire an image of an appropriate viewing range according to the displayable range of the display device restricted by the visor or the like.

(Clause 5) In the maneuvering support system according to any one of the items 1 to 4, the actual image acquisition unit constitutes the image pickup unit provided on the wearable object worn by the user on the head. It can be used as a component to be used, and the image information obtained by the imaging unit can be preferentially used for the matching process to realize high-speed processing.

The imaging unit provided on the wearable object (for example, the above display device) to be worn on the user's head coincides with the direction of the user's head, and unlike other sensors, there is no need to correct the angle of view. is there. Therefore, according to the maneuvering support system described in the fifth item, by using the image information obtained by the imaging unit for the matching process, it is possible to secure high-precision superposition with the outside world and speed up the process. ..

1 ... Control / processing unit 10 ... Symbol image creation unit 11 ... 3D map image creation unit 12 ... Image matching unit 13 ... Visual field range specification processing unit 14 ... Feature part extraction unit 15 ... Visibility index calculation unit 16 ... Disparity correction unit 17 ... Scaling unit 18 ... Display image creation processing unit 180 ... Image selection unit 181 ... Image complementing unit 182 ... Field range image extraction unit 183 ... Image superimposing unit 2A, 2B ... Head-mounted display device 20 ... Head angle / position / Angle speed and line-of-sight direction detection unit 21 ... High-sensitivity sensor unit 22 ... Display unit 23 ... Image selection / composition unit 24 ... Battery 3 ... Sensor pod 30 ... Multi-sensor unit 31 ... Infrared camera 32 ... Visible camera 33 ... Distance measurement sensor 34 ... Lighting unit 35 ... Sensor pod control unit 4 ... Flight information collection unit 5 ... GNSS / INS device 6 ... Digital map database 9 ... Connector 100 ... Helicopter

Claims (5)

A visual field range specifying part that specifies the visual field range that the user in the moving body is viewing, and
A real image acquisition unit that acquires a real image of the outside world seen from the moving body,
A 3D map image creation unit that creates a virtual image of the outside world that is assumed to be visible from the moving body based on the 3D map information near the current position of the moving body.
A display processing unit that displays the matched image superimposed on the user's field of view in a state where at least a portion corresponding to the visual field range is matched between the real image and the virtual image.
A maneuvering support system equipped with. The visual field range specifying portion is provided in the wearable object worn by the user on the head or in the moving body space in which the user is present, and detects the static and / or dynamic state of the user's head. The maneuvering support system according to claim 1, further comprising a state detection unit for specifying a visual field range of the user based on the information obtained by the state detection unit. The maneuvering support according to claim 1, wherein the visual field range specifying unit includes a line-of-sight measuring unit that measures the user's line of sight, and specifies the user's visual field range based on the information obtained by the line-of-sight measuring unit. system. The maneuvering support system according to any one of claims 1 to 3, wherein the visual field range specifying unit determines the visual field range of the user based on the displayable range of the display device worn by the user. .. The imaging unit provided on the wearable object worn by the user on the head is used as a component component of the actual image acquisition unit, and the image information obtained by the imaging unit is preferentially used for the matching process for processing. The maneuvering support system according to any one of claims 1 to 4, which realizes high speed.