JP2007241085A - Photographed image processing system and photographed image processing device, and photographed image display method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To achieve a photographed image processing system in which an image photographed from an airborne system with a common photographing device is superposed on a map in an on-ground system and displayed. <P>SOLUTION: The airborne system 100 includes a common photographing device 101 and an airframe position detecting section 103 for detecting the position of the airframe, and transmits the image photographed by the photographing section 101 and the airframe position information detected by the airframe position detecting section 103 to the on-ground system 200. In the on-ground system 200, on the basis of the image and the airframe position information transmitted from the airborne system 100, a map processing section 209 superposes the image on a prestored map, and the superposed image is displayed on a monitor display section 210. In the map processing section 209, a photographing area on the map corresponding to the photographing area of the image is obtained by making a feature point in the image correspond to a corresponding point on the map which corresponds to the feature point, and the image is deformed so as to match the obtained photographing area and superposed. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a photographic video processing system, a photographic video processing device, and a photographic video display method for superimposing and displaying videos taken by an on-board imaging device on a map of a terrestrial geographic information system.

  The conventional captured video processing system described in Patent Document 1 receives an on-board system composed of a flying body (airframe) such as a helicopter equipped with an imaging device (camera) and the like, and receives images and signals from the on-board system. And a ground system installed on the ground to be processed.

The onboard system is equipped with a camera, which is an imaging device supported by a gimbal mechanism that captures the ground from the air. The airframe has a body position detection unit that obtains current position information by receiving GPS signals from the antenna and detects the body position, and a body attitude detection unit that detects the attitude of the aircraft, that is, the elevation angle (pitch) and the roll angle. Yes.
A camera supported by the gimbal mechanism, which is an on-machine imaging device, shoots the ground and outputs the video signal as well as camera information such as the camera aperture and zoom. The camera is attached to a gimbal mechanism, and this gimbal mechanism has a camera attitude detection unit that detects the rotation angle and tilt of the camera, and outputs the values.
The output signal from the airframe attitude detection unit, the video signal of the camera, the camera information signal, and the output signal from the camera attitude detection unit are buffered in the temporary storage and are synchronized with the timing of the output signal from the airframe position detection unit. Is multiplexed and modulated by the multiplex conversion unit, converted into a digital signal by the signal conversion unit, and transmitted from the antenna having the tracking function toward the terrestrial system.

In the terrestrial system, the signal from the onboard system is received by the antenna having the tracking function, the signal conversion unit converts the signal, and through the multiple demodulation unit, the video signal, the aircraft position, the body posture, the camera posture, the camera information, etc. Retrieve the information signal. These extracted signals are signal-processed by the signal processing unit, and the video signal is stored as moving image data and still image data, and used for map processing in the map processing unit, which is the next step.
The map processing unit performs processing based on moving image data that is a shooting signal, still image data and body position, body posture, camera posture information signal, geographic information system 2D map data, and 3D terrain data. On the basis of information for three-dimensionally specifying the shooting position, a shooting range on the map of the geographic information system corresponding to the shooting range of the shot image shot by the shooting device is obtained, and is combined with the obtained shooting range. Then, the captured video is transformed and superimposed and displayed on the monitor.

JP2003-316259 (Pages 5-12, FIG. 1)

In the conventional captured video processing system of Patent Document 1, a camera supported by a gimbal mechanism is mounted as an on-board system, and a camera that detects and outputs information such as camera information and camera posture in addition to the video signal of the camera. It is necessary to have an attitude detection unit. These photographing apparatuses are dedicated devices for performing image stabilization control, and have a certain volume and mass.
When the captured image processing system is applied to an aircraft such as a helicopter for the purpose of firefighting / disaster prevention, security / patrol, etc. There was a problem that priority should be given to securing mass and space, and there was a problem that emergency dispatch was not possible because it took time to mount the photographing apparatus.

  The present invention has been made in order to solve the above-described problems, and images taken by a photographing device that does not detect and output information such as photographing information and the posture of the photographing device from an onboard system, It is an object of the present invention to obtain a photographic video processing system, a photographic video processing device, and a photographic video display method that are displayed superimposed on a map.

The photographed video processing system according to the present invention is equipped with a photographing system mounted on the aircraft and equipped with a photographing device, and a video processing system that displays the images photographed by the photographing device superimposed on a map,
The imaging system includes an imaging device, an aircraft position detection unit that detects the position of the aircraft, and a transmission unit that transmits the image captured by the imaging device and the aircraft position information detected by the aircraft position detection unit to the video processing system. Have
The video processing system displays a map processing means for superimposing the video on a map stored in advance and a video superimposed on the map by the map processing means based on the video transmitted from the imaging system and the aircraft position information. Monitor display means,
The map processing means obtains a shooting range on the map corresponding to the shooting range of the video by associating the feature point in the video with the corresponding point on the map, and the calculated shooting range. The video is deformed so as to be matched and superposed.

As described above, the present invention is equipped with a photographing system mounted on an airframe and equipped with a photographing device, and a video processing system that displays an image photographed by the photographing device on a map,
The imaging system includes an imaging device, an aircraft position detection unit that detects the position of the aircraft, and a transmission unit that transmits the image captured by the imaging device and the aircraft position information detected by the aircraft position detection unit to the video processing system. Have
The video processing system displays a map processing means for superimposing the video on a map stored in advance and a video superimposed on the map by the map processing means based on the video transmitted from the imaging system and the aircraft position information. Monitor display means,
The map processing means obtains a shooting range on the map corresponding to the shooting range of the video by associating the feature point in the video with the corresponding point on the map, and the calculated shooting range. Since the video is deformed and superimposed so as to match, the photographing system photographing device does not need to detect and output information such as photographing information and the posture of the photographing device, and therefore, using a general photographing device, The captured video can be displayed superimposed on the map.

Embodiment 1 FIG.
Embodiment 1 of the present invention will be described below with reference to the drawings.
FIG. 1 is a functional configuration diagram showing a captured video processing system according to Embodiment 1 of the present invention.
In FIG. 1, a captured video processing system is mounted on an aircraft such as a helicopter and has an onboard system 100 (onboard apparatus) (imaging system) having an imaging apparatus and the like, and signals from the onboard system 100 on the ground. It consists of a ground system 200 (ground equipment) (video processing system) that receives and processes.
The onboard system 100 includes a photographing device 101 that is a camera for photographing the ground surface from the sky, and the photographing device 101 outputs a photographed photographed image. The photographing apparatus 101 is a general photographing apparatus such as a handy camera, and does not have camera information such as a camera aperture and zoom and camera photographing angle (pan, tilt) information. The onboard system 100 further includes an airframe position detection unit 103 (aircraft position detection means) that acquires information for specifying the position of the photographing apparatus 101 three-dimensionally via the GPS signal receiving unit 102.
Further, the onboard system 100 multiplex-modulates the captured image from the imaging apparatus 101 and the body position information from the body position detection unit 103 by the multiplex modulation unit 104, converts the signal into a digital signal by the signal conversion unit 105, and performs a tracking function. Transmission is performed from the antenna (transmission means) having the unit 106 toward the ground system 200.

The terrestrial system 200 receives a signal from the onboard system 100 by an antenna having a tracking function unit 201, converts the signal by a signal conversion unit 202, and takes out signals of video information and body position information by a multiplex demodulation unit 203. In addition, a signal processing unit 204 that performs signal processing on the extracted information and a geographic information system that displays a map on the screen (including information processed by the signal processing unit 204) is video-processed and superimposed on the map data And a map processing unit 209 (map processing unit) for displaying on the monitor display unit 210 (monitor display unit).
These signals extracted by the multiplex demodulator 203 are processed by the signal processor 204, and the video signal is stored as the moving image data 206 and the still image data 205, and the next step is the map processing in the map processor 209. Used for. In addition, the information signal including the two-dimensional map data 208 of the geographic information system and the terrain data 207 obtained by converting the undulations of the ground surface is also used for the map processing in the map processing unit 209.

FIG. 2 is a schematic diagram showing a map processing unit of the photographed video processing system according to Embodiment 1 of the present invention.
In FIG. 2, the map processing unit 209 performs processing based on still image data 205, moving image data 206, which is a video signal, an information signal of the position of the aircraft, terrain data 207 of the geographic information system, and two-dimensional map data 208.
The map processing unit 209 selects several points as features in the still image data 205 and corresponds to the feature points selected from the center point of the image and the still image data on the two-dimensional map data 208 of the geographic information system. Corresponding point input unit 211 for selecting and associating corresponding points on the map, and camera posture calculating unit 212 (imaging device posture calculating means) for calculating camera posture information from the helicopter body position and corresponding points (several points) An image frame calculation unit 213 for obtaining a shooting range on the two-dimensional map data 208 of the geographic information system corresponding to the shooting range of the shot image shot by the shooting device, and the shooting range determined by the image frame calculation unit 213. The image transformation unit 214 for transforming the still image data 205 and the superposition unit 215 for superposing and displaying the transformed photographed image on the photographing range on the map.

3A and 3B are diagrams for explaining the input method of the corresponding point input unit of the photographed video processing system according to Embodiment 1 of the present invention. FIG. 3A is still image data, and FIG. 3B is map data. is there.
In FIG. 3, the center point ● of the image and the feature point ★ are shown.

FIG. 4 is a diagram for explaining the angle parameter of the image center point used in the photographic image frame calculation in the camera posture calculation unit of the photographic image processing system according to Embodiment 1 of the present invention.
In FIG. 4, the camera rotation angle φ0 and the camera tilt θ0 are shown on the coordinate axes x, y, and z with the helicopter body position as the origin. The z axis indicates the height direction.

FIG. 5 is a diagram for explaining angle parameters of image corresponding points used in shooting image frame calculation in the camera posture calculation unit of the shooting video processing system according to Embodiment 1 of the present invention.
In FIG. 5, the difference between the rotation angle and inclination from the helicopter aircraft position on the coordinate axes x, y, and z with the helicopter aircraft position as the origin, and the camera rotation angle and camera inclination obtained from the center point of the image. Δφ and Δθ are shown.

FIG. 6 is a diagram for explaining a camera posture calculation method in the camera posture calculation unit of the photographed video processing system according to Embodiment 1 of the present invention.
FIG. 7 is a diagram for explaining a camera frame plane and a camera image plane in the camera posture calculation unit of the photographed video processing system according to Embodiment 1 of the present invention.
FIG. 7 shows a relationship between a camera frame plane that is an imaging plane of the camera and a camera image frame plane that is an imaging plane of the ground surface photographed by the camera.

FIG. 8 is a diagram for explaining a camera image frame calculation unit of the photographed video processing system according to Embodiment 1 of the present invention.
FIG. 8A shows that the positions of the four image frames are calculated as relative coordinates using the position of the aircraft as the origin, and FIG. 8B shows the image frame taken from the camera tilt angle θ0 in the z-axis direction. FIG. 8C shows that the photographic image frame is rotated in the y-axis direction from the azimuth angle φ0 of the camera, and FIG. 8D shows that the photographic image frame of the camera is projected on the ground surface. It is shown that the projection plane (photographing image frame) projected on the ground surface is obtained.

Next, processing of the map processing unit 209 in FIG. 2 will be described.
In the map processing unit 209 of FIG. 2, first, the corresponding point input unit 211 selects several feature points (for example, buildings, intersections, etc.) from the still image data 205, and the image is extracted from the two-dimensional map data 208 of the geographic information system. A point on the map corresponding to the center point and the selected feature point is selected and associated.
Next, in the camera posture calculation unit 212, camera posture information (camera rotation angle, tilt (tilt), tilt), based on the information on several points associated with the corresponding point input unit 211 and the helicopter body position information. Focal length).
Further, the image frame calculation unit 213 three-dimensionally identifies the shooting position in the air based on the body position information, and calculates the shooting range (shooting image frame) of the ground surface based on the camera posture information by calculation. Perform frame calculation. The video transformation unit 214 performs video transformation of the captured video in accordance with the image frame. This video transformation is to transform the video so that the captured video becomes a quadrangle that matches the map.
Finally, the superposed unit 215 superimposes (pastes) the deformed video on the map of the geographic information system, and then displays it on the monitor display means 210 such as a CRT.

Hereinafter, the processing of each unit of the map processing unit 209 will be described in more detail.
First, the processing of the corresponding point input unit 211 will be described.
FIG. 3 shows the center points and feature points of the images of the still image data 205 and the two-dimensional map data 208 set by the corresponding point input unit 211, and corresponding points on the corresponding map. . 3A represents the center point of the image of the still image data 205, and ★ in FIG. 3A is a feature point selected from the still image data 205. FIG. The ● and ★ in the map data of FIG. 3B indicate the corresponding points on the map corresponding to the center point and feature points of the image selected in the still image data of FIG.
The corresponding point input unit 211 selects the corresponding points between the image and the map in this way, and sends the position information to the camera posture calculation unit 212.

Next, processing of the camera posture calculation unit 212 will be described.
By designating the corresponding point on the map corresponding to the center point of the image (marked with ● in FIG. 3) by the corresponding point input unit 211, the camera rotation angle and inclination are first obtained from the camera posture information.
Aircraft position information (latitude, longitude, altitude) is input to the map processing unit 209. By selecting a point corresponding to the center point of the image on the map, the three-dimensional position information (latitude) of the point is selected. , Longitude, altitude) is obtained from the two-dimensional map data 208 and the terrain data 207. Accordingly, it is possible to obtain the relative positional relationship between the machine body position and the point corresponding to the image center point.
Therefore, as shown in FIG. 4, the helicopter body position is the origin (0, 0, 0), and the coordinates of the point corresponding to the center point of the image specified on the map are (x0, y0, h). Camera posture information is calculated as coordinates.
As the camera posture information, the rotation angle (φ0) and tilt (θ0) of the camera can be obtained by (Expression 1).

  Similarly, as shown in FIG. 5, with respect to corresponding points of the two-dimensional map data 208 corresponding to the feature points selected on the still image data 205, the helicopter body position (camera position C) is set to the origin (0, 0, 0). ), The coordinates of the corresponding point B selected on the map are (x1, y1, h), the rotation angle (φ1) and the inclination (θ1) from the aircraft position of the helicopter, and the camera posture information obtained from the center point of the image Considering an xyz space where the differences between the camera rotation angle (φ0) and tilt (θ0) are Δφ and Δθ, respectively, the coordinates of each point can be expressed by the following (formula 2). .

  Here, when coordinate conversion is performed to rotate the angle φ0 around the z axis, translational coordinate conversion is performed so that the center of the image frame is the origin, and coordinate conversion is performed to rotate the angle θ0 around the x axis, Each coordinate in the x′y′z ′ space is represented by (Equation 3) as shown in FIG.

  The coordinates (x1 ″, y1 ″, z1 ″) of the map corresponding point B ″, which is the intersection of the straight line B′C ′ and the x′y ′ plane, are expressed by (Expression 4).

  By arranging (Equation 3) and (Equation 4), (Equation 5) is obtained.

  Next, consider the corresponding point β on the still image of the camera frame corresponding to the map corresponding point B in FIG. Now, when there is a corresponding point β at a position (px, py) on the still image on the width PX and the height PY (pixel), the coordinates β (δx, δy) when the image center is set as the origin is 6).

  When the frame size of the camera is a width CX and a height CY (mm), the coordinates of the corresponding point β ′ on the camera frame are (Expression 7). (The coordinate α ′ of the center point of the camera frame is (0, 0, 0))

FIG. 7 shows the relationship between the camera frame plane XY at the camera focal distance f1 (mm) away from the camera position and the x′y ′ plane (FIG. 6) away from the camera position by h / cos θ0. .
The coordinates β ″ (xf ′, yf ′) obtained by projecting the image corresponding point β ′ (xf, yf) on the camera frame plane XY onto the x′y ′ plane is expressed by (Equation 8).

  Now, the coordinates β ″ obtained by projecting the image corresponding point β ′ on the x′y ′ plane and the map corresponding point B ″ coincide with each other, and the relationship of (Equation 9) is established.

  (Expression 10) is obtained from the above (Expression 5), (Expression 8), and (Expression 9).

  In this way, the camera attitude calculation unit 212 can calculate and obtain the camera focal length f and camera angle information (rotation angle, tilt).

Next, processing of the image frame calculation unit 213 will be described.
The image frame calculation unit 213 calculates the image frame taken by the camera. This calculation method can be obtained by rotating and projecting a rectangle (image frame) in 3D coordinates as the basis of computer graphics. .
Basically, a target image frame can be obtained by performing a conversion process on the image frame of the camera based on the camera posture information and the body position information, and calculating a figure frame when projected onto the ground.
The calculation method of each coordinate in a three-dimensional coordinate is obtained using the matrix calculation method described in the following (1) (2) (3) (4).

(1) Calculation of Shooting Image Frame in Reference State First, as shown in FIG. 8A, the positions of the four image frames are calculated as relative coordinates with the position of the machine as the origin. The captured image frame is calculated as a reference position based on the focal length of the camera, camera angle information, and altitude, and four coordinates are obtained.

(2) Calculate the position after rotation of the four points with the camera tilt (y-axis) As shown in FIG. 8B, the photographic image frame is rotated in the y-axis direction from the camera tilt angle θ0. The coordinates after rotation are calculated by (Equation 11). In (Expression 11), x, y, and z are the coordinates of the four points in FIG. 8A, and x ′, y ′, and z ′ are the corresponding coordinates in FIG. 8B.

(3) Calculating the position after rotation of four points with the camera azimuth (z-axis) As shown in FIG. 8C, the photographic image frame is rotated in the z-axis direction from the camera azimuth φ0. . The coordinates after rotation are obtained by conversion using (Equation 12). In (Expression 12), x, y, and z are the coordinates of the four points in FIG. 8B, and x ′, y ′, and z ′ are the corresponding coordinates in FIG. 8C.

(4) Calculating a picture frame in which the post-rotation image frame is projected from the origin (airframe position) to the ground surface (z-axis altitude point), as shown in FIG. The projection plane (photographing image frame) is obtained by projecting to the altitude point. The projected coordinates are obtained by conversion using (Equation 13). In (Expression 13), x, y, and z are the coordinates of the four points in FIG. 8C, and x ′, y ′, and z ′ are the corresponding coordinates in FIG. 8D.

  In the following (Expression 14), a general homogeneous coordinate system [X, Y, Z, W] is obtained. Where d is the altitude above sea level.

  Next, when the general homogeneous coordinate system [X, Y, Z, W] is divided by W (z / d) and returned to the three-dimensional coordinate system, (Expression 15) is obtained.

Next, processing of the video transformation unit 214 will be described.
As described above, the video deformation unit 214 performs video deformation in accordance with the image frame obtained by the image frame calculation unit 213. This video transformation is to transform the video so that the video has a trapezoidal or diamond-like shape that matches the map.

  Finally, the superposed unit 215 superimposes (pastes) the deformed video on the map of the geographic information system, and then displays it on the monitor display unit 210 such as a CRT.

According to the first embodiment, as an on-board system, a feature point in a video and a map corresponding to the feature point in the video are captured based on a video captured by a general imaging device such as a handy camera that does not have a special mechanism. By associating the corresponding points, a shooting range on the map of the geographic information system corresponding to the shooting range of the shot video shot by the shooting device is obtained, and the shot video transformed to match the obtained shooting range Is superimposed on the shooting range on the map, the video and the map are completely matched, it is easy to check the consistency between the video information and the map information, and the target point can be easily identified.
Therefore, the photographing apparatus needs to be a dedicated camera having a camera posture detection unit that detects and outputs information such as camera information and camera posture in addition to the video signal of the camera as a camera supported by the gimbal mechanism. In addition, a general handy camera can be used.

  In addition to being able to display the image of the image frame captured by the camera superimposed on the map, it is also possible to easily display only the image frame with the video information deleted. For example, when there is a video at the time of the occurrence of a disaster, it is possible to quickly recognize where the disaster site corresponds to on the map by displaying only the image frame.

Embodiment 2. FIG.
FIG. 9 is a diagram for explaining the input method of the corresponding point input unit of the photographed video processing system according to the second embodiment of the present invention. FIG. 9 (a) is still image data, and FIG. 9 (b) is map data. is there.
In FIG. 9, the center point ● of the image and the feature points ★ and ▲ are shown.

FIG. 9 shows the center points and feature points of the images of the still image data 205 and the two-dimensional map data 208 set by the corresponding point input unit 211, and corresponding points on the map. In FIG. A center point ● of the image and feature points ★ and ▲ selected from the still image data 205 are shown. The ●, ★, and ▲ in the map data of FIG. 9B indicate the corresponding points on the map corresponding to the center point and feature points of the image selected in the still image data of FIG. 9A, respectively.
The corresponding point input unit 211 selects a plurality of corresponding points between the image (still image data) and the map (map data) in this way, and sends the position information to the camera posture calculation unit 212.
As shown in (Expression 10), if a plurality of image feature points and map corresponding points are designated, the focal lengths f1, f2,. In the camera posture calculation unit 212, an average value of a plurality of calculated focal lengths is obtained and the value is adopted.

  According to the second embodiment, when a feature point on a plurality of images and a corresponding point on the map corresponding thereto are designated, a more accurate focal length can be obtained. Thereby, it is possible to improve the accuracy of pasting the video that is superimposed on the map.

Embodiment 3 FIG.
FIG. 10 is a diagram showing a screen for finely adjusting the camera posture information calculated by the camera posture calculation unit of the photographed video processing system according to the third embodiment of the present invention.
In FIG. 10, the screen 301 includes numerical setting units for pan 302, tilt 303, and zoom 304 for manually fine-tuning camera posture information (pan, tilt, zoom) obtained by the camera posture calculation unit 212. Provided.

  On the screen 301, the camera posture information obtained by the camera posture calculation unit 212 is input with numerical values of pan 302, tilt 303, and zoom 304 by default, and each value can be changed to an arbitrary value. it can.

  According to the third embodiment, it is possible to correct a slight deviation between the corresponding points of the image and the map and to check the superimposed state on the screen, so that it is possible to realize a more accurate superimposed display.

Embodiment 4 FIG.
FIG. 11 is a conceptual diagram showing a state in which the photographing device of the photographed video processing system according to the fourth embodiment of the present invention is fixed to the airframe.
In FIG. 11, the photographing apparatus 101 is fixed to the helicopter body 400.
FIG. 12 is a diagram showing a screen image of the photographed video processing system according to Embodiment 4 of the present invention.

In FIG. 11, by fixing the photographing apparatus 101 (camera) to the helicopter body 400, the ground is always photographed from the helicopter body 400 at a constant angle.
Thus, as in the first to third embodiments, when the correspondence between the image and the map is input by the corresponding point input unit 211 and the camera posture calculation unit 212 calculates the camera posture information once, the helicopter body If the posture of 400 does not change, then, using the camera posture information, it is possible to superimpose and display images continuously photographed on the map of the geographic information system as shown in FIG. .

  According to the fourth embodiment, for example, a wide range of situations in a disaster area and accurate position information thereof can be obtained by overlaying and displaying captured images on a map over a wide range.

Embodiment 5 FIG.
FIG. 13 is a functional configuration diagram showing a photographed video processing system according to Embodiment 5 of the present invention.
In FIG. 13, reference numerals 100 to 106 and 200 to 210 are the same as those in FIG. In FIG. 13, an airframe posture detection unit 107 that detects the airframe posture is provided in the onboard system 100.
FIG. 14 is a diagram for explaining the inclination of the helicopter fuselage of the photographed video processing system according to the fifth embodiment of the present invention.
FIG. 14 shows the roll angle, the traveling direction pitch angle, and the rotation angle and inclination of the photographing apparatus when the helicopter body is inclined.

In the fifth embodiment, an on-board system 100 (on-board apparatus) that is mounted on an aircraft such as a helicopter and has a photographing device, and a ground system 200 that receives and processes signals from the on-board system 100 on the ground. (Ground equipment).
The onboard system 100 includes a photographing device 101 that is a camera for photographing the ground surface from the sky, and the photographing device 101 outputs a photographed photographed image. The photographing apparatus 101 is a general photographing apparatus such as a handy camera, and does not have camera information such as a camera aperture and zoom and camera photographing angle (pan, tilt) information.
The onboard system 100 further includes an airframe position detection unit 103 (airframe position detection means) for acquiring information for three-dimensionally specifying the position of the imaging device 101 via the GPS signal receiving unit 102, and an airframe such as a helicopter. Has a body posture detecting unit 107 (airframe posture detecting means) for three-dimensionally identifying the posture.
The onboard system 100 multiplex-modulates the captured image from the imaging apparatus 101, the position information from the body position detection unit 103, and the body posture information from the body position detection unit 107 by the multiplex modulation unit 104, and the signal conversion unit 105. Then, the signal is converted into a digital signal and transmitted from the antenna (transmission means) having the tracking function unit 106 toward the ground system 200.

The terrestrial system 200 receives a signal from the onboard system 100 by an antenna having a tracking function unit 201, converts the signal by a signal conversion unit 202, and uses a multiple demodulation unit 203 to store video information, body position information, and body posture information. Retrieve the signal.
In addition, a signal processing unit 204 that performs signal processing on the extracted information and a geographic information system that displays a map on the screen (including information processed by the signal processing unit 204) is video-processed and superimposed on the map data And a map processing unit 209) for displaying on the monitor display unit 210.
These signals including the aircraft attitude information extracted by the multiplex demodulator 203 are processed by the signal processor 204, and the video signal is stored as the moving image data 206 and the still image data 205, and the next step is map processing. Used for map processing in the unit 209. In addition, information signals including the two-dimensional map data 208 and the terrain data 207 of the geographic information system are also used for map processing in the map processing unit 209.

  As a result, as shown in FIG. 14, when shooting while the aircraft such as a helicopter is tilted and sailing, the camera angle from the ground surface is obtained in consideration of the aircraft posture (roll angle, pitch angle) of the helicopter. Can do.

  According to the fifth embodiment, since the aircraft attitude detection unit is provided in the on-board system, even if the aircraft such as a helicopter is tilted and navigates while shooting, the aircraft attitude (roll angle, pitch angle) of the helicopter is determined. It is possible to obtain a camera angle from the ground surface that is taken into consideration, and to improve the accuracy of pasting the video displayed superimposed on the map.

Embodiment 6 FIG.
FIG. 15 is a functional block diagram showing a captured video processing system according to Embodiment 6 of the present invention.
In FIG. 15, reference numerals 100 to 106 and 200 to 210 are the same as those in FIG. In FIG. 15, the onboard system 100 is provided with a signal processing unit 108, a map processing unit 113, a monitor display unit 114, still image data 109, moving image data 110, terrain data 111, and two-dimensional map data 112 similar to those on the ground system. ing.

The captured image processing system according to the sixth embodiment is mounted on an aircraft such as a helicopter, and receives an onboard system 100 (onboard apparatus) having an imaging device and the like and a signal from the onboard system 100 on the ground. It consists of the ground system 200 (ground equipment) to be processed.
The onboard system 100 includes a photographing device 101 that is a camera for photographing the ground surface from the sky, and the photographing device 101 outputs a photographed photographed image. The photographing apparatus 101 is a general photographing apparatus such as a handy camera, and does not have camera information such as a camera aperture and zoom and camera photographing angle (pan, tilt) information.
The onboard system 100 further includes an airframe position detection unit 103 (aircraft position detection means) that acquires information for specifying the position of the photographing apparatus 101 three-dimensionally via the GPS signal receiving unit 102.
Further, the onboard system 100 multiplex-modulates the captured image from the imaging apparatus 101 and the body position information from the body position detection unit 103 by the multiplex modulation unit 104, converts the signal into a digital signal by the signal conversion unit 105, and performs a tracking function. Transmission is performed from the antenna (transmission means) having the unit 106 toward the ground system 200.

Furthermore, the onboard system 100 includes a signal processing unit 108 for performing signal processing on a captured image from the imaging apparatus 101 modulated by the multiplex modulation unit 104 and the aircraft position information from the aircraft position detection unit 103, and a screen. And a geographical information system (a map processing unit 113 that performs video processing including information processed by the signal processing unit 108 and superimposes it on map data and displays it on the monitor display unit 114).
The signal processing unit 108 performs signal processing, and the video signal is stored as moving image data 110 and still image data 109, and is used for map processing in the map processing unit 113, which is the next step. In addition, information signals including the two-dimensional map data 112 and the terrain data 111 of the geographic information system are also used for map processing in the map processing unit 113.

The terrestrial system 200 receives a signal from the onboard system 100 by an antenna having a tracking function unit 201, converts the signal by a signal conversion unit 202, and takes out video information and body position information signals by a multiple demodulation unit 203. .
In addition, a signal processing unit 204 that performs signal processing on the extracted information and a geographic information system that displays a map on the screen (including information processed by the signal processing unit 204) is video-processed and superimposed on the map data And a map processing unit 209) for displaying on the monitor display unit 210.
These signals extracted by the multiplex demodulator 203 are processed by the signal processing unit 204, and the video signal is stored as the moving image data 206 and the still image data 205, and the map in the map processing unit 209, which is the next step. Used for processing. In addition, information signals including the two-dimensional map data 208 and the terrain data 207 of the geographic information system are also used for map processing in the map processing unit 209.
Thereby, helicopter crew members can also see the superimposed display of the captured video on the map of the geographic information system.

  According to the sixth embodiment, helicopter crew members can also see the superimposed display of the photographed video on the map of the geographic information system, and can recognize the situation such as the positional information of the photographed video that is the same as a person on the ground. Thus, for example, it is possible to perform efficient collaborative work such as disaster information collection work at the disaster site, lifesaving work, and recovery work.

Embodiment 7 FIG.
FIG. 16 is a functional block diagram showing a captured video processing system according to Embodiment 7 of the present invention.
In FIG. 16, the captured video processing system is mounted on an aircraft such as a helicopter, and includes an onboard system 100 (onboard apparatus) having an imaging apparatus and the like, and a ground system 200 (ground apparatus).
The onboard system 100 includes a photographing device 101 that is a camera for photographing the ground surface from the sky, and the photographing device 101 outputs a photographed photographed image. The photographing apparatus 101 is a general photographing apparatus such as a handy camera, and does not have camera information such as a camera aperture and zoom and camera photographing angle (pan, tilt) information.
The onboard system 100 further includes an airframe position detection unit 103 (aircraft position detection means) that acquires information for specifying the position of the photographing apparatus 101 three-dimensionally via the GPS signal receiving unit 102.
Further, the onboard system 100 performs signal processing on the video image taken from the photographing apparatus 101 and the machine body position information from the machine body position detection unit 103 by the signal processing unit 115, and the signal storage unit 116 (storage unit) records the recording medium. A captured image and airframe position information converted into a digital signal are stored in 401.

  On the other hand, the terrestrial system 200 displays a captured image stored in the recording medium 401, a signal reproduction unit 220 (reproduction unit) for reproducing the body position information, a signal processing unit 204 for performing signal processing, and a map on the screen. A geographic information system (a map processing unit 209 that performs video processing including information processed by the signal processing unit 204 and superimposes it on map data and displays it on the monitor display unit 210). For the processing of the map processing unit 209, still image data 205, moving image data 206, terrain data 207, and two-dimensional map data 208 are used.

  In the seventh embodiment, the captured image and the aircraft position information stored in the recording medium 401 are reproduced by the signal reproduction unit 220 after the photographing flight is completed (after the helicopter returns), and the reproduced signals are subjected to signal processing. Signal processing is performed by the unit 204, and the video signal is stored as moving image data 206 and still image data 205, and is used for map processing by the map processing unit 209 as the next step. In addition, information signals including the two-dimensional map data 208 and the terrain data 207 of the geographic information system are also used for map processing in the map processing unit 209. The processing of the map processing unit 209 is the same as that described in the first embodiment, and the monitor display unit 210 performs superimposed display of the captured video on the map.

  According to the seventh embodiment, the signal storage unit 116 is provided in the onboard system 100 and the signal reproduction unit 220 is provided in the terrestrial system 200, so that the multiple modulation unit 104, the signal conversion unit 105, and the tracking of the onboard system 100 in FIG. Even if the function unit 106, the tracking function unit 201 of the terrestrial system 200, the signal conversion unit 202, and the multiplex demodulation unit 203 are not provided, the captured video can be superimposed and displayed on the map of the geographic information system of the terrestrial system 200. .

It is a functional block diagram which shows the picked-up image processing system by Embodiment 1 of this invention. It is the schematic which shows the map processing part of the picked-up image processing system by Embodiment 1 of this invention. It is a figure explaining the input method of the corresponding point input part of the picked-up image processing system by Embodiment 1 of this invention. It is a figure explaining the angle parameter of the image center point used by the photography image frame calculation in the camera attitude | position calculation part of the imaging | photography video processing system by Embodiment 1 of this invention. It is a figure explaining the angle parameter of the image corresponding point used for the imaging | photography image frame calculation in the camera attitude | position calculation part of the imaging | photography video processing system by Embodiment 1 of this invention. It is a figure explaining the camera attitude | position calculation method in the camera attitude | position calculation part of the picked-up image processing system by Embodiment 1 of this invention. It is a figure explaining the camera frame plane and camera image frame plane in the camera attitude | position calculation part of the picked-up image processing system by Embodiment 1 of this invention. It is a figure explaining the camera frame calculation part of the picked-up image processing system by Embodiment 1 of this invention. It is a figure explaining the input method of the corresponding point input part of the picked-up image processing system by Embodiment 2 of this invention. It is a figure which shows the screen for finely adjusting the camera attitude | position information calculated by the camera attitude | position calculation part of the picked-up image processing system by Embodiment 3 of this invention. It is a conceptual diagram which shows the state which fixed the imaging device of the picked-up image processing system by Embodiment 4 of this invention to the body. It is a figure which shows the screen image of the picked-up image processing system by Embodiment 4 of this invention. It is a functional block diagram which shows the picked-up image processing system by Embodiment 5 of this invention. It is a figure explaining the inclination of the helicopter airframe of the picked-up image processing system by Embodiment 5 of this invention. It is a functional block diagram which shows the picked-up image processing system by Embodiment 6 of this invention. It is a function block diagram which shows the picked-up image processing system by Embodiment 7 of this invention.

Explanation of symbols

100 on-board system, 101 photographing device, 102 GPS signal receiver,
103 Airframe position detection unit, 104 Multiplex modulation unit, 105 Signal conversion unit,
106 tracking function unit, 107 airframe posture detection unit, 108 signal processing unit,
109 still image data, 110 video data, 111 terrain data,
112 2D map data, 113 Map processing unit, 114 Monitor display unit,
115 signal processing unit, 116 signal storage unit,
200 ground system, 201 tracking function unit, 202 signal conversion unit,
203 multiplex demodulator, 204 signal processor, 205 still image data,
206 video data, 207 terrain data, 208 two-dimensional map data,
209 Map processing unit, 210 Monitor display unit, 211 Corresponding point input unit,
212 camera posture calculation unit, 213 image frame calculation unit, 214 video transformation unit,
215 superposition unit, 220 signal reproduction unit,
301 screen, 302 pan, 303 tilt, 304 zoom,
400 Helicopter fuselage, 401 Recording media.

Claims (11)

It is equipped with a shooting system mounted on the airframe and equipped with a shooting device, and a video processing system that displays the images shot by the shooting device superimposed on a map,
The imaging system includes the imaging device, an aircraft position detection unit that detects the position of the aircraft, an image captured by the imaging device, and an aircraft position information detected by the aircraft position detection unit in the video processing system. Transmission means for transmitting,
The video processing system includes map processing means for superimposing the video on a map stored in advance based on the video and aircraft position information transmitted from the imaging system, and superimposing the video on the map by the map processing means. Monitor display means for displaying the recorded video,
The map processing means obtains a shooting range on the map corresponding to the shooting range of the video by associating a feature point in the video with a corresponding point on the map corresponding to the feature point. A photographic video processing system, wherein the video is deformed so as to match a photographic range and the superposition is performed.   The photographing system includes a body posture detection unit that acquires the posture of the body, and the posture information of the body acquired by the body posture detection unit is transmitted to the video processing system, and the video by the map processing unit The photographed image processing system according to claim 1, wherein the photographed image processing system is used for superimposing images.   The photographed image processing system according to claim 1, wherein the photographing system photographing device is fixed to the body.   The map processing means has a photographing device posture calculation means for calculating the posture of the photographing device based on the feature points in the video and the corresponding corresponding points on the map, and is calculated by the photographing device posture calculation means. 2. The captured video display method according to claim 1, wherein a shooting range on the map corresponding to the shooting range of the video is obtained based on an attitude of the shooting device.   5. The photographed image display method according to claim 4, wherein the posture of the photographing device calculated by the photographing device posture calculating means is finely adjusted by numerical input after the calculation.   The photographed video processing system according to claim 1, wherein the map processing means and the monitor display means of the video processing system are also provided in the photographing system. It is equipped with a shooting system mounted on the airframe and equipped with a shooting device, and a video processing system that displays the images shot by the shooting device superimposed on a map,
The photographing system includes the photographing apparatus, a body position detecting unit that detects the position of the body, and a storage unit that stores the image captured by the photographing apparatus and the body position information detected by the body position detecting unit. Have
The video processing system includes a playback unit that plays back the video and the aircraft position information stored by the storage unit, and the video on the map stored in advance based on the video and the machine location information played back by the playback unit. A map processing means for superimposing and a monitor display means for displaying an image superimposed on the map by the map processing means,
The map processing means obtains a shooting range on the map corresponding to the shooting range of the video by associating a feature point in the video with a corresponding point on the map corresponding to the feature point. A photographic video processing system, wherein the video is deformed so as to match a photographic range and the superposition is performed. In a photographed video processing apparatus that displays an image that is photographed by a photographing device mounted on a fuselage and that is transmitted together with the fuselage position information detected by a fuselage position detection unit on a map.
Map processing means for superimposing the video on a map stored in advance based on the transmitted video and aircraft position information,
The map processing means obtains a shooting range on the map corresponding to the shooting range of the video by associating a feature point in the video with a corresponding point on the map corresponding to the feature point. A photographic video processing apparatus, wherein the video is deformed so as to match a photographic range and the superposition is performed.   By associating the feature points in the video shot from the air with the corresponding points on the map corresponding thereto, the shooting range on the map corresponding to the shooting range of the video is obtained, and the obtained shooting range The captured image is displayed so that the image is deformed so as to match with the image, and the image is superimposed on the captured range on the map.   10. The captured image display method according to claim 9, wherein a plurality of feature points in the image and corresponding points on the map corresponding to the feature points are associated with each other. Based on the feature points in the video and the corresponding points on the map, the attitude of the imaging device is calculated. On the map, the shooting on the map corresponding to the imaging range of the video is calculated based on the calculated attitude of the imaging device. 11. The captured image display method according to claim 9, wherein a range is obtained, and an attitude of the photographing apparatus is finely adjusted by numerical input after the calculation.