JPWO2020075430A1 - Video generator and video generation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image generator capable of showing the distribution of meteorological data in a more understandable form.
The peripheral monitoring device 1 includes a captured video input unit 21, a weather information input unit 22, and a rendering unit 32. The captured image input unit 21 inputs the captured image of the camera 3. The weather information input unit 22 inputs weather information including position information in a plurality of three dimensions. The rendering unit 32 creates a weather information figure, which is a two-dimensional figure obtained by converting the outer shape of a three-dimensional shape based on a plurality of three-dimensional position information included in the weather information so as to geometrically match the captured image. Generate. The rendering unit 32 generates a composite image in which the weather information diagram is combined with the captured image.
[Selection diagram] Fig. 1

Description

The present invention mainly relates to an image generator that synthesizes a figure with a camera image.

Patent Document 1 discloses a distribution data display device that generates a graphic image in which the intensity distribution state of meteorological data is projected according to the size of an elevation angle and superimposes it on a camera image. Patent Document 1 refers to a configuration in which the distribution status by altitude is selectively displayed when the meteorological data is layered in the altitude direction.

Japanese Patent No. 6087712

The configuration of Patent Document 1 is premised on expressing the planar distribution of meteorological data when the camera is facing so as to look up at the sky. Therefore, with the display device of License Document 1, for example, the precipitation distribution according to the altitude between the rain cloud and the ground surface cannot be seen at a glance, and there is room for improvement in that it lacks a sense of reality and presence. there were.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image generator capable of showing the distribution of meteorological data in a more understandable form.

Means and effects to solve problems

The problem to be solved by the present invention is as described above, and next, the means for solving this problem and its effect will be described.

According to the first aspect of the present invention, an image generator having the following configuration is provided. That is, the image generation device includes a captured image input unit, a weather information acquisition unit, a weather information graphic generation unit, and a composite image generation unit. The captured image input unit inputs the captured image of the camera. The meteorological information acquisition unit acquires meteorological information. The meteorological information figure generation unit is a two-dimensional figure obtained by converting the outer shape of a three-dimensional shape based on the plurality of three-dimensional position information when the meteorological information includes a plurality of three-dimensional position information. Generate a meteorological information figure. The composite image generation unit generates a composite image in which the weather information figure is combined with the captured image.

As a result, the user can easily understand the weather information together with the camera image, including the information on the outer shape in the height direction.

In the image generation device, it is preferable that the weather information figure generation unit generates the weather information figure by rendering the three-dimensional shape in two dimensions.

As a result, the weather information figure can be realized by three-dimensional computer graphics.

In the image generation device, the weather information graphic generation unit renders the virtual reality object as the three-dimensional shape arranged in the three-dimensional space in two dimensions at the position and orientation of the viewpoint corresponding to the captured image. Therefore, it is preferable to generate the weather information figure.

As a result, the weather information figure can be generated so as to be geometrically matched with the captured image.

In the image generator, it is preferable that the meteorological information includes precipitation echo information acquired by a meteorological radar.

As a result, the user can easily understand the three-dimensional outline of the distribution of precipitation with a sense of reality.

In the image generation device, it is preferable that the meteorological information graphic generation unit can generate a graphic indicating a region on the ground or water where there is precipitation based on the precipitation echo information.

This allows the user to easily understand the distribution of the above-ground / water area with precipitation.

In the image generator, the weather information preferably includes the position of the sun.

As a result, the position of the sun can be easily understood with a sense of reality.

In the image generator, it is preferable that the weather information includes a three-dimensional shape of a cloud.

As a result, the three-dimensional outline of the shape of the cloud can be easily understood with a sense of reality.

In the image generator, the meteorological information graphic generation unit can generate a graphic showing a region on the ground or water where sunlight is blocked by the cloud, based on the three-dimensional shape of the cloud and the position of the sun. It is preferable to have.

This allows the user to easily understand the information about solar radiation.

In the image generator, the meteorological information includes wind direction distribution information, wind speed distribution information, temperature distribution information, atmospheric pressure distribution information, water vapor amount distribution information, and lightning position information. , Information on the predicted distribution of solar radiation, and information on the predicted distribution of precipitation are preferably included.

This allows the user to easily understand various information about the weather.

The video generator preferably has the following configuration. That is, this image generation device includes a target information input unit for inputting target information including the position of the target on the water. The composite image generation unit synthesizes the weather information figure with the photographed image and converts the position of the target so as to be geometrically matched with the photographed image. The target information figure is a figure showing the position. It is possible to generate a composite image obtained by synthesizing the above-mentioned captured image.

As a result, not only the information on the weather but also the information on the position of the target can be grasped in an integrated manner by the image with a sense of reality.

In the image generator, it is preferable that the target information includes information obtained by detecting a target with a target monitoring radar.

This makes it possible to easily understand the monitoring result of the target by the radar.

In the image generation device, it is preferable that the target information includes information obtained from the automatic ship identification system.

This makes it possible to easily understand the monitoring result of the target by the automatic ship identification system.

The video generator preferably has the following configuration. That is, this video generation device includes a warning level acquisition unit that acquires at least the relationship between the direction and the warning level indicating the degree of caution regarding the direction based on the weather information. The composite image generation unit can synthesize an alertness chart showing the relationship between the orientation and the alertness with respect to the captured image.

This allows the user to easily understand which orientation to pay particular attention to.

The video generator preferably has the following configuration. That is, the composite video generation unit can synthesize a directional scale indicating the orientation with respect to the captured video. The vertical position where the directional scale is combined with respect to the captured image is automatically changed.

As a result, the user can easily grasp the direction by the directional scale, and can prevent the display of the directional scale from obstructing the grasp of the situation.

According to the second aspect of the present invention, the following video generation method is provided. That is, the captured image of the camera is input. Get weather information. When the weather information includes a plurality of three-dimensional position information, the outer shape of the three-dimensional shape based on the plurality of three-dimensional position information is converted so as to be geometrically matched with the captured image. Generates a weather information figure that is a two-dimensional figure. A composite image is generated by synthesizing the weather information figure with the photographed image.

As a result, the user can easily understand the weather information together with the camera image, including the information in the height direction.

The block diagram which shows the overall structure of the peripheral monitoring apparatus which concerns on one Embodiment of this invention. A side view showing various devices installed in a port monitoring facility, etc. The figure which shows the example of the photographed image input from a camera. A conceptual diagram illustrating 3D scene data constructed by arranging virtual reality objects in a 3D virtual space and a projection screen arranged in the 3D virtual space. The figure which shows the composite image which the augmented reality image generation part outputs. The flowchart explaining the process performed in the peripheral monitoring apparatus.

Next, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an overall configuration of a peripheral monitoring device 1 according to an embodiment of the present invention. FIG. 2 is a side view showing various devices provided in the port monitoring facility 4 and the like.

The peripheral monitoring device (video generator) 1 shown in FIG. 1 is installed in, for example, the port monitoring facility 4 shown in FIG. The peripheral monitoring device 1 can generate an image in which information for supporting the monitoring of the weather condition and the movement monitoring of the ship in the port is superimposed based on the image taken by the camera (photographing device) 3. The image generated by the peripheral monitoring device 1 is displayed on the display 2.

The display 2 can be configured as, for example, a display arranged at a monitoring desk where the operator performs monitoring work at the port monitoring facility 4. However, the display 2 is not limited to the above, and may be, for example, a display of a portable computer carried by an operator or an assistant thereof.

The peripheral monitoring device 1 synthesizes a surrounding image taken by a camera 3 installed in the port monitoring facility 4 and a figure that virtually represents additional display information (detailed later) regarding the surrounding situation. By doing so, a composite image which is an output image to the display 2 is generated.

Next, the camera 3 and various devices electrically connected to the peripheral monitoring device 1 will be described mainly with reference to FIG.

The camera 3 is configured as a visible light video camera that photographs the surroundings of the port monitoring facility 4. The camera 3 is configured as a wide-angle camera, and is mounted slightly downward in a high place with good visibility. The camera 3 has a live output function, and can generate moving image data (video data) as a shooting result in real time and output it to the peripheral monitoring device 1. FIG. 3 shows an example of a captured image input from the camera 3 to the peripheral monitoring device 1.

As a general rule, the camera 3 performs fixed point shooting. However, the camera 3 is attached via a rotation mechanism (not shown), and the shooting direction can be changed by inputting a signal instructing a pan / tilt operation from the peripheral monitoring device 1.

In addition to the above-mentioned camera 3, the peripheral monitoring device 1 of the present embodiment includes a weather radar 15 as various devices, a spherical camera 16, a ship monitoring radar (target monitoring radar) 12, and an AIS receiver. (Automatic identification system) It is electrically connected to 9 mag.

The meteorological radar 15 detects raindrops, rain clouds, and the like by transmitting and receiving radio waves from the antenna while changing the elevation angle and the azimuth angle. The meteorological radar 15 acquires the elevation angle and azimuth angle at which raindrops and rain clouds are present, the distance to the raindrops and rain clouds, and the amount of precipitation by performing a known calculation. The meteorological radar 15 outputs precipitation echo information, which is a three-dimensional distribution of water content according to latitude, longitude, and altitude, to the peripheral monitoring device 1. The meteorological radar 15 corresponds to a meteorological information generator.

The spherical camera 16 is composed of an upward camera equipped with a fisheye lens, and observes the entire sky at a fixed point. The omnidirectional cameras 16 are installed at three or more locations, and by analyzing the cloud images taken by each omnidirectional camera 16, the three-dimensional distribution of clouds according to latitude, longitude, and altitude. To get. The three-dimensional distribution of clouds is output to the peripheral monitoring device 1. The spherical camera 16 corresponds to a weather information generator.

There are various other possible weather information generators. For example, in the peripheral monitoring device 1, information such as temperature, atmospheric pressure, humidity, solar radiation, wind direction, and wind speed output by a meteorological observation station installed at an appropriate location in the area monitored by the camera 3 is input. be able to. In addition, wave height information output by a wave height meter installed at an appropriate location in the sea can be input to the peripheral monitoring device 1. Further, information on the amount of water vapor measured by the meteorological observation satellite by the microwave radiometer can be input to the peripheral monitoring device 1. The position of the lightning measured by the known lightning monitoring system observing the lightning discharge can also be input to the peripheral monitoring device 1.

The ship monitoring radar 12 can detect a target such as a ship existing in a port to be monitored by transmitting and receiving radio waves from an antenna. In addition, the ship monitoring radar 12 has a known target tracking function (Target Tracking, TT) capable of capturing and tracking a target, and the position and velocity vector (TT information) of the target. Can be obtained. The ship monitoring radar 12 corresponds to a target detection device.

The AIS receiver 9 receives the AIS information transmitted from the ship. AIS information includes the position (latitude / longitude) of the vessel navigating the monitored port, the length and width of the vessel, the type and identification information of the vessel, the speed, course and destination of the vessel, etc. Contains various information. The AIS receiver 9 corresponds to a target detection device.

The peripheral monitoring device 1 is connected to a keyboard 36 and a mouse 37 operated by the user. By operating the keyboard 36 and the mouse 37, the user can give various instructions regarding the generation of the image. This instruction includes a pan / tilt operation of the camera 3.

Next, the configuration of the peripheral monitoring device 1 will be described in detail mainly with reference to FIG.

As shown in FIG. 1, the peripheral monitoring device 1 includes a captured image input unit 21, a weather information input unit 22, a target information input unit 23, an additional display information acquisition unit 17, and a camera position / orientation setting unit 25. A sun position acquisition unit 26, an image recognition unit 28, and an augmented reality image generation unit 30 are provided.

Specifically, the peripheral monitoring device 1 is configured as a known computer and includes a CPU, ROM, RAM, HDD, and the like, although not shown. Further, the peripheral monitoring device 1 is provided with a GPU for performing the three-dimensional image processing described later at high speed. Then, for example, the HDD stores software for executing the video generation method of the present invention. By the cooperation of this hardware and software, the peripheral monitoring device 1 is converted into a captured video input unit 21, a weather information input unit 22, a target information input unit 23, an additional display information acquisition unit 17, and a camera position / orientation setting unit 25. , The sun position acquisition unit 26, the image recognition unit 28, the augmented reality image generation unit 30, and the like.

The captured video input unit 21 can input the video data output by the camera 3 at, for example, 30 frames per second. The captured video input unit 21 outputs the input video data to the image recognition unit 28 and the augmented reality video generation unit 30 (rendering unit 32 described later).

The weather information input unit 22 can input various weather information output by the weather radar 15 and the spherical camera 16. Therefore, the weather information input unit 22 corresponds to the weather information acquisition unit. The weather information input unit 22 outputs the input weather information to the additional display information acquisition unit 17.

The target information input unit 23 can input information related to a target such as a ship (hereinafter referred to as target information) output by the ship monitoring radar 12 and the AIS receiver 9. The target information input unit 23 corresponds to the target information acquisition unit. The target information input unit 23 outputs the input target information to the additional display information acquisition unit 17.

The additional display information acquisition unit 17 took a picture with the camera 3 based on the information input to the peripheral monitoring device 1 from the weather information input unit 22 and the target information input unit 23, the information acquired by the image recognition unit 28, and the like. Acquire information to be additionally displayed on the image (additional display information, weather information, target information). Various types of additional display information can be considered. For example, the three-dimensional distribution of precipitation obtained from the meteorological radar 15, the three-dimensional distribution of clouds obtained from the all-sky camera 16, and the sun position acquisition unit described later. The position of the sun obtained from 26, the position and speed of the ship obtained from the AIS receiver 9, the position and speed of the target obtained from the ship monitoring radar 12, and the position and speed of the target obtained from the image recognition unit 28. And so on. The additional display information acquisition unit 17 outputs the acquired information to the augmented reality image generation unit 30. The details of the additional display information will be described later.

The camera position / orientation setting unit 25 can set the position (shooting position) and orientation of the camera 3 in the port monitoring facility 4. The information on the position of the camera 3 is specifically information indicating latitude, longitude, and altitude. The information on the orientation of the camera 3 is specifically information indicating an azimuth angle and a depression angle. This information can be obtained, for example, by making measurements during the installation work of the camera 3. The orientation of the camera 3 can be changed within a predetermined angle range as described above, but when the pan / tilt operation of the camera 3 is performed, the latest orientation information is set in the camera position / orientation setting unit 25 accordingly. Will be done. The camera position / orientation setting unit 25 outputs the set information to the sun position acquisition unit 26, the additional display information acquisition unit 17, and the augmented reality image generation unit 30.

The sun position acquisition unit 26 obtains the position of the sun (azimuth and elevation) by calculation based on a known formula based on the position where the camera 3 is installed and the current time. Since the position of the sun is a kind of meteorological information, the sun position acquisition unit 26 corresponds to the meteorological information acquisition unit. The sun position acquisition unit 26 outputs the acquired position of the sun to the additional display information acquisition unit 17.

The image recognition unit 28 cuts out a part of the image acquired from the captured video input unit 21 that is considered to be a ship or the like, and collates it with a pre-registered target image database to allow the ship, diver, drifting object, etc. Recognize the target of. Specifically, the image recognition unit 28 detects a moving object by an inter-frame difference method or the like, cuts out an area where a difference occurs, and collates it with an image database. As the collation method, a known appropriate method such as template matching can be used. Image recognition can also be achieved using other publicly known methods, such as neural networks. The image recognition unit 28 outputs the information of the position recognized on the image to the additional display information acquisition unit 17 for each of the recognized targets. The image recognition unit 28 corresponds to the target information acquisition unit, similarly to the target information input unit 23.

The augmented reality image generation unit 30 generates a composite image to be displayed on the display 2. The augmented reality image generation unit 30 includes a three-dimensional scene generation unit 31 and a rendering unit 32.

As shown in FIG. 4, the three-dimensional scene generation unit 31 arranges the virtual reality objects 41v, 42v, ... Corresponding to the additional display information in the three-dimensional virtual space 40, thereby arranging the three-dimensional scene of the virtual reality. To build. As a result, 3D scene data (3D display data) 50, which is 3D scene data, is generated. The details of the 3D scene will be described later.

The rendering unit 32 draws the three-dimensional scene data 50 generated by the three-dimensional scene generation unit 31, and thereby transforms the outer dimensions of the virtual reality objects 41v, 42v, ... Into two dimensions.・ Generate. Further, the rendering unit 32 generates an image obtained by synthesizing the two-dimensional figures 41f, 42f, ... And the image captured by the camera 3 at the same time as generating the above-mentioned two-dimensional figures 41f, 42f, ... Therefore, the rendering unit 32 functions as a weather information graphic generation unit and a composite video generation unit. As shown in FIG. 5, in this composite video, figures 41f, 42f, ... Showing additional display information are superimposed on the video captured by the camera 3. The rendering unit 32 (augmented reality image generation unit 30) outputs the generated composite image to the display 2. The details of the figure generation process and the data composition process will be described later.

Next, the additional display information acquired by the additional display information acquisition unit 17 described above will be described in detail.

The additional display information is information to be additionally displayed with respect to the image captured by the camera 3, and various information can be considered depending on the purpose and function of the device connected to the peripheral monitoring device 1.

With respect to the meteorological radar 15, the three-dimensional precipitation distribution can be used as additional display information. With respect to the spherical camera 16, the distribution of clouds in three dimensions can be used as additional display information. The latest information on precipitation distribution and cloud distribution is input to the peripheral monitoring device 1 from each device.

Regarding the ship monitoring radar 12, the position and speed of the detected target can be used as additional display information. Regarding the AIS receiver 9, the received AIS information (for example, the position and orientation of the ship) can be used as additional display information. These information are input to the peripheral monitoring device 1 from each device in real time.

Further, in the present embodiment, the additional display information includes the position and speed of the target identified by the image recognition unit 28 performing image recognition.

The position and velocity vectors of the target included in the information obtained from the ship monitoring radar 12 are relative to the position and orientation of the antenna of the ship monitoring radar 12. The additional display information acquisition unit 17 sets the position and velocity vector of the target obtained from the ship monitoring radar 12 on the earth based on the position and orientation information of the antenna preset in the peripheral monitoring device 1. Convert to standard.

Similarly, the position and velocity vectors of the target obtained from the image recognition unit 28 are relative to the position and orientation of the camera 3. The additional display information acquisition unit 17 converts the position and velocity vector of the target obtained from the image recognition unit 28 into the earth reference based on the information obtained from the camera position / orientation setting unit 25.

An example of additional display information will be described below. In the camera image of FIG. 3, many clouds 41r appear in the distant sky, and these clouds 41r are detected by the spherical camera 16. Furthermore, rain 42r is falling from a part of the clouds 41r in the sky, and the precipitation based on this rain 42r is detected by the meteorological radar 15.

The additional display information regarding the weather includes at least information indicating the position (latitude and longitude) of the point where it is placed. In addition, the additional display information regarding the weather may include information indicating altitude. For example, the additional display information indicating the cloud 41r includes information indicating the position (latitude, longitude and altitude) of each point when the cloud is expressed as a collection of a large number of points. The additional display information indicating the rain 42r includes information indicating the positions (latitude, longitude and altitude) of a plurality of points included in the shape of the rain distribution.

Further, in the situation of FIG. 3, a large ship 46r is sailing toward the back right side of the camera image in the harbor, and this ship 46r has a ship monitoring radar 12, an AIS receiver 9, and an image recognition unit. Detected by 28.

The additional display information regarding a target such as a ship includes at least information indicating the position (latitude and longitude) of a point on the sea surface (water surface) on which the target is placed. For example, the additional display information indicating the ship 46r includes information indicating the position of the ship 46.

Next, the construction of the three-dimensional scene by the three-dimensional scene generation unit 31 and the composition of the video by the rendering unit 32 will be described in detail with reference to FIG. FIG. 4 shows the three-dimensional scene data 50 generated by arranging the virtual reality objects 41v, 42v, ... In the three-dimensional virtual space 40, and the projection screen 51 arranged in the three-dimensional virtual space 40. It is a conceptual diagram to explain.

The three-dimensional virtual space 40 in which the virtual reality objects 41v, 42v, ... Are arranged by the three-dimensional scene generation unit 31 is composed of an orthogonal coordinate system as shown in FIG. The origin of this Cartesian coordinate system is set at a point at altitude zero just below the installation position of the camera 3. In the three-dimensional virtual space 40, the xz plane, which is a horizontal surface including the origin, is set to simulate the sea surface (water surface) or the ground. In the example of FIG. 4, the coordinate axes are set so that the + z direction always coincides with the azimuth angle of the camera 3, the + x direction is the right direction, and the + y direction is the upward direction. Each point (coordinate) in the three-dimensional virtual space 40 is set so as to correspond to an actual position around the camera 3.

FIG. 4 shows an example in which virtual reality objects 41v, 42v, ... Are arranged in the three-dimensional virtual space 40 according to the situation of the port shown in FIG.

In this embodiment, the virtual reality object 41v for clouds is represented as a collection of small spheres. Each spherical camera 16 can acquire the presence / absence and thickness of clouds for each azimuth angle and elevation angle. The thickness of the cloud can be estimated from the color depth of the cloud in the photographed image. The three-dimensional shape of a cloud can be obtained by combining images taken simultaneously by a plurality of spherical cameras 16 and processing them three-dimensionally.

Further, the virtual real object 42v relating to rain is expressed as a three-dimensional shape whose contour is an equi-intensity surface having a plurality of stages of precipitation intensity. The meteorological radar 15 can acquire the intensity of precipitation for each direction angle and elevation angle based on the radar echo. The above-mentioned three-dimensional shape can be obtained by converting this data into a distribution of precipitation intensity in a three-dimensional lattice and performing three-dimensional interpolation as necessary. Further, in the present embodiment, a figure showing a region on the ground or water with precipitation is expressed as a polygon placed on a plane (xz plane) showing the ground or water.

A virtual reality object 43v indicating the sun is also arranged in the three-dimensional virtual space 40. The azimuth and elevation of the sun seen from the origin in the three-dimensional virtual space 40 can be easily obtained based on the azimuth of the camera 3 and the azimuth and elevation of the sun acquired by the sun position acquisition unit 26. .. The virtual reality object 43v is arranged at a position immediately in front of the projection screen 51 described later. As a result, the figure 43f, which will be described later, showing the sun can be prevented from being hidden by the camera image.

The virtual reality object 44v for a region on the ground or water where sunlight is blocked by clouds and shaded is represented as a polygon placed on a plane (xz plane) indicating the ground or water. This polygonal region can be determined by projecting a virtual reality object 41v of clouds onto the xz plane in the direction of sunlight.

The virtual reality object 45v representing the wind direction is represented as a three-dimensional figure of an arrow. The position of the virtual reality object 45v corresponds to the position of the anemometer provided in the meteorological observation station described above. The direction of the arrow indicates the direction of the wind, and the length of the arrow indicates the strength of the wind.

In the present embodiment, the virtual reality object 46v relating to the target to be monitored includes a downward cone indicating the position of the recognized target (that is, the ship 46r) and an arrow indicating the velocity vector of the target. I'm out. The downward cone represents that the target is located directly below it. The direction of the arrow represents the direction of the velocity of the target, and the length of the arrow represents the magnitude of the velocity.

The virtual reality objects 41v, 42v, ... Are arranged so as to reflect the relative position of the additional display information represented by the azimuth angle of the camera 3 with respect to the camera 3. In determining the position where these virtual reality objects 41v, 42v, ... Are arranged, a calculation is performed using the position and orientation of the camera 3 set by the camera position / orientation setting unit 25 shown in FIG. It is said.

The three-dimensional scene generation unit 31 generates the three-dimensional scene data 50 as described above. In the example of FIG. 4, the virtual reality objects 41v, 42v, ... Are arranged based on the azimuth with the origin directly below the camera 3, so that when the azimuth angle of the camera 3 changes from the state of FIG. 3, the virtual reality A new 3D scene in which the objects 41v, 42v, ... Are rearranged is constructed, and the 3D scene data 50 is updated. In addition, when the content of the additional display information is changed, for example, when the shape of the cloud 41r changes from the state shown in FIG. 3 or the ship 46r moves, the three-dimensional scene data is reflected so as to reflect the latest additional display information. 50 is updated.

Further, the rendering unit 32 arranges a projection screen 51 in the three-dimensional virtual space 40, which determines the position and range in which the image captured by the camera 3 is projected. By setting the position and orientation of the viewpoint camera 55, which will be described later, so that both the projection screen 51 and the virtual reality objects 41v, 42v, ... Are included in the field of view, it is possible to realize the composition of images. ..

The rendering unit 32 arranges the viewpoint camera 55 so as to simulate the position and orientation of the camera 3 in the real space in the three-dimensional virtual space 40. Further, the rendering unit 32 arranges the projection screen 51 so as to face the viewpoint camera 55. Regarding the simulation of the position of the camera 3, the position of the camera 3 can be obtained based on the set value of the camera position / orientation setting unit 25 shown in FIG.

The azimuth angle of the viewpoint camera 55 in the three-dimensional virtual space 40 does not change even if the azimuth angle changes due to the pan operation of the camera 3. Instead, when the camera 3 is panned, the rendering unit 32 creates virtual reality objects 41v, 42v, ... In the three-dimensional virtual space 40 by the amount of the changed azimuth in the horizontal plane centered on the y-axis. Relocate to rotate.

The depression angle of the viewpoint camera 55 is controlled so as to always be equal to the depression angle of the camera 3. The rendering unit 32 sets the position and orientation of the projection screen 51 arranged in the three-dimensional virtual space 40 in conjunction with the change in the depression angle (change in the depression angle of the viewpoint camera 55) due to the tilt operation of the camera 3 to the viewpoint camera 55. Change so that the state facing the camera is always maintained.

Then, the rendering unit 32 generates a two-dimensional image by performing a known rendering process on the tertiary scene data 50 and the projection screen 51. More specifically, the rendering unit 32 arranges the viewpoint camera 55 in the three-dimensional virtual space 40, and has the viewpoint camera 55 as the apex of the viewing frustum 56 that defines the range to be rendered. Define the line-of-sight direction as the central axis. Subsequently, the rendering unit 32 determines the apex coordinates of the polygons located inside the viewing cone 56 among the polygons constituting each object (virtual reality objects 41v, 42v, ... And the projection screen 51). By fluoroscopic projection, the coordinates are converted into the coordinates of a two-dimensional virtual screen corresponding to the display area of the composite image on the display 2. Then, based on the vertices arranged on the virtual screen, a two-dimensional image is generated by performing pixel generation / processing processing at a predetermined resolution.

In the two-dimensional image generated in this way, the figure obtained by drawing the three-dimensional scene data 50 (in other words, the figure as the rendering result of the virtual reality objects 41v, 42v, ...) ) Is included. Further, in the process of generating the secondary original image, the image captured by the camera 3 is arranged so as to be attached to the position corresponding to the projection screen 51. As a result, the image composition by the rendering unit 32 is realized.

Since the projection screen 51 has a curved shape along the spherical shell centered on the viewpoint camera 55, it is possible to prevent distortion of the captured image due to perspective projection. Further, the camera 3 is a wide-angle camera, and the photographed image has lens distortion as shown in FIG. 3, but the lens distortion is removed when the photographed image is attached to the projection screen 51. .. The method of removing the lens distortion is arbitrary, but for example, it is conceivable to use a look-up table in which the positions of the pixels before correction and the positions of the pixels after correction are associated with each other. As a result, the three-dimensional virtual space 40 shown in FIG. 4 and the captured image can be well matched.

Next, the relationship between the image captured by the camera 3 and the composite image will be described with reference to an example.

FIG. 5 shows the result of synthesizing the above-mentioned two-dimensional image by rendering the three-dimensional scene data 50 of FIG. 4 with the photographed image shown in FIG. However, in FIG. 5, the portion where the image captured by the camera 3 appears is shown by a broken line for convenience so that it can be easily distinguished from the other portions. In the composite image of FIG. 5, the figures 41f, 42f, ... Representing the additional display information are arranged so as to overlap the captured image.

The above-mentioned figures 41f, 42f, ... Are the three-dimensional shapes of the virtual reality objects 41v, 42v, ... Constituting the three-dimensional scene data 50 shown in FIG. Generated as a result of drawing. Therefore, since the geometrical consistency is maintained, even if the figures 41f, 42f, ... Are superimposed on the realistic image taken by the camera 3, the appearance is not uncomfortable. Can be done.

This creates a virtual reality cloud that floats in the sky, a precipitation distribution is formed between the rain cloud and the water surface (ground), and a mark indicating a ship appears to be floating in the air above the water surface, which is natural and real. You can get a high-quality augmented reality image. Further, by looking at the display 2, the user can comprehensively see the figures 41f, 42f, ... Representing the virtual reality, so that the necessary information can be obtained without missing.

As described above, the lens distortion is removed when the captured image input from the camera 3 is projected on the projection screen 51 of the three-dimensional virtual space 40. However, the rendering unit 32 reapplies the lens distortion to the rendered composite image by the inverse transformation using the above-mentioned look-up table. As a result, as can be seen by comparing FIG. 5 and FIG. 3, it is possible to obtain a composite image in which a sense of incongruity is less likely to occur in relation to the camera image before compositing. However, the addition of lens distortion may be omitted.

As shown in FIG. 5, the rendering unit 32 further synthesizes character information that describes information useful for understanding the situation at positions in the vicinity of the figures 41f, 42f, ... In the composite video. .. The content of the character information can be arbitrary. Examples of weather include when the weather is expected to change (for example, how many minutes later it will be sunny), the distinction between rain clouds and simple clouds, water vapor areas, snowfall areas, areas where sunshine is blocked by clouds, factories. Areas where the temperature is high in the surrounding area can be mentioned. Examples of a ship include information for identifying the ship (ship name, etc.), information indicating the size of the ship, and the like. As a result, it is possible to realize a monitoring screen with abundant information.

As shown in FIG. 5, in the present embodiment, in addition to the weather information figures 41f, 42f, ..., Which are figures representing additional display information based on the weather detection result, the ship monitoring radar 12 and the AIS receiver 9 , And the target information figure 46f, which is a figure representing additional display information based on the detection result obtained by the image recognition unit 28, is combined with the captured image. As a result, the display related to the weather and the target can be integrated, and the user can easily monitor the information obtained from various information sources with one composite video. As a result, the monitoring burden can be satisfactorily reduced.

The directional scale 48f is displayed in the composite image of FIG. The directional scale 48f is formed in an arc shape so as to connect the left edge and the right edge of the screen. On the direction scale 48f, a numerical value indicating the direction corresponding to the image is described. As a result, the user can intuitively grasp the direction in which the camera 3 is looking.

For example, when the camera 3 is tilted or the cloud 41r is moved and the azimuth scale 48f is likely to overlap with other figures 41f, 42f, ... The position where the scale 48f is combined is automatically moved in the vertical direction of the image. As a result, the directional scale 48f can be prevented from interfering with other displays.

The directional scale 48f in the composite image of FIG. 5 is obtained by arranging the virtual reality object 48v of the directional scale in the three-dimensional scene data 50 shown in FIG. 4 and rendering it by the rendering unit 32.

As shown in FIG. 5, immediately above the directional scale 48f, a warning level chart 49f is displayed for each direction, which indicates the degree of caution from the viewpoint of ship navigation safety and the like. Specifically, the additional display information acquisition unit 17 considers various factors that affect safety, such as wave strength and wind strength distribution, and sets the alertness level centered on the installation position of the camera 3. Obtained for each direction. The three-dimensional scene generation unit 31 generates a virtual reality object 49v of the alertness chart based on the information output from the additional display information acquisition unit 17, and arranges the virtual reality object 49v in the three-dimensional virtual space 40. By rendering the virtual reality object 49v by the rendering unit 32, the alertness chart 49f can be combined with the camera image as shown in FIG. Therefore, the additional display information acquisition unit 17 corresponds to the alertness acquisition unit. With this alertness chart 49f, the operator can easily notice the situation to be noted.

The figure included in the composite video is not limited to the information described above, and can be configured to show various information. For example, when it comes to weather, rain echoes on the ground, snow echoes, rain echoes inside clouds, wind direction, wind speed, temperature, barometric pressure, water vapor content, solar radiation or precipitation predictions, high temperature areas around factories, lightning. Location, snowfall area, etc. are conceivable. This information may be given as a two-dimensional distribution or a three-dimensional distribution. Further, the time series data in which the meteorological information is accumulated may be analyzed, and the obtained information may be configured to be shown in a composite video. For ships, identification numbers such as MMSI numbers included in AIS information, length and width of ships, etc. can be considered.

It is preferable that the user can set whether or not to additionally display the above information in the composite video for each type of information. As a result, the user can set, for example, a state in which only the precipitation distribution figure 42f is combined with the camera image depending on the situation. By setting to display only the required virtual reality figures, it is possible to prevent the display from becoming crowded.

FIG. 6 shows a series of processes performed by the peripheral monitoring device 1 by a flowchart.

When the flow of FIG. 6 starts, the peripheral monitoring device 1 inputs the captured image captured by the camera 3 from the captured image input unit 21 (step S101). Subsequently, the peripheral monitoring device 1 inputs various meteorological information from the meteorological radar 15 and the like (step S102). Next, the augmented reality image generation unit 30 generates a two-dimensional figure obtained by converting the outer shape of the three-dimensional shape of the weather information (step S103). At the same time, the augmented reality image generation unit 30 generates a composite image by synthesizing the obtained two-dimensional figure (weather information figure) with the captured image, and outputs the obtained composite image (step S104). ). After that, the process returns to step S101, and the processes of steps S101 to S104 are repeated.

As described above, the peripheral monitoring device 1 of the present embodiment includes a captured image input unit 21, a weather information input unit 22, and a rendering unit 32. The captured image input unit 21 inputs the captured image of the camera 3. The meteorological information input unit 22 inputs meteorological information including position information in a plurality of three dimensions. The rendering unit 32 geometrically matches the outer shape of the three-dimensional shape (virtual reality objects 41v, 42v, 43v, 44v, 45v) based on the position information in a plurality of three dimensions included in the weather information with the captured image. The weather information figures 41f, 42f, 43f, 44f, and 45f, which are the two-dimensional figures converted as described above, are generated. The rendering unit 32 generates a composite image in which the weather information figures 41f, 42f, ... Are combined with the captured image.

As a result, the operator can easily understand the weather information together with the camera image, including the information on the outer shape in the height direction.

Although the preferred embodiment of the present invention has been described above, the above configuration can be changed as follows, for example.

The rendering unit 32 simultaneously generates a two-dimensional figure based on the virtual reality objects 41v, 42v, ..., And synthesizes the two-dimensional figure and the camera image. However, these processes may be performed separately. Specifically, the augmented reality image generation unit 30 further includes a two-dimensional composition unit. The rendering unit 32 renders only the virtual reality objects 41v, 42v, ... With the projection screen 51 omitted, and generates a two-dimensional image. This two-dimensional image includes a two-dimensional figure (meteorological information figure and target information figure) in which virtual real objects 41v, 42v, ... Are converted. After that, the two-dimensional compositing unit superimposes the rendered image on the camera image. In this configuration, the rendering unit 32 corresponds to the weather information graphic generation unit, and the two-dimensional composition unit corresponds to the composite video generation unit.

The augmented reality image generation unit 30 may arrange, for example, the vector data of the coastline obtained from the nautical chart as a virtual reality object in the three-dimensional scene data 50 and render it. In this case, the figure of the coastline can be displayed on the composite video.

The augmented reality image generation unit 30 may arrange a three-dimensional shape representing a map as a virtual reality object in the three-dimensional scene data and render it. In this case, for example, a figure of a road or a building can be displayed on the composite image. It is also possible to calculate and display a figure showing the shadow cast by the building based on the above-mentioned position of the sun.

When the weather information affects the surrounding conditions (for example, the passage of vehicles on bridges, the operation of railroad vehicles, etc.), a graphic or text indicating a special warning is displayed on the composite video to call the operator's attention. You may.

Not limited to the present, past or future weather information (for example, precipitation distribution, cloud distribution, etc.) may be configured so that the operator can display it by specifying the date and time. By accumulating the acquired current meteorological information in an appropriate storage unit, past meteorological information can be obtained. Future meteorological information can be obtained by the peripheral monitoring device 1 estimating from past and present meteorological information by calculation. For example, when the operator specifies the past or future date and time using the keyboard 36, the mouse 37, or the like, the past weather information or the weather information figure showing the estimated future weather information is the current camera image. It is conceivable to configure it so that it is displayed superimposed on.

The pan / tilt function may be omitted in the camera 3 so that the shooting direction cannot be changed.

When the 3D scene generation unit 31 generates the 3D scene data 50, in the above embodiment, as described with reference to FIG. 4, the virtual reality objects 41v, 42v, ... Have the position of the camera 3 as the origin. It is arranged based on the camera orientation. However, the virtual reality objects 41v, 42v, ... may be arranged based on true north reference in which the + z direction is always true north, instead of the camera orientation reference. In this case, when the azimuth changes due to the pan operation of the camera 3, instead of rearranging the virtual reality objects 41v, 42v, ..., The position and orientation of the camera 3 change in the three-dimensional virtual space 40. By changing the azimuth angle of the viewpoint camera 55 so as to simulate and performing rendering, it is possible to obtain exactly the same rendering result as in the case of the above-mentioned camera orientation reference.

Further, in the coordinate system of the three-dimensional virtual space 40, instead of using the position directly below the camera 3 as the origin, the origin is a fixed point appropriately determined on the earth, for example, the + z direction is due north and the + x direction. May be set to be true east.

The figure representing the additional display information is not limited to the one shown in FIG. 5, and can be changed to various figures. For example, the virtual reality object 41v representing a cloud can be configured as a three-dimensional model having a complicated contour shape imitating a cloud mass instead of a collection of small spheres.

It is also possible to display a rendered figure of the three-dimensional model of the ship at the position where the ship 46r is detected. As a result, it is possible to realize a more realistic display. In the three-dimensional virtual space 40 of FIG. 4, the three-dimensional model of the ship is arranged in a direction that matches the direction of the ship obtained from AIS information or the like. The size of the three-dimensional model of the ship arranged in the three-dimensional virtual space 40 may be changed according to the size of the ship obtained from AIS information or the like.

At least one of the ship monitoring radar 12 and the AIS receiver 9 does not have to be connected to the peripheral monitoring device 1. Further, the image recognition unit 28 may be omitted from the peripheral monitoring device 1.

The peripheral monitoring device 1 is not limited to facilities on the ground, and may be provided on a moving body such as a ship.

1 Peripheral monitoring device (video generator)
3 Camera 12 Ship monitoring radar (target monitoring radar)
15 Meteorological radar 21 Captured video input section 22 Meteorological information input section (weather information acquisition section)
26 Sun position acquisition department (weather information acquisition department)
32 Rendering unit (weather information graphic generation unit, composite video generation unit)
41f, 42f, 43f, 44f, 45f Meteorological information figure 46f Target information figure

the term

Not all objectives or effects / advantages can be achieved in accordance with any particular embodiment described herein. Thus, for example, those skilled in the art will appreciate that certain embodiments will be taught herein without necessarily achieving other objectives or effects / advantages as taught or suggested herein. You will conclude that it may be configured to work to achieve or optimize one or more effects / benefits.

All processes described herein can be embodied by software code modules executed by a computing system that includes one or more computers or processors and can be fully automated. The code module can be stored on any type of non-temporary computer-readable medium or other computer storage device. Some or all methods may be embodied in dedicated computer hardware.

It is clear from the present disclosure that there are many other variations other than those described herein. For example, depending on the embodiment, any particular action, event, or function of any of the algorithms described herein may be performed in different sequences and may be added, merged, or excluded altogether. (For example, not all described actions or events are required to execute the algorithm). Further, in certain embodiments, operations or events are performed in parallel rather than sequentially, for example through multithreading, interrupt handling, or through multiple processors or processor cores, or on other parallel architectures. Can be done. In addition, different tasks or processes can be performed by different machines and / or computing systems that can work together.

The various exemplary logical blocks and modules described in connection with the embodiments disclosed herein can be implemented or executed by a machine such as a processor. The processor may be a microprocessor, but instead, the processor may be a controller, a microcontroller, or a state machine, or a combination thereof. The processor can include an electrical circuit configured to process computer executable instructions. In another embodiment, the processor includes an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable device that performs logical operations without processing computer executable instructions. Processors can also be a combination of computing devices, such as a combination of a digital signal processor (digital signal processor) and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other of that. It can be implemented as such a configuration. Although described primarily with respect to digital technology herein, the processor may also include primarily analog elements. For example, some or all of the signal processing algorithms described herein can be implemented by analog circuits or mixed analog and digital circuits. Computing environments include, but are not limited to, any type of computer system that is based on a microprocessor, mainframe computer, digital signal processor, portable computing device, device controller, or computing engine within the device. be able to.

Unless otherwise stated, conditional languages such as "can," "can," "will," or "possibly" have certain embodiments that include certain features, elements, and / or steps, but other. The embodiment is understood in the context commonly used to convey that it does not include. Thus, such conditional languages are generally any method in which features, elements and / or steps are required for one or more embodiments, or one or more embodiments are these features. , Elements and / or steps are not meant to necessarily include logic to determine whether they are included or performed in any particular embodiment.

Disjunctive languages such as the phrase "at least one of X, Y, Z" have items, terms, etc. of X, Y, Z, or any combination thereof, unless otherwise stated. Understood in the context commonly used to indicate that it can be (eg X, Y, Z). Thus, such a disjunctive language generally requires at least one of X, at least one of Y, or at least one of Z, each of which has a particular embodiment. Does not mean.

Any process description, element or block in the flow diagram described herein and / or shown in the accompanying drawings is one or more executable instructions for implementing a particular logical function or element in the process. Should be understood as potentially representing a module, segment, or part of code, including. Alternative embodiments are included within the scope of the embodiments described herein, where the element or function is substantive, depending on the functionality involved, as will be appreciated by those skilled in the art. Can be performed simultaneously or in reverse order, deleted from those illustrated or described, in no particular order.

Unless otherwise stated, a numeral such as "one" should generally be construed as containing one or more described items. Thus, terms such as "one device configured to" are intended to include one or more listed devices. One or more of these enumerated devices can also be collectively configured to perform the described citations. For example, "a processor configured to run A, B, and C below" is a first processor configured to run A and a second processor configured to run B and C. Can include processors with. In addition, even if an enumeration of specific numbers of introduced examples is explicitly enumerated, those skilled in the art will appreciate that such enumerations are typically at least the enumerated number (eg, other modifiers). A mere enumeration of "two enumerations" without is usually to be construed as meaning at least two enumerations, or two or more enumerations).

In general, the terms used herein should generally be construed as "non-limiting" terms (eg, the term "including" should be construed as "not only that, but at least including" and "~. The term "has" should be interpreted as "having at least", and the term "including" should be interpreted as "including, but not limited to,"). Those skilled in the art will judge that this is the case.

For purposes of illustration, the term "horizontal" as used herein refers to a plane parallel to or parallel to the floor or surface of the area in which the system being described is used, regardless of its orientation. The method to be done is defined as the plane on which it is carried out. The term "floor" can be replaced with the term "ground" or "water surface". The term "vertical / vertical" refers to the direction perpendicular / vertical to the defined horizon. Terms such as "upper", "lower", "lower", "upper", "side", "higher", "lower", "upper", "beyond", and "lower" are defined for the horizontal plane. ing.

The terms "attach", "connect", "pair" and other related terms used herein are removable, movable, fixed, adjustable, unless otherwise noted. And / or should be construed as including removable connections or connections. Connections / connections include direct connections and / or connections with an intermediate structure between the two components described.

Unless otherwise stated, numbers preceded by terms such as "approximately," "about," and "substantially," as used herein, include enumerated numbers, and further. Represents an amount close to the stated amount that performs the desired function or achieves the desired result. For example, "approximately," "about," and "substantially" mean values less than 10% of the stated values, unless otherwise stated. As used herein, features of embodiments in which terms such as "approximately," "about," and "substantially" are previously disclosed perform further desired functions. Or represents a feature that has some variability to achieve the desired result for that feature.

Many modifications and modifications can be added to the embodiments described above, and their elements should be understood as being among other acceptable examples. All such modifications and modifications are intended to be included within the scope of this disclosure and are protected by the following claims.

Claims (15)

A shooting video input unit that inputs the shot video of the camera,
The meteorological information acquisition department that acquires meteorological information,
Meteorological information that generates a meteorological information figure that is a two-dimensional figure obtained by converting the outer shape of a three-dimensional shape based on the plurality of three-dimensional position information when the meteorological information includes a plurality of three-dimensional position information. Figure generator and
A composite image generation unit that generates a composite image obtained by synthesizing the weather information figure with the captured image,
An image generator characterized by being equipped with. The video generator according to claim 1.
The meteorological information figure generation unit is an image generation device characterized in that the meteorological information figure is generated by rendering the three-dimensional shape in two dimensions. The video generator according to claim 2.
The meteorological information figure generation unit creates the meteorological information figure by rendering the virtual reality object as the three-dimensional shape arranged in the three-dimensional space in two dimensions at the position and orientation of the viewpoint corresponding to the captured image. An image generator characterized by generating. The video generator according to claim 1.
An image generator characterized in that the meteorological information includes precipitation echo information acquired by a meteorological radar. The video generator according to claim 4.
The meteorological information graphic generation unit is an image generation device capable of generating a graphic indicating a region on the ground or water where there is precipitation based on the precipitation echo information. The video generator according to any one of claims 1 to 5.
An image generator characterized in that the weather information includes the position of the sun. The video generator according to any one of claims 1 to 6.
An image generator characterized in that the meteorological information includes a three-dimensional shape of a cloud. The video generator according to claim 7.
The image generation unit is capable of generating a figure showing a region on the ground or water where sunlight is blocked by the cloud, based on the three-dimensional shape of the cloud and the position of the sun. Device. The video generator according to any one of claims 1 to 8.
The meteorological information includes wind direction distribution information, wind speed distribution information, temperature distribution information, atmospheric pressure distribution information, water vapor amount distribution information, lightning position information, and predicted solar radiation distribution. An image generator comprising at least one of information and information on the predicted precipitation distribution. The video generator according to any one of claims 1 to 9.
Equipped with a target information input unit for inputting target information including the position of the target on the water
The composite image generation unit is a target information graphic that is a figure that synthesizes the weather information figure with the photographed image and converts the position of the target so as to be geometrically matched with the photographed image. An image generation device characterized in that it is possible to generate a composite image obtained by synthesizing the above-mentioned captured image. The video generator according to claim 10.
An image generation device characterized in that the target information includes information obtained by detecting a target with a target monitoring radar. The video generator according to claim 10 or 11.
An image generation device characterized in that the target information includes information obtained from an automatic ship identification system. The video generator according to any one of claims 1 to 12.
It is provided with a warning level acquisition unit that acquires the relationship between the direction and the warning level indicating the degree of caution regarding the direction at least based on the weather information.
The composite video generation unit is a video generation device capable of synthesizing a warning chart showing the relationship between the direction and the warning degree with respect to the captured video. The video generator according to any one of claims 1 to 13.
The composite image generation unit can synthesize a directional scale indicating the direction with respect to the captured image.
An image generation device characterized in that the position in the vertical direction in which the azimuth scale is combined with respect to the captured image is automatically changed. Enter the image taken by the camera and
Get weather information,
When the weather information includes a plurality of three-dimensional position information, the outer shape of the three-dimensional shape based on the plurality of three-dimensional position information is converted so as to be geometrically matched with the captured image. Generate a weather information figure that is a two-dimensional figure,
A video generation method characterized by generating a composite video in which the weather information figure is combined with the captured video.

Images