WO2020075430A1 - Image generation device and image generation method - Google Patents

Abstract

[Problem] To provide an image generation device that can show the distribution of weather data in a more understandable manner. [Solution] A periphery monitoring device 1 is provided with; a photographed image input unit 21; a weather information input unit 22; and a rendering unit 32. The photographed image input unit 21 inputs a photographed image of a camera 3. The weather information input unit 22 inputs weather information which includes multiple sets of three-dimensional position information. The rendering unit 32 generates a weather information figure which is a two-dimensional figure obtained by converting the outer shape of a three-dimensional shape based on the multiple sets of three-dimensional position information included in the weather information so as to geometrically match the photographed image. The rendering unit 32 generates a synthesized image in which the weather information figure is synthesized with the photographed image.

Description

Video generation apparatus and video generation method

The present invention mainly relates to an image generation device that synthesizes a figure on a camera image.

Patent Document 1 discloses a distribution data display device that generates a graphic image in which the intensity distribution status of meteorological data is projected according to the size of the elevation angle, and superimposes it on the camera image for display. Patent Document 1 mentions a configuration in which, when the meteorological data is hierarchized in the altitude direction, the distribution status for each altitude is selectively displayed.

Japanese Patent No. 6087712

The configuration of Patent Document 1 above is premised on expressing the planar distribution of the meteorological data when the camera is oriented so as to look up at the sky. Therefore, in the display device of Patent Document 1, for example, the precipitation distribution according to the altitude between the rain cloud and the surface of the earth cannot be seen at a glance, and there is room for improvement in that it lacks realism and realism. there were.

The present invention has been made in view of the above circumstances, and an object thereof is to provide a video generation device capable of showing the distribution of meteorological data in a more easily understandable form.

Means and effects for solving the problem

The problem to be solved by the present invention is as described above. Next, the means for solving the problem and the effect thereof will be described.

According to the first aspect of the present invention, a video generation device having the following configuration is provided. That is, the video generation device includes a captured video input unit, a weather information acquisition unit, a weather information graphic generation unit, and a synthetic video generation unit. The captured image input unit inputs a captured image of the camera. The weather information acquisition unit acquires weather information. The weather information figure generation unit is a two-dimensional figure obtained by converting an outer shape of a three-dimensional shape based on the plurality of three-dimensional position information when the weather information includes a plurality of three-dimensional position information. Generate a weather information figure. The synthetic image generation unit generates a synthetic image in which the weather information graphic is synthesized with the captured image.

With this, the user can easily understand the weather information together with the camera image, including the outline information in the height direction.

In the image generation device, it is preferable that the weather information graphic generation unit generate the weather information graphic by rendering the three-dimensional shape in two dimensions.

With this, the weather information graphic can be realized by three-dimensional computer graphics.

In the image generation device, the weather information graphic generation unit two-dimensionally renders the virtual reality object as the three-dimensional shape arranged in a three-dimensional space at the position and orientation of the viewpoint corresponding to the captured image. Therefore, it is preferable to generate the weather information graphic.

With this, it is possible to generate a weather information figure so as to geometrically match the captured image.

In the image generation device, it is preferable that the weather information includes precipitation echo information acquired by a weather radar.

With this, the user can easily understand the three-dimensional contour of the distribution of precipitation with a sense of reality.

In the above-mentioned image generation device, it is preferable that the weather information graphic generation unit can generate a graphic indicating a region on the ground or above water where precipitation is based on the precipitation echo information.

With this, the user can easily understand the distribution of the ground / water area where there is precipitation.

In the above image generation device, it is preferable that the weather information includes the position of the sun.

With this, you can easily understand the position of the sun with a sense of reality.

In the above image generation device, it is preferable that the weather information includes a three-dimensional shape of a cloud.

With this, you can easily understand the three-dimensional contour of the cloud shape with a sense of reality.

In the image generation device, the meteorological information graphic generation unit can generate a graphic indicating an area on the ground or on the water where the sunlight is blocked by the cloud, based on the three-dimensional shape of the cloud and the position of the sun. Preferably there is.

With this, the user can easily understand the information about solar radiation.

In the image generation device, the weather information includes wind direction distribution information, wind velocity distribution information, temperature distribution information, atmospheric pressure distribution information, water vapor distribution information, and lightning location information. It is preferable that at least one of information on the predicted distribution of solar radiation and information on the predicted distribution of precipitation is included.

This allows the user to easily understand various weather information.

The above-mentioned video generation device preferably has the following configuration. That is, this video generation device includes a target information input unit that inputs target information including the position of the target on the water. The synthetic image generation unit synthesizes the weather information figure with the photographed image and converts the position of the target so as to geometrically match the photographed image with the target information figure which is a figure indicating the position. It is possible to generate a composite video by combining the captured video with the captured video.

With this, not only the information on the weather but also the information on the position of the target can be comprehensively grasped by the video with a sense of reality.

In the above-mentioned image generation device, it is preferable that the target information includes information obtained by detecting the target with the target monitoring radar.

With this, you can easily understand the result of target monitoring by radar.

In the above-mentioned image generation device, it is preferable that the target information includes information obtained from the automatic ship identification device.

With this, you can easily understand the result of target monitoring by the automatic ship identification device.

The above-mentioned video generation device preferably has the following configuration. That is, this video generation device includes a caution degree acquisition unit that acquires the relationship between the azimuth and the caution degree indicating the degree of caution for the azimuth, based on at least the weather information. The synthetic video generation unit can synthesize a caution level chart showing a relationship between the orientation and the caution level with the captured video.

With this, the user can easily understand which azimuth to pay attention to.

The above-mentioned video generation device preferably has the following configuration. That is, the synthetic image generation unit can synthesize an azimuth scale indicating an azimuth with the captured image. The vertical position at which the azimuth scale is combined with the captured image is automatically changed.

With this, the user can easily grasp the direction by the azimuth scale and avoid the display of the azimuth scale from interfering with the grasp of the situation.

According to the second aspect of the present invention, the following video generation method is provided. That is, the image captured by the camera is input. Get weather information. When the weather information includes a plurality of three-dimensional position information, the outer shape of the three-dimensional shape based on the plurality of three-dimensional position information is converted so as to geometrically match the captured image. Generates a weather information figure that is a two-dimensional figure. A composite image is generated by combining the weather information graphic with the captured image.

With this, the user can easily understand the weather information including the information in the height direction together with the camera image.

The block diagram which shows the whole structure of the periphery monitoring apparatus which concerns on one Embodiment of this invention. The side view which shows various equipment with which a port surveillance facility etc. are equipped. The figure which shows the example of the picked-up video input from a camera. FIG. 3 is a conceptual diagram illustrating 3D scene data constructed by arranging virtual reality objects in a 3D virtual space, and a projection screen arranged in the 3D virtual space. The figure which shows the synthetic | combination image which an augmented reality image generation part outputs. The flowchart explaining the process performed in a periphery monitoring apparatus.

Next, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the overall configuration of a perimeter monitoring device 1 according to an embodiment of the present invention. FIG. 2 is a side view showing various devices provided in the port monitoring facility 4 and the like.

The perimeter monitoring device (video generation device) 1 shown in FIG. 1 is installed in the port monitoring facility 4 shown in FIG. 2, for example. The surroundings monitoring device 1 can generate a video on which information for supporting the weather condition monitoring and the motion monitoring of a ship in a port is superimposed on the basis of the video captured by the camera (imaging device) 3. The video generated by the peripheral monitoring device 1 is displayed on the display 2.

The display 2 can be configured, for example, as a display placed on a monitoring desk where an operator performs monitoring work in the port monitoring facility 4. However, the display 2 is not limited to the above, and may be, for example, a display of a portable computer carried by an operator or its assistant.

The surroundings monitoring device 1 combines a surrounding image captured by the camera 3 installed in the harbor monitoring facility 4 and a figure that virtually represents additional display information (detailed later) regarding the surroundings. By doing so, a composite video that is an output video to the display 2 is generated.

Next, the camera 3 and various devices electrically connected to the peripheral monitoring device 1 will be described mainly with reference to FIG.

The camera 3 is configured as a visible light video camera that takes a picture around the port surveillance facility 4. The camera 3 is configured as a wide-angle type camera, and is mounted slightly downward in a place with high visibility. This camera 3 has a live output function and can generate moving image data (image data) as a shooting result in real time and output it to the surroundings monitoring device 1. FIG. 3 shows an example of a captured image input from the camera 3 to the peripheral monitoring device 1.

-Camera 3 performs fixed point shooting in principle. However, the camera 3 is attached via a rotation mechanism (not shown), and the photographing direction can be changed by inputting a signal instructing a pan / tilt operation from the peripheral monitoring device 1.

In addition to the camera 3 described above, the surroundings monitoring device 1 of the present embodiment includes a weather radar 15 as a variety of devices, a omnidirectional camera 16, a ship monitoring radar (target monitoring radar) 12, and an AIS receiver. (Automatic ship identification device) 9 and the like are electrically connected.

The weather radar 15 detects radio waves and rain clouds by transmitting and receiving radio waves from the antenna while changing the elevation angle and azimuth angle. The meteorological radar 15 obtains the elevation angle and the azimuth angle at which the raindrops and rainclouds are present, the distances to the raindrops and rain clouds, and the amount of precipitation by performing known calculations. The weather radar 15 outputs precipitation echo information, which is a three-dimensional distribution of water content according to latitude, longitude, and altitude, to the surroundings monitoring device 1. The weather radar 15 corresponds to a weather information generation device.

The omnidirectional camera 16 is composed of an upward facing camera equipped with a fish-eye lens, and observes the whole sky at a fixed point. The omnidirectional cameras 16 are installed in three or more places, and by analyzing the images of the clouds taken by each omnidirectional camera 16, the three-dimensional distribution of clouds according to latitude, longitude, and altitude. To get. The three-dimensional distribution of clouds is output to the peripheral monitoring device 1. The spherical camera 16 corresponds to a weather information generation device.

There are various other possible weather information generation devices. To give an example, the surroundings monitoring device 1 inputs information such as temperature, pressure, humidity, solar radiation, wind direction, and wind speed output from a weather observation station installed at an appropriate location in the area monitored by the camera 3. be able to. In addition, information on the height of waves output from a wave height meter installed at an appropriate place in the sea can be input to the surroundings monitoring device 1. In addition, information on the amount of water vapor measured by the weather observation satellite by the microwave radiometer can be input to the peripheral monitoring device 1. The location of the lightning measured by a known lightning monitoring system observing the lightning discharge can be input to the peripheral monitoring device 1.

The ship monitoring radar 12 can detect targets such as ships existing in the monitored port by transmitting and receiving radio waves from the antenna. In addition, the ship monitoring radar 12 has a known target tracking function (Target Tracking, TT) capable of capturing and tracking a target, and the position and velocity vector (TT information) of the target. Can be asked. The ship monitoring radar 12 corresponds to a target detection device.

The AIS receiver 9 receives the AIS information transmitted from the ship. The AIS information includes the position (latitude / longitude) of the ship navigating the monitored port, the length and width of the ship, the type and identification information of the ship, the ship speed, the course, and the destination. Various information is included. The AIS receiver 9 corresponds to a target detection device.

The peripheral monitoring device 1 is connected to a keyboard 36 and a mouse 37 operated by the user. By operating the keyboard 36 and the mouse 37, the user can give various instructions regarding image generation. This instruction includes a pan / tilt operation of the camera 3 and the like.

Next, the configuration of the peripheral monitoring device 1 will be described in detail mainly with reference to FIG.

As shown in FIG. 1, the peripheral monitoring device 1 includes a captured image input unit 21, a weather information input unit 22, a target information input unit 23, an additional display information acquisition unit 17, and a camera position / orientation setting unit 25. And a sun position acquisition unit 26, an image recognition unit 28, and an augmented reality video generation unit 30.

Specifically, the peripheral monitoring device 1 is configured as a known computer and includes a CPU, a ROM, a RAM, an HDD, and the like, which are not shown. Further, the peripheral monitoring device 1 is equipped with a GPU for performing three-dimensional image processing described later at high speed. Then, for example, the HDD stores software for executing the image generation method of the present invention. By the cooperation of the hardware and the software, the peripheral monitoring device 1 can be used for the captured image input unit 21, the weather information input unit 22, the target information input unit 23, the additional display information acquisition unit 17, the camera position / direction setting unit 25. , The sun position acquisition unit 26, the image recognition unit 28, the augmented reality video generation unit 30, and the like.

The captured video input unit 21 can input the video data output by the camera 3 at, for example, 30 frames per second. The captured video input unit 21 outputs the input video data to the image recognition unit 28 and the augmented reality video generation unit 30 (a rendering unit 32 described below).

The weather information input unit 22 can input various weather information output by the weather radar 15 and the omnidirectional camera 16. Therefore, the weather information input unit 22 corresponds to a weather information acquisition unit. The weather information input unit 22 outputs the input weather information to the additional display information acquisition unit 17.

The target information input unit 23 can input information (hereinafter referred to as target information) about targets such as ships output by the ship monitoring radar 12 and the AIS receiver 9. The target information input unit 23 corresponds to the target information acquisition unit. The target information input unit 23 outputs the input target information to the additional display information acquisition unit 17.

The additional display information acquisition unit 17 takes an image with the camera 3 based on the information input to the peripheral monitoring device 1 from the weather information input unit 22 and the target object information input unit 23, the information acquired by the image recognition unit 28, and the like. The information (additional display information, weather information, target information) to be additionally displayed on the video is acquired. Although various types of additional display information are conceivable, for example, the three-dimensional distribution of precipitation obtained from the weather radar 15, the three-dimensional distribution of clouds obtained from the omnidirectional camera 16, the sun position acquisition unit described later. Position and speed of the ship obtained from AIS receiver 9, position and speed of the ship obtained from AIS receiver 9, position and speed of the target acquired from radar for ship monitoring 12, position and speed of the target acquired from image recognition unit 28 And so on. The additional display information acquisition unit 17 outputs the acquired information to the augmented reality video generation unit 30. The details of the additional display information will be described later.

The camera position / direction setting unit 25 can set the position (shooting position) and direction of the camera 3 in the port monitoring facility 4. The information on the position of the camera 3 is specifically information indicating latitude, longitude, and altitude. The information on the orientation of the camera 3 is specifically information indicating an azimuth angle and a depression angle. These pieces of information can be obtained by, for example, performing measurements during installation work of the camera 3. The orientation of the camera 3 can be changed within a predetermined angle range as described above, but when the pan / tilt operation of the camera 3 is performed, the latest orientation information is set in the camera position / orientation setting unit 25 accordingly. To be done. The camera position / orientation setting unit 25 outputs the set information to the sun position acquisition unit 26, the additional display information acquisition unit 17, and the augmented reality video generation unit 30.

The sun position acquisition unit 26 calculates the position of the sun (azimuth and elevation) by calculation based on a known formula based on the position where the camera 3 is installed and the current time. Since the position of the sun is a type of weather information, the sun position acquisition unit 26 corresponds to the weather information acquisition unit. The sun position acquisition unit 26 outputs the acquired sun position to the additional display information acquisition unit 17.

The image recognition unit 28 cuts out a portion of the image acquired from the captured image input unit 21, which is considered to be a ship or the like, and collates it with a target image database registered in advance to detect a ship, a diver, a drifting object, or the like. Recognize the target of. Specifically, the image recognition unit 28 detects a moving object by an inter-frame difference method or the like, cuts out a region where a difference occurs, and collates it with an image database. As a matching method, a known appropriate method such as template matching can be used. Image recognition can also be realized by using another publicly known method, for example, a neural network. The image recognition unit 28 outputs the information on the position recognized on the image for each of the recognized targets to the additional display information acquisition unit 17. The image recognition unit 28, like the target information input unit 23, corresponds to the target information acquisition unit.

The augmented reality video generation unit 30 generates a composite video to be displayed on the display 2. The augmented reality video generation unit 30 includes a three-dimensional scene generation unit 31 and a rendering unit 32.

As shown in FIG. 4, the 3D scene generation unit 31 arranges the virtual reality objects 41v, 42v, ... Corresponding to the additional display information in the 3D virtual space 40 to generate a 3D scene of the virtual reality. To build. As a result, the three-dimensional scene data (three-dimensional display data) 50, which is the three-dimensional scene data, is generated. The details of the 3D scene will be described later.

The rendering unit 32 draws the three-dimensional scene data 50 generated by the three-dimensional scene generation unit 31 to convert the outer shapes of the virtual reality objects 41v, 42v, ... into two- dimensional shapes 41f, 42f, ...・ Generate. Further, the rendering unit 32 simultaneously generates the above-mentioned two-dimensional figures 41f, 42f, ... And at the same time, generates an image in which the figures 41f, 42f ,. Therefore, the rendering unit 32 functions as a weather information graphic generation unit and a composite video generation unit. As shown in FIG. 5, in this composite video, graphics 41f, 42f, ... Showing additional display information are superimposed on the video captured by the camera 3. The rendering unit 32 (augmented reality video generation unit 30) outputs the generated synthetic video to the display 2. The details of the graphic generation processing and the data synthesis processing will be described later.

Next, the additional display information acquired by the above-mentioned additional display information acquisition unit 17 will be described in detail.

The additional display information is information that is additionally displayed on the video image captured by the camera 3, and various information can be considered depending on the purpose and function of the device connected to the peripheral monitoring device 1.

Regarding the weather radar 15, the three-dimensional precipitation distribution can be used as additional display information. For the omnidirectional camera 16, the three-dimensional cloud distribution can be used as additional display information. The latest information regarding the precipitation distribution and the cloud distribution is input to the peripheral monitoring device 1 from each device.

Regarding the ship monitoring radar 12, the position and speed of the detected target can be used as additional display information. Regarding the AIS receiver 9, the above-mentioned received AIS information (for example, the position and orientation of the ship) can be used as additional display information. These pieces of information are input from the respective devices to the peripheral monitoring device 1 in real time.

Further, in the present embodiment, the additional display information includes the position and speed of the target identified by the image recognition unit 28 performing image recognition.

The position and velocity vector of the target included in the information obtained from the ship monitoring radar 12 are relative to the position and orientation of the antenna of the ship monitoring radar 12. The additional display information acquisition unit 17 determines the position and velocity vector of the target obtained from the ship monitoring radar 12 based on the position and orientation information of the antenna preset in the surroundings monitoring device 1. Convert to standard.

Similarly, the position and velocity vector of the target obtained from the image recognition unit 28 are relative to the position and orientation of the camera 3. The additional display information acquisition unit 17 converts the position and velocity vector of the target obtained from the image recognition unit 28 into the earth reference based on the information obtained from the camera position / orientation setting unit 25.

Below, an example of additional display information is explained. In the camera image of FIG. 3, many clouds 41r appear in the distant sky, but the clouds 41r are detected by the spherical camera 16. Furthermore, rain 42r is falling from a part of the clouds 41r in the sky, and the precipitation based on the rain 42r is detected by the weather radar 15.

-At least the additional display information related to weather includes information indicating the position (latitude and longitude) of the point where it is located. Further, the additional display information related to weather may include information indicating altitude. For example, the additional display information indicating the cloud 41r includes information indicating the position (latitude, longitude, and altitude) of each point when the cloud is expressed as a set of many points. The additional display information indicating the rain 42r includes information indicating the positions (latitude, longitude, and altitude) of a plurality of points included in the shape of the rain distribution.

Further, in the situation of FIG. 3, a large vessel 46r is sailing toward the inner right side of the camera image in the harbor, but the vessel 46r includes the vessel monitoring radar 12, the AIS receiver 9, and the image recognition unit. 28 is detected.

The additional display information related to the target such as a ship includes at least information indicating the position (latitude and longitude) of the point on the sea surface (water surface) on which it is placed. For example, the additional display information indicating the ship 46r includes information indicating the position of the ship 46.

Next, the construction of the 3D scene by the 3D scene generation unit 31 and the composition of the video by the rendering unit 32 will be described in detail with reference to FIG. FIG. 4 shows 3D scene data 50 generated by arranging virtual reality objects 41v, 42v, ... In the 3D virtual space 40, and a projection screen 51 arranged in the 3D virtual space 40. It is a conceptual diagram explaining.

The three-dimensional virtual space 40 in which the virtual reality objects 41v, 42v, ... Are arranged by the three-dimensional scene generation unit 31 is configured by an orthogonal coordinate system as shown in FIG. The origin of this Cartesian coordinate system is set at a point of zero altitude just below the installation position of the camera 3. In the three-dimensional virtual space 40, the xz plane, which is a horizontal plane including the origin, is set so as to simulate the sea surface (water surface) or the ground. In the example of FIG. 4, the coordinate axes are determined so that the + z direction always matches the azimuth angle of the camera 3, the + x direction is the right direction, and the + y direction is the up direction. Each point (coordinate) in this three-dimensional virtual space 40 is set so as to correspond to the actual position around the camera 3.

FIG. 4 shows an example in which virtual reality objects 41v, 42v, ... Are arranged in the three-dimensional virtual space 40 corresponding to the situation of the port shown in FIG.

In the present embodiment, the virtual reality object 41v related to the cloud is represented as a collection of small spheres. Each spherical camera 16 can acquire the presence / absence of cloud and the thickness for each azimuth and elevation. The thickness of the cloud can be estimated by the color density of the cloud of the captured image. The three-dimensional shape of the cloud can be acquired by combining the images captured by the plurality of spherical cameras 16 at the same time and processing them three-dimensionally.

Also, the virtual actual object 42v related to rain is represented as a three-dimensional shape having contours of isointensity surfaces of multiple levels of precipitation intensity. The weather radar 15 can acquire the rainfall intensity for each of the orientation angle and the elevation angle based on the radar echo. The above-mentioned three-dimensional shape can be obtained by converting this data into a distribution of precipitation intensity on a three-dimensional grid and performing three-dimensional interpolation if necessary. Further, in the present embodiment, a figure showing a region on the ground or above water where precipitation is present is expressed as a polygon placed on a plane (xz plane) showing above the ground or above water.

A virtual reality object 43v indicating the sun is also arranged in the three-dimensional virtual space 40. The azimuth and elevation of the sun viewed from the origin in the three-dimensional virtual space 40 can be easily obtained based on the azimuth of the camera 3 and the azimuth and elevation of the sun acquired by the sun position acquisition unit 26. . The virtual reality object 43v is arranged at a position immediately in front of a projection screen 51 described later. This makes it possible to prevent the later-described graphic 43f indicating the sun from being hidden by the camera image.

The virtual reality object 44v relating to the area on the ground or above the water where the sunlight is shaded by the clouds is represented as a polygon placed on a plane (xz plane) showing the ground or above the water. This polygonal area can be obtained by projecting the virtual reality object 41v of the cloud on the xz plane in the direction of sunlight.

The virtual reality object 45v representing the wind direction is represented as a three-dimensional figure of an arrow. The position of the virtual reality object 45v corresponds to the position of the anemometer of the above-mentioned meteorological observation station. The direction of the arrow indicates the wind direction, and the length of the arrow indicates the strength of the wind.

In the present embodiment, the virtual reality object 46v relating to the target to be monitored includes a downward cone indicating the position of the recognized target (that is, the vessel 46r) and an arrow indicating the velocity vector of the target. I'm out. The downward cone represents that the target is located just below it. The direction of the arrow expresses the direction of the speed of the target, and the length of the arrow expresses the magnitude of the speed.

Each virtual reality object 41v, 42v, ... Is arranged so that the relative position of the additional display information represented by the virtual reality object 41v, 42v, ... With respect to the camera 3 is reflected. In determining the positions at which these virtual reality objects 41v, 42v, ... Are arranged, calculations are performed using the positions and orientations of the cameras 3 set by the camera position / orientation setting unit 25 shown in FIG. Be seen.

The three-dimensional scene generation unit 31 generates the three-dimensional scene data 50 as described above. In the example of FIG. 4, since the virtual reality objects 41v, 42v, ... Are arranged on the basis of the azimuth with the origin directly below the camera 3, when the azimuth of the camera 3 changes from the state of FIG. A new three-dimensional scene in which objects 41v, 42v, ... Are rearranged is constructed and the three-dimensional scene data 50 is updated. In addition, when the content of the additional display information is changed due to, for example, the shape of the cloud 41r changing from the state of FIG. 3 or the ship 46r moving, the three-dimensional scene data is reflected so as to reflect the latest additional display information. 50 is updated.

Furthermore, the rendering unit 32 arranges a projection screen 51 in the three-dimensional virtual space 40, which defines a position and a range on which the image captured by the camera 3 is projected. By setting the position and orientation of the viewpoint camera 55, which will be described later, such that both the projection screen 51 and the virtual reality objects 41v, 42v, ... Are included in the field of view, it is possible to realize image synthesis. .

The rendering unit 32 arranges the viewpoint camera 55 so as to simulate the position and orientation of the camera 3 in the real space in the three-dimensional virtual space 40. The rendering unit 32 also arranges the projection screen 51 so as to face the viewpoint camera 55. Regarding the simulation of the position of the camera 3, the position of the camera 3 can be obtained based on the set value of the camera position / orientation setting unit 25 shown in FIG.

The azimuth angle of the viewpoint camera 55 in the three-dimensional virtual space 40 does not change even if the azimuth angle changes due to the pan operation of the camera 3. Instead, when the panning operation of the camera 3 is performed, the rendering unit 32 causes the virtual reality objects 41v, 42v, ... In the three-dimensional virtual space 40 to move in the horizontal plane around the y axis by the changed azimuth angle. Relocate to rotate.

The depression angle of the viewpoint camera 55 is controlled so as to be always equal to the depression angle of the camera 3. The rendering unit 32 links the position and orientation of the projection screen 51 arranged in the three-dimensional virtual space 40 in conjunction with the change in the depression angle (change in the depression angle of the viewpoint camera 55) due to the tilt operation of the camera 3, and the viewpoint camera 55. Change so as to always keep the state of facing.

Then, the rendering unit 32 generates a two-dimensional image by performing known rendering processing on the tertiary source scene data 50 and the projection screen 51. More specifically, the rendering unit 32 arranges the viewpoint camera 55 in the three-dimensional virtual space 40, sets the view frustum 56 that defines the range to be subjected to the rendering process, and sets the viewpoint camera 55 as the apex. The line-of-sight direction is defined as the central axis. Subsequently, the rendering unit 32 determines the vertex coordinates of polygons located inside the view frustum 56 among the polygons forming each object (the virtual reality objects 41v, 42v, ... And the projection screen 51). By perspective projection, the coordinates are converted into the coordinates of a two-dimensional virtual screen corresponding to the display area of the composite image on the display 2. Then, based on the vertices arranged on the virtual screen, a two-dimensional image is generated by performing pixel generation / processing processing at a predetermined resolution.

In the two-dimensional image generated in this way, a figure obtained by drawing the three-dimensional scene data 50 (in other words, a figure as a rendering result of the virtual reality objects 41v, 42v, ...). ) Is included. Further, in the process of generating the secondary image, the image captured by the camera 3 is arranged at a position corresponding to the projection screen 51. As a result, the rendering unit 32 realizes image synthesis.

Since the projection screen 51 has a curved shape along a spherical shell centered on the viewpoint camera 55, it is possible to prevent distortion of the captured image due to perspective projection. Further, the camera 3 is a wide-angle camera, and a lens distortion occurs in the captured image as shown in FIG. 3, but the lens distortion is removed when the captured image is attached to the projection screen 51. . The method for removing the lens distortion is arbitrary, but for example, it is conceivable to use a look-up table in which the positions of pixels before correction and the positions of pixels after correction are associated with each other. Thereby, the three-dimensional virtual space 40 shown in FIG. 4 and the captured image can be well matched.

Next, the relationship between the video captured by the camera 3 and the composite video will be described with reference to an example.

FIG. 5 shows the result of combining the above-described two-dimensional image by rendering the three-dimensional scene data 50 of FIG. 4 with the captured video shown in FIG. However, in FIG. 5, the part in which the image captured by the camera 3 appears is shown by a broken line for convenience so that it can be easily distinguished from other parts. In the composite video image of FIG. 5, figures 41f, 42f, ... Representing additional display information are arranged so as to overlap the captured video image.

The figures 41f, 42f, ... Show the three-dimensional shape of the virtual reality objects 41v, 42v, ..., Which compose the three-dimensional scene data 50 shown in FIG. 4, from the viewpoint of the same position and orientation as the camera 3. It is generated as a result of drawing. Therefore, the geometrical consistency is maintained, so that even if the graphics 41f, 42f, ... Are superimposed on the realistic image by the camera 3, it should not cause a sense of discomfort in appearance. You can

As a result, virtual reality clouds float in the sky, the precipitation distribution is formed between the rain clouds and the water surface (ground), and the mark indicating the ship appears to float in the air above the water surface. It is possible to obtain a sense of augmented reality video. Further, the user can comprehensively obtain the necessary information by looking at the display 2 because the figures 41f, 42f, ... Representing the virtual reality are comprehensively included in the field of view.

As described above, the lens distortion is removed from the captured image input from the camera 3 when it is projected on the projection screen 51 of the three-dimensional virtual space 40. However, the rendering unit 32 again adds lens distortion to the synthesized image after rendering by inverse conversion using the above-mentioned lookup table. As a result, as can be seen by comparing FIG. 5 and FIG. 3, it is possible to obtain a composite image that does not cause a sense of discomfort in relation to the camera image before composition. However, the application of lens distortion may be omitted.

As shown in FIG. 5, the rendering unit 32 further synthesizes character information describing useful information for understanding the situation at positions near the figures 41f, 42f, ... In the synthesized video. . The content of character information can be arbitrary. As an example of weather, the weather will change (for example, how many minutes will be fine), the distinction between rain clouds and simple clouds, areas with a lot of water vapor, snowfall areas, areas where sunlight is blocked by clouds, factory Examples include areas where the temperature is high in the surroundings. Examples of the ship include information for identifying the ship (ship name, etc.), information indicating the size of the ship, and the like. As a result, it is possible to realize a monitoring screen with abundant information.

As shown in FIG. 5, in the present embodiment, in addition to the weather information graphics 41f, 42f, ... Which are graphics representing additional display information based on the weather detection result, the ship monitoring radar 12, the AIS receiver 9 , And a target information graphic 46f, which is a graphic representing additional display information based on the detection result obtained by the image recognition unit 28, is combined with the captured video. With this, it is possible to integrate the display of the weather and the target, and the user can easily monitor the information obtained from various information sources with one combined video. As a result, the burden of monitoring can be favorably reduced.

Azimuth scale 48f is displayed in the composite image of FIG. The azimuth scale 48f is formed in an arc shape so as to connect the left end and the right end of the screen. Numerical values indicating the azimuth corresponding to the image are written on the azimuth scale 48f. This allows the user to intuitively understand the direction in which the camera 3 is looking.

For example, when the camera 3 is tilted or the cloud 41r is moved and the azimuth scale 48f is likely to overlap with other figures 41f, 42f, ... On the composite video, the rendering unit 32 determines The position where the scale 48f is combined is automatically moved in the vertical direction of the image. As a result, the azimuth scale 48f can be prevented from interfering with other displays.

The azimuth scale 48f in the composite image of FIG. 5 is obtained by arranging the virtual reality object 48v of the azimuth scale in the three-dimensional scene data 50 shown in FIG. 4 and rendering this by the rendering unit 32.

As shown in FIG. 5, just above the azimuth scale 48f, a caution level chart 49f is displayed for each azimuth indicating the degree to which one should be vigilant in terms of the safety of navigation of the ship. Specifically, the additional display information acquisition unit 17 considers various factors that affect safety such as the strength of waves and the distribution of wind strength, and sets the degree of caution around the installation position of the camera 3. Ask for each direction. The 3D scene generation unit 31 generates the virtual reality object 49v of the alertness chart based on the information output from the additional display information acquisition unit 17, and arranges it in the 3D virtual space 40. By rendering the virtual reality object 49v by the rendering unit 32, as shown in FIG. 5, the alertness chart 49f can be combined with the camera image. Therefore, the additional display information acquisition unit 17 corresponds to a warning level acquisition unit. The alert level chart 49f allows the operator to easily notice a situation that requires attention.

The figures included in the composite video are not limited to the information described above, and can be configured to show various information. For example, regarding weather, it is a rain echo on the ground, an echo of snow, an echo of rain inside a cloud, wind direction, wind speed, temperature, atmospheric pressure, water vapor amount, forecast of solar radiation or precipitation, high temperature area around the factory, lightning. The location, snowfall area, etc. can be considered. These pieces of information may be given as a two-dimensional distribution or a three-dimensional distribution. Further, the time series data accumulating the weather information may be analyzed, and the obtained information may be shown in the composite video. For vessels, the identification number such as the MMSI number included in the AIS information, and the length and width of the vessel can be considered.

It is preferable that the user can set whether or not to additionally display the above information in the composite video for each type of information. Thereby, the user can be in a state in which, for example, only the graphic 42f of the precipitation distribution is combined with the camera image depending on the situation. By setting to display only the required virtual reality figure, it is possible to prevent the display from being crowded.

FIG. 6 is a flowchart showing a series of processes performed in the peripheral monitoring device 1.

When the flow of FIG. 6 starts, the peripheral monitoring device 1 inputs the captured image captured by the camera 3 from the captured image input unit 21 (step S101). Subsequently, the peripheral monitoring device 1 inputs various weather information from the weather radar 15 and the like (step S102). Next, the augmented reality video generation unit 30 generates a two-dimensional figure obtained by converting the outer shape of the three-dimensional shape of the weather information (step S103). At the same time, the augmented reality video generation unit 30 generates a composite video by combining the obtained two-dimensional graphic (weather information graphic) with the captured video, and outputs the obtained composite video (step S104). ). After that, the process returns to step S101, and the processes of steps S101 to S104 are repeated.

As described above, the surroundings monitoring device 1 of the present embodiment includes the captured image input unit 21, the weather information input unit 22, and the rendering unit 32. The captured image input unit 21 inputs a captured image of the camera 3. The weather information input unit 22 inputs weather information including a plurality of three-dimensional position information. The rendering unit 32 geometrically matches the outer shape of the three-dimensional shape ( virtual reality objects 41v, 42v, 43v, 44v, 45v) based on the plurality of three-dimensional position information included in the weather information with the captured image. The weather information graphics 41f, 42f, 43f, 44f, 45f, which are the two-dimensional graphics converted as described above, are generated. The rendering unit 32 generates a composite video in which the weather information graphics 41f, 42f, ... Are composited with the captured video.

With this, the operator can easily understand the weather information, including the camera image, including the outline information in the height direction.

The preferred embodiment of the present invention has been described above, but the above configuration can be modified as follows, for example.

The rendering unit 32 simultaneously generates a two-dimensional figure based on the virtual reality objects 41v, 42v, ... And synthesizes the two-dimensional figure and the camera image. However, these processes may be performed separately. Specifically, the augmented reality video generation unit 30 further includes a two-dimensional synthesis unit. The rendering unit 32 renders only the virtual reality objects 41v, 42v, ... With the projection screen 51 omitted to generate a two-dimensional image. The two-dimensional image includes two-dimensional figures (meteorological information figure and target information figure) obtained by converting the virtual actual objects 41v, 42v, .... After that, the two-dimensional synthesis unit superimposes the rendering result image on the camera image. In this configuration, the rendering unit 32 corresponds to the weather information graphic generation unit, and the two-dimensional synthesis unit corresponds to the synthetic video generation unit.

The augmented reality image generation unit 30 may arrange vector data of a coastline obtained from a nautical chart, for example, in the three-dimensional scene data 50 as a virtual reality object for rendering. In this case, the figure of the coastline can be displayed on the composite image.

The augmented reality video generation unit 30 may arrange a three-dimensional shape representing a map as virtual reality objects in three-dimensional scene data for rendering. In this case, for example, a figure of a road or a building can be displayed on the composite image. Further, it is also possible to calculate and display the figure showing the shadow generated by the building based on the position of the sun described above.

When the weather information affects surrounding conditions (for example, traffic of vehicles on a bridge, operation of railway vehicles, etc.), a graphic or text indicating a special warning is displayed on the composite video to alert the operator. May be.

Not limited to the present, past or future meteorological information (for example, precipitation distribution, cloud distribution, etc.) may be configured so that the operator can display the date and time. By accumulating the acquired current weather information in an appropriate storage unit, past weather information can be obtained. The future weather information can be obtained by the surroundings monitoring device 1 estimating from past and present weather information by calculation. For example, by the operator specifying the past or future date and time using the keyboard 36, the mouse 37, etc., the weather information graphic indicating the past weather information or the future estimated weather information is displayed as the current camera image. It is conceivable that it is configured so as to be superimposed on and displayed.

The above pan / tilt function may be omitted in the camera 3, and the shooting direction may be unchangeable.

When the 3D scene generation unit 31 generates the 3D scene data 50, in the above embodiment, the virtual reality objects 41v, 42v, ... Have the position of the camera 3 as the origin, as described in FIG. It is arranged based on the camera orientation. However, the virtual reality objects 41v, 42v, ... May be arranged not on the camera orientation reference but on the true north reference in which the + z direction is always true north. In this case, when the azimuth changes due to the pan operation of the camera 3, instead of rearranging the virtual reality objects 41v, 42v, ..., The position and orientation of the camera 3 in the three-dimensional virtual space 40 are changed. By changing the azimuth angle of the viewpoint camera 55 so as to perform the rendering, it is possible to obtain the same rendering result as in the case of the camera azimuth reference described above.

Further, in the coordinate system of the three-dimensional virtual space 40, instead of using the position directly under the camera 3 as the origin, a fixed point appropriately set on the earth is used as the origin, and, for example, + z direction is true north, + x direction is May be set to be true east.

The figure representing the additional display information is not limited to the one shown in FIG. 5, and can be changed to various figures. For example, the virtual reality object 41v representing a cloud can be configured as a three-dimensional model having a complicated contour shape imitating a cloud lump instead of a collection of small spheres.

・ It is also possible to display a graphic rendering a three-dimensional model of the ship at the position where the ship 46r is detected. This makes it possible to realize a more realistic display. In the three-dimensional virtual space 40 of FIG. 4, the three-dimensional model of the ship is arranged in a direction that matches the direction of the ship obtained from the AIS information and the like. The size of the three-dimensional model of the ship arranged in the three-dimensional virtual space 40 may be changed according to the size of the ship obtained from the AIS information or the like.

At least one of the ship monitoring radar 12 and the AIS receiver 9 may not be connected to the peripheral monitoring device 1. Further, the image recognition unit 28 may be omitted from the peripheral monitoring device 1.

The peripheral monitoring device 1 is not limited to a facility on the ground, and may be provided in a moving body such as a ship.

1 Surrounding monitoring device (video generation device)
3 cameras 12 Ship surveillance radar (target surveillance radar)
15 Meteorological Radar 21 Photographed Video Input Section 22 Meteorological Information Input Section (Meteorological Information Acquisition Section)
26 Sun position acquisition unit (weather information acquisition unit)
32 Rendering unit (weather information graphic generation unit, synthetic image generation unit)
41f, 42f, 43f, 44f, 45f Weather information graphic 46f Target information graphic

the term

Not all the objectives or effects / advantages can be achieved according to any specific embodiments described in the present specification. Thus, for example, a person of ordinary skill in the art will recognize that certain embodiments are taught herein without necessarily achieving the other objects or advantages as taught or suggested herein. It will be appreciated that it may be configured to operate to achieve or optimize one or more effects or advantages.

All processes described herein may be embodied by software code modules executed by a computing system including one or more computers or processors and may be fully automated. The code modules may be stored on any type of non-transitory computer readable medium or other computer storage device. Some or all of the methods may be embodied in dedicated computer hardware.

It will be apparent from this disclosure that there are many other variations other than those described herein. For example, depending on the embodiment, particular acts, events, or functions of any of the algorithms described herein can be performed in different sequences, added, merged, or omitted altogether. (For example, not all described acts or events are required to execute an algorithm). Further, in certain embodiments, the acts or events may be executed in parallel rather than serially, eg, via multithreading, interrupt handling, or through multiple processors or processor cores, or on other parallel architectures. You can Moreover, different tasks or processes may be performed by different machines and / or computing systems that may work together.

The various exemplary logic blocks and modules described in connection with the embodiments disclosed herein can be implemented or executed by a machine such as a processor. The processor may be a microprocessor, but in the alternative, the processor may be a controller, microcontroller, or state machine, combinations thereof, or the like. The processor can include electrical circuitry configured to process computer-executable instructions. In another embodiment, the processor comprises an application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other programmable device that performs logical operations without processing computer-executable instructions. A processor may also be a combination of computing devices, such as a combination of digital signal processors (digital signal processors) and microprocessors, multiple microprocessors, one or more microprocessors in combination with DSP cores, or any other thereof. It can be implemented as such a configuration. Although described primarily herein in terms of digital technology, a processor may also include primarily analog components. For example, some or all of the signal processing algorithms described herein may be implemented by analog circuits or mixed analog and digital circuits. A computing environment includes any type of computer system including, but not limited to, a microprocessor, mainframe computer, digital signal processor, portable computing device, device controller, or computing engine-based computer system in an apparatus. be able to.

Unless otherwise stated, conditional languages such as “capable”, “capable”, “possible” or “possible” refer to particular features, elements and / or steps that a particular embodiment includes. Embodiments are understood in the sense of the context commonly used to convey the inclusion. Accordingly, such a conditional language is generally any method in which features, elements and / or steps are required for one or more embodiments, or one or more embodiments It is not meant to necessarily include logic to determine whether an element and / or step is included in or executed by any particular embodiment.

A disjunctive language such as the phrase "at least one of X, Y, and Z", unless otherwise specified, has an item, term, etc. that is any of X, Y, Z, or any combination thereof. Understood in a commonly used context to indicate that it can be (eg, X, Y, Z). Accordingly, such disjunctive languages generally require each of at least one of X, at least one of Y, or at least one of Z for which a particular embodiment is present. Does not mean.

Any process description, element or block in the flow diagrams described herein and / or shown in the accompanying drawings is one or more executable instructions for implementing a particular logical function or element in the process. Should be understood as potentially representing modules, segments, or portions of code, including. Alternate embodiments are included within the scope of the embodiments described herein, where an element or function is substantially dependent on the functionality involved, as will be appreciated by those of skill in the art. May be performed simultaneously, or in reverse order, deleted from those illustrated or described, in any order.

Numerals such as "one" should generally be construed to include one or more described items unless specifically stated. Thus, phrases such as "one device configured to" are intended to include one or more of the listed devices. Such one or more enumerated devices may also be collectively configured to perform the recited citations. For example, "a processor configured to perform A, B and C below" refers to a first processor configured to perform A and a second processor configured to perform B and C. Processor. In addition, even if a specific number enumeration of the introduced examples is explicitly recited, one of ordinary skill in the art will appreciate that such an enumeration typically includes at least the recited number (e.g., other modifiers The mere enumeration of "with two enumerations" without "is usually meant to mean at least two enumerations, or more than one enumeration."

In general, the terms used in this specification generally should be construed as "non-limiting" (eg, the term "comprising" includes "as well as including at least") The term "has" should be construed as "has at least" and the term "includes" should be construed as "including but not limited to".) The person skilled in the art will judge that

For purposes of description, the term "horizontal" as used herein, regardless of its orientation, is a plane parallel to the plane or surface of the floor of the area in which the described system is used, or description. Is defined as the plane in which the method is performed. The term "floor" can be replaced with the terms "ground" or "water surface". The term "vertical / vertical" refers to the direction vertical / vertical to the defined horizontal line. Terms such as “upper”, “lower”, “lower”, “upper”, “side”, “higher”, “lower”, “upward”, “above”, and “below” are defined with respect to the horizontal plane. ing.

As used herein, the terms "attach," "connect," "pair," and other related terms, unless otherwise noted, are removable, moveable, fixed, adjustable, And / or should be construed as including a removable connection or coupling. Connections / couplings include direct connections and / or connections having an intermediate structure between the two described components.

As used herein, the numbers preceded by terms such as “approximately”, “about”, and “substantially” include the recited numbers, and unless otherwise indicated. Represents an amount near the stated amount that performs a desired function or achieves a desired result. For example, "approximately", "about" and "substantially" refer to values less than 10% of the stated numerical value, unless expressly specified otherwise. As used herein, the features of the embodiments in which the terms "about," "about," and "substantially" are previously disclosed perform additional desirable functions. Or represents a feature with some variability in achieving that desired result for that feature.

Many variations and modifications can be made to the above-described embodiments, and those elements should be understood as being among other acceptable examples. All such modifications and variations are intended to be included within the scope of this disclosure and protected by the following claims.

Claims (15)

1. The captured image input section for inputting the captured image of the camera,
   A weather information acquisition unit that acquires weather information,
   When the meteorological information includes a plurality of three-dimensional position information, the meteorological information for generating a meteorological information figure which is a two-dimensional figure obtained by converting the outer shape of the three-dimensional shape based on the plurality of three-dimensional position information. A figure generator,
   A composite video generation unit that generates a composite video in which the weather information graphic is composited with the captured video;
   An image generation apparatus comprising:
2. The video generation device according to claim 1,
   The image generation device, wherein the weather information graphic generation unit generates the weather information graphic by rendering the three-dimensional shape in two dimensions.
3. The video generation device according to claim 2, wherein
   The meteorological information graphic generation unit renders the meteorological information graphic by two-dimensionally rendering the virtual reality object as the three-dimensional shape arranged in a three-dimensional space at the position and orientation of the viewpoint corresponding to the captured video. An image generation device characterized by generating.
4. The video generation device according to claim 1,
   The image generation device, wherein the weather information includes precipitation echo information acquired by a weather radar.
5. The video generation device according to claim 4, wherein
   The image generation device, wherein the weather information graphic generation unit is capable of generating a graphic showing a region on the ground or above water where precipitation is present based on the precipitation echo information.
6. The video generation device according to any one of claims 1 to 5,
   The image generation device, wherein the weather information includes the position of the sun.
7. The video generation device according to any one of claims 1 to 6,
   The image generation device, wherein the weather information includes a three-dimensional shape of a cloud.
8. The video generation device according to claim 7, wherein
   The weather information graphic generation unit is capable of generating a graphic showing a region on the ground or on the water where the sunlight is blocked by the cloud, based on the three-dimensional shape of the cloud and the position of the sun. apparatus.
9. The video generation device according to any one of claims 1 to 8,
   The weather information includes wind direction distribution information, wind speed distribution information, temperature distribution information, atmospheric pressure distribution information, water vapor distribution information, lightning position information, and predicted solar radiation distribution information. An image generating apparatus characterized by including at least one of information and information of predicted precipitation distribution.
10. The video generation device according to any one of claims 1 to 9,
    A target information input unit for inputting target information including the position of the target on the water is provided.
    The synthetic image generation unit synthesizes the weather information figure with the photographed image, and also a target information figure which is a figure indicating a position where the position of the target is converted so as to geometrically match the photographed image. An image generating apparatus capable of generating a combined image by combining the captured image with the captured image.
11. The video generation device according to claim 10, wherein:
    An image generating apparatus, wherein the target information includes information obtained by detecting a target by a target monitoring radar.
12. The video generation device according to claim 10 or 11, wherein
    An image generating apparatus, wherein the target information includes information obtained from an automatic ship identifying apparatus.
13. The video generation device according to any one of claims 1 to 12,
    An azimuth and an alertness indicating a degree to which the azimuth should be alerted, and a relation between the alertness and an alertness acquisition unit that acquires based on the weather information
    The image generation device, wherein the synthetic image generation unit is capable of synthesizing a caution level chart showing a relationship between the orientation and the level of caution with the captured video.
14. The video generation device according to any one of claims 1 to 13,
    The synthetic image generation unit can synthesize an azimuth scale indicating an azimuth with respect to the captured image.
    An image generation apparatus, wherein a vertical position at which the azimuth scale is combined with the captured image is automatically changed.
15. Enter the video captured by the camera,
    Get weather information,
    When the weather information includes a plurality of three-dimensional position information, the outer shape of a three-dimensional shape based on the plurality of three-dimensional position information is converted so as to geometrically match the captured image. Generate a weather information graphic that is a two-dimensional graphic,
    A method for generating an image, comprising generating the combined image by combining the weather information graphic with the captured image.

Images