JP7211621B2 - Image generation device and image generation program - Google Patents

Description

The present invention relates to an image generation device and an image generation program.

2. Description of the Related Art Conventionally, there has been known an image generation device that combines a rotating object with a camera to obtain an image of the surroundings as seen from the object (see, for example, Non-Patent Document 1).

Jonas Pfeil, Kristain Hildebrand, Carsten Gremzow, Bernd Bickel, and Marc Alexa.Throwable panoramic ball camera. In SIGGRAPH Asia 2011 Emerging Technologies, SA’11, pp.4:1-4:2

However, in such a conventional image generation device, the orientation of the camera changes depending on the degree of rotation of the object, making the line of sight unstable. Therefore, in order to obtain a stable image, it is desired to fix the viewpoint, but the moving image generated by the blurring of the viewpoint becomes difficult to watch.
Also, in the method of connecting still images, the playback part jumps to the previous image, and the movement stops for a moment, or the image jumps from one image to the next, resulting in an unnatural image that is not continuous. In some cases, so-called dropped frames may occur.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image generation apparatus and an image generation program capable of generating an easy-to-view moving image from stable images by fixing the viewpoint in the same direction.

The image generating apparatus according to the present invention captures feature points present in the image when generating a moving image in the same viewpoint direction based on the image captured by the image capturing unit attached to a moving object and the image capturing unit. and a control unit that calculates the amount of movement by matching the feature points in the image after the movement, repeats returning the image by the amount of movement, and continues the frames of the moving image.

According to the present invention, it is possible to generate a moving image in which the viewpoint is fixed in the same direction and the stable image is good.

1 is a schematic block diagram showing the configuration of an image generation device; FIG. 1 is a perspective view showing a configuration of a ball provided with an imaging section in an image generating device; FIG. 3 is a schematic block diagram showing information stored in a storage unit of the image generation device; FIG. 4 is a schematic block diagram showing the configuration of a control section of the image generation device; FIG. FIG. 10 is a diagram showing an equirectangular projection; FIG. 11 is an exploded perspective view showing the configuration of a ball provided with an imaging unit in the image generation device of the modification; 4 is a flowchart of an image generation program used in the image generation device; FIG. 2 is a conceptual diagram showing how feature points are extracted by an image generating device; FIG. 10 is a conceptual diagram showing how feature points are collated in an image generating device; FIG. 10 is a conceptual diagram showing how an image generation device estimates a posture change. FIG. 10 is a conceptual diagram showing how the image generating device corrects the posture; FIG. 12 is a schematic exploded perspective view showing the configuration of a ball with a built-in omnidirectional camera in the image generation device of the modification. FIG. 11 is an overall view of a capsule endoscope used in an image generating device of another embodiment; FIG. 2 is a schematic conceptual diagram showing how a capsule endoscope is used; FIG. 4 is a diagram showing an example of an image captured by a general capsule endoscope; FIG. 2 is a schematic conceptual diagram showing an example of images captured by two cameras; FIG. 11 is a schematic conceptual diagram showing an example of successful feature point matching; FIG. 11 is a schematic conceptual diagram showing an example in which feature point matching has failed; FIG. 16 is a diagram showing an example of an interpolated video corresponding to FIG. 15; FIG. 4 is a schematic conceptual diagram illustrating how missing frames are complemented; FIG. 10 is a schematic conceptual diagram illustrating conversion from an imperfect fisheye image to an omnidirectional image; FIG. 4 is a schematic conceptual diagram for explaining how a frame is restored by correction when an error frame occurs. FIG. 10 is a schematic conceptual diagram showing an example of an image generation program of another embodiment, and showing a state in which a meaningless image flows after an error in the existing method shown in the image on the left for comparison. The image on the right side is a schematic conceptual diagram showing the improved state, and explaining the state of the frame recovered by correction even if an error frame occurs in the frame. 4 is a flowchart of an image generation program used in the image generation device of the embodiment; In the image generation program of another embodiment, the first line from the top shows frames in which captured images are continuous, and the second line shows the step of matching the feature points after the step of extracting the feature points. Row 3 shows the frames after stabilization using the conventional image generation program for comparison, and Row 4 shows the frames after stabilization using the image generation program of this embodiment. is shown. It is a perspective view explaining the crane truck of the modification 1 of embodiment. It is a perspective view explaining the drone (unmanned aerial vehicle) of the modification 2 of other embodiment. It is a perspective view explaining the fishing rod of the modification 3 of other embodiment.

Embodiments of the present invention are illustrated in detail with reference to FIGS. 1-12. In the description, the same elements are given the same numbers, and overlapping descriptions are omitted.

FIG. 1 is a schematic block diagram illustrating the configuration of an image generation device S according to an embodiment.
The image generation device S of the embodiment includes an imaging unit 1, a PC (personal computer: control unit) 2 for control, and a camera 4 attached to an object.

Among them, as shown in FIG. 2, a ball 3 as an object is provided with an imaging unit 1 that moves with rotational motion. The imaging unit 1 has a plurality of cameras 4 (here, X=6) evenly or almost evenly on the spherical surface of the surface.
Each camera 4 is provided with a lens 5, and each lens 5 is arranged facing outward. Thereby, the camera 4 can function as an omnidirectional camera capable of photographing the entire space around the ball 3 .

That is, with the six cameras 4 provided on the ball 3 of the embodiment, the captured video is an omnidirectional video, and the omnidirectional video captures information in all directions (360° centering on the ball 3). be able to.
Therefore, the omnidirectional moving image does not cause any break in the field of view, such as the front in the moving direction or the back in the moving direction, regardless of the viewpoint in any direction. Here, six cameras 4 are used for photographing, but X may be any number of cameras as long as one omnidirectional image can be obtained.

The PC 2 in FIG. 1 also includes a communication interface unit 10, a storage unit 11 for storing information and programs captured by the camera 4, and based on images received by the communication interface unit 10, generates moving images in the same viewpoint direction. a display unit 13 including a monitor for outputting and displaying the moving image calculated by the control unit 12; and an input unit 20 such as a keyboard.

The communication interface unit 10 is connected to the network 100 to perform transmission and reception, and includes communication means capable of receiving data photographed by the camera 4 of the ball 3 .
The communication interface unit 10 is configured to be able to receive image signals sent from the moving ball 3 including rotation through wireless communication such as Bluetooth (registered trademark). Here, Bluetooth is used as an example of wireless communication, but it is not limited to this, and communication means based on other wireless communication standards can be used as long as the image sent from the ball 3 can be received. good too.

The storage unit 11 is configured with a volatile memory and a nonvolatile memory. Volatile memory includes static random access memory (SRAM) and dynamic random access memory (DRAM). In addition, BIOS, program code, system software, etc. are stored in non-volatile memory. Non-volatile memory includes read-only memory (ROM), EPROM, EEPROM, flash memory, magnetic storage media, and optical disks such as compact disks.

As shown in FIG. 3, the storage unit 11 of this embodiment stores a moving image 11a, a feature point image 11b, an equation 11c, and a posture transformation matrix 11d so that they can be called at any time.

As shown in FIG. 4 , the control unit 12 is mainly composed of a CPU (Central Processing Unit) or the like, and executes programs stored in the storage unit 11 or application programs downloaded from the network 100 . Thereby, the control unit 12 is configured to calculate the image information sent from the camera 4 and generate a moving image. The moving image is output and displayed on the display unit 13 (see FIG. 1).

The control unit 12 includes an acquisition unit 12 a that acquires moving images from the six cameras 4 . This acquisition unit 12a generates a single omnidirectional image 41 .
Further, the control unit 12 next selects n images (n is an arbitrary natural number, π/n, 6 images here, but is not related to the number of cameras 4) from this one omnidirectional image 41 . ), an extraction unit 12c for extracting feature points in low latitude portions, and an integration unit 12d for integrating n images into one overall image. Further, the control unit 12 includes a calculation unit 12e that calculates the matching degree, a generation unit 12f that estimates the posture, and an output unit 12g that outputs the generated image.

That is, when generating a moving image, the control unit 12 captures the feature points present in the image and matches them with the feature points in the image after movement, thereby calculating the amount of movement and correcting the posture. makes the video frames continuous. Here, the feature point is, for example, a portion where the change in pixel value is larger or rarely smaller than that of other portions, and indicates a portion where the difference from other portions is clear.
First, when a moving image is acquired by the acquiring unit 12a, n rotated images (rotated by π/n) are generated from one omnidirectional image 41 by the dividing unit 12b. See rotated images 42a-47a).

The extraction unit 12c extracts feature points from each frame using SIFT (Scale-Invariant Feature Transform). SIFT is an algorithm for finding feature points that are invariant to the scale and rotation of an image. This SIFT generates a DoG (Difference-of-Gaussian) image from the difference of images smoothed with various intensities, and measures the change in value in the Gaussian direction, i. Feature points are extracted based on changes in pixel values.

FIG. 5 represents spherical coordinates. Also, FIG. 6 shows omnidirectional image coordinates by the equirectangular projection method. An image represented by an equirectangular projection has the property that longitudes and latitudes form right angles and intersect at equally spaced longitudes and latitudes. However, on the spherical surface they are not evenly spaced. Therefore, when the omnidirectional image is displayed as a two-dimensional image on the display unit 13, the information is correctly displayed in the low latitude area, but the information is stretched in the longitude direction and distorted in the high latitude area. For this reason, in high latitude regions, it may be impossible to obtain feature points or accurately extract feature amounts.
Therefore, the extracting unit 12c performs feature point extraction using only the low latitude portions for all n images.
Although SIFT is used to extract feature points in this embodiment, other algorithms such as SURF and AKAZE may be used instead of SIFT to extract feature points.

Then, the integration unit 12d creates one image having feature points using the extraction results of the n rotated images. These operations are performed for frames at each time.
Further, the calculation unit 12e performs feature point matching using the feature points of the previous frame and the current frame, and obtains the posture transformation matrix R. FIG.
If the feature point is located in a portion of the periphery of the image with much distortion, the control unit 12 moves the image to a portion of low latitude, which is a portion of the periphery with less distortion in the same viewpoint direction, and then matches the feature points.
The generation unit 12f uses the posture transformation matrix R to restore the current frame image to the previous frame image. The output unit 12g outputs the image to the display unit 13 such as a monitor.

The display unit 13 outputs and displays the moving image in the same viewpoint direction generated by the control unit 12 .
The three-dimensional omnidirectional image captured by the camera 4 is displayed in real time as a two-dimensional image by the display section 13 .

At this time, assuming a three-dimensional spherical surface centered on the camera as a camera model, the omnidirectional camera configured by the camera 4 has the latitude and longitude (θ, φ) of the three-dimensional spherical surface and the all-sky image by the equirectangular projection. The relationship with spherical image coordinates (u, v) is as follows.
u=w/2π・(θ+π) (−π≦θ<π) (1)
v=h/π·(φ+π/2) (−π/2≦θ<π/2) (2)
However, w is the width of the image, and h is the height of the image.

Next, image processing of the image generation device S of the embodiment will be described along the flowchart shown in FIG. 7 and with reference to FIGS. 8 to 11. FIG.
When the image processing is started based on the data photographed by the camera 4, the image is obtained by the obtaining unit 12a in step S0. In step S1, in order to extract the feature points, the feature points present in the moving image captured by the imaging unit 1 are captured (see FIG. 8).

Here, first, a method of extracting feature points and feature amounts from an omnidirectional image will be described. As described above, in the omnidirectional image represented by the equirectangular projection, the information is correctly displayed only in the low latitude area, and the information is distorted like the image 41 in the high latitude area.
By extracting feature points in the low latitude region where information is correctly displayed, feature point matching (matching) used for posture correction can be performed.

When the region of interest from which feature points are to be extracted is at a high latitude, the acquisition unit 12a (see FIG. 4) of the control unit 12 of this embodiment rotates the region of interest in the direction of the same viewpoint forward in the moving direction by rotating the spherical surface. Move to a lower latitude region with less peripheral distortion.
That is, the control unit 12 captures and generates one omnidirectional image 41 shown in FIG. 8 at each time (original). From this omnidirectional image 41, the dividing unit 12b (see FIG. 4) generates n rotated images 42 to 47 (from π/n). The rotated images 42a to 47a are rotated by π/6 (the value of n images is arbitrarily determined), and here, 6 images (6 because π/6) are generated.
Then, the extraction unit 12c extracts feature points using only the low latitude portions with little image distortion in each of the six rotated images 42a to 47a. Furthermore, in this embodiment, the integration unit 12 d puts together the feature points obtained from these six images and uses them as the feature points of the entire image 49 . As a result, feature points that are less affected by distortion are extracted from the original image.

In this way, before extracting feature points and feature quantities, the area of interest is moved to a lower latitude area by rotating the spherical surface. That is, by rotating the coordinate system of the camera 4 by an appropriate amount around the X-axis, the high latitude area can be pseudo-moved to the low latitude.
For example, in FIG. 8, as shown in rotated images 42a to 47a, a region of interest existing at a high latitude is moved to a low latitude region with less peripheral distortion in the viewpoint direction which is the same as the moving direction. It is possible to extract feature points and feature quantities as accurately as if they exist in .

Since the feature points are extracted in a low latitude region, the coordinates of the extracted feature points are different from the coordinates of the feature points in the original image. Therefore, based on the orientation estimated by the generation unit 12f (see FIG. 4), the rotation opposite to the rotation performed to extract the feature points and feature amounts is applied to the coordinates of the feature points. As a result, the coordinates of the feature points in the original image can be corrected.

In step S2, feature point collation is performed by the calculation unit 12e (see FIG. 4) (see FIG. 9). In the feature point collation, when it is determined that the feature point is located in a portion of the periphery of the image with much distortion, the image is moved to a portion of the periphery of the image with less distortion in the same viewpoint direction. match the feature points in the image of
Here, the feature points of frame t and the feature points of frame (t+1) are collated. In the figure, a straight line extending over two frames t and (t+1) connects feature points that are presumed to be the same.
Since the feature points and feature amounts of each frame are accurately extracted from feature points that have been moved to low latitudes, the reliability of matching results is improved.

In step S3, the posture change is estimated by the calculator 12e (see FIG. 4) (see FIG. 10). Here, from the collation information obtained, the basic matrix E is decomposed into rotation and translation, and an orientation conversion matrix R for the orientation change of the camera 4 is obtained.
That is, the coordinates of the feature points extracted on the omnidirectional image are converted into normalized image coordinates. For example, the result of matching is used to estimate the fundamental matrix E between the cameras when the two frames were taken by an 8-point algorithm. It is more preferable to use RANSAC or LMeds for robust estimation.
By decomposing the fundamental matrix E into rotation and translation in this way, an orientation transformation matrix R is obtained that provides a rotation matrix and a translation vector between two frames. Therefore, the posture change of the camera 4 between two frames can be obtained from the posture transformation matrix R. FIG.

In step S4, posture correction is performed by the generator 12f (see FIG. 4) (see FIG. 11A). Here, since the attitude transformation matrix R indicating the attitude change for each frame is obtained, the attitude of an arbitrary frame is corrected as the product of these, and the attitude of all adjacent frames from frame F1 to frame Fn is calculated. Calculate change.
That is, the pose transformation matrix R is estimated between all adjacent frames starting from the initial frame to the current frame.
Then, the sum of them can be regarded as a rotation matrix representing the posture change between the initial frame and the current frame as shown in FIG. 11(b).
As a result, the current frame can be made continuous with the viewpoint of the initial frame by applying the attitude transformation matrix R(n-1) (n is the number of frames) to the current frame.

In this embodiment, we do not estimate the pose directly between the initial frame and the current frame.
There are the following reasons for this. (i) the time difference between adjacent frames is minute; Therefore, the amount of movement of the feature point is also limited to a small amount. This makes it possible to filter matching between feature points with a large amount of movement on the three-dimensional spherical surface. (ii) the time difference between adjacent frames is minute; The scene is therefore static. (iii) the time difference between adjacent frames is minute; Therefore, there is no change in how the feature points appear due to translation.

At step S5, it is determined whether or not there is a next frame. If there is a next frame in step S5 (Yes in step S5), the process returns to step S2 to continue extraction of feature points. Further, in step S5, if there is no next frame (No in step S5), the control unit 12 terminates the process.

Next, the effects of the image generation device S of this embodiment will be described.
In the image generation device S of this embodiment, when it is determined in the feature point collation step S2 that the feature point is located in a portion of the periphery of the image with much distortion, the image is moved to a portion of the periphery with less distortion in the same viewpoint direction. , the feature points in the image before movement are matched with the feature points in the image after movement.
Therefore, even if the ball 3 on which the camera 4 is mounted rotates at high speed, it is possible to perform feature point extraction using only the low latitude portions with little image distortion in each of the six rotated images 42a to 47a. can. Therefore, it is possible to improve the accuracy of the extracted feature points and feature amounts.
Adjacent frames can be connected by calculating an accurate amount of movement based on the information of feature points with high accuracy and correcting the posture. Therefore, it is possible to generate a stable moving image with smooth continuity between frames and a fixed viewpoint. In addition, the generated moving image can be a stable moving image in which the surroundings of the ball 3 that rotates and moves are all photographed and there is no hidden portion. Therefore, it can be used in various fields such as sports, medicine, and disaster sites.

In addition, in this embodiment, since the omnidirectional camera 4 that combines a plurality of lenses 5 is used, it is possible to create a break in the frame at any viewpoint in any direction, such as the front in the moving direction or the back in the moving direction. It is possible to generate a moving image with a continuous field of view.

Furthermore, in this embodiment, as shown in FIG. 8, six rotated images 42a to 47a are generated by rotating a single omnidirectional image 41 by π/6. Feature point extraction is performed using only low-latitude portions with little distortion. For this reason, a large number of feature points that are less affected by distortion are grouped together as feature points of the entire image 49 . Therefore, even when calculating the posture change between adjacent frames, it is possible to improve the accuracy of estimating the amount of movement and stabilize the position of the viewpoint of the moving image.

<Example>
FIG. 12 is an exploded perspective view showing the configuration of a ball 30 in which an omnidirectional camera 14 as an imaging unit is built in an image generation device S according to a modification of the embodiment. Note that descriptions of the same or equivalent portions as those of the above embodiment will be omitted.
The ball 30 of this modified example is constructed so that a pair of hollow hemispherical transparent acrylic resin domes 31 and 32 are engaged to form a spherical outer shape.
A fixing circular plate 15 is provided inside the ball 30 . The fixing circular plate 15 is fixed to the inner side surfaces 31a and 32a of the domes 31 and 32 so as not to move. A mounting hole 16 for fixing the omnidirectional camera 14 is opened in a part of the fixing circular plate 15 .

In addition, the omnidirectional camera 14 of this modification has two lenses 14c each having a photographable angle of view of 180 degrees or more, preferably about 270 degrees, on both front and back sides 14b of a box-shaped housing 14a. 14c is provided. Two lenses 14c and 14c are arranged substantially at the center of the domes 31 and 32 with the housing 14a fitted in the mounting hole 16 of the fixing circular plate 15. As shown in FIG.
Therefore, the omnidirectional camera 14 can generate a 360-degree omnidirectional image by synthesizing the images of the two lenses.

In the omnidirectional camera 14 of the modified example configured in this manner, the two lenses 14c provided on the front and back side surfaces 14b are capable of photographing the surroundings at 360 degrees. For this reason, in the omnidirectional camera 14 of this modified example, in addition to the effects of the embodiment, the number of cameras is even smaller, the video frames are continuous in any direction, and the line of sight is displayed in a fixed direction. can be smoothly generated.

<Other embodiments>
13 to 22 show, as another embodiment, a medical capsule endoscope 80 to which the image generating device and image generating program of the present invention are applied.
In recent years, in the medical field, as shown in FIG. 13, a capsule endoscope 80 having fisheye lenses 83, 83 and built-in cameras 82, 82 capable of photographing a wide range at both ends has become popular. As shown in FIG. 14, the capsule endoscope 80 images the surrounding organs from the inside while flowing down the inside of the digestive organ by swallowing of the subject k. This makes it easy to see inside the body.

On the other hand, the capsule endoscope 80 has some problems. For example, there is a problem that the images obtained from the two cameras 82, 82 cannot be observed at the same time.
To solve this problem, by generating an omnidirectional moving image from the images obtained from the cameras 82, 82, the practitioner can recognize the positional relationship between the two images at a glance.

However, at this time, since the capsule endoscope 80 advances while rotating inside the body, it is difficult to understand how the capsule endoscope 80 advances even if an omnidirectional moving image is simply generated.
Therefore, this problem is solved by applying the method of stabilizing the omnidirectional moving image described in the above embodiment to the image obtained from the capsule endoscope 80 .

Therefore, to explain the configuration of another embodiment, as shown in FIG. 13, a capsule endoscope 80 is a capsule endoscope incorporating small cameras 82,82. Capsule endoscopes 80 are available in a form in which a small camera is attached as an imaging unit on only one side or on each of both ends.
Of these, this embodiment will be described using a capsule endoscope having small cameras 82, 82 on both sides. In this type of capsule endoscope 80, small cameras 82, 82 are attached to both axial ends of a cylindrical tubular member 84, respectively.
Transparent optical domes 86, 86 having a substantially hemispherical shape are attached to the ends of the cylindrical member 84 in the axial direction so as to block the openings of the ends. Each camera 82 is covered by an optical dome 86 attached to each end.

The two cameras 82, 82 provided at each end of the cylindrical member 84 are capable of photographing an almost 360-degree omnidirectional image.
Then, the photographed image is transmitted to the recorder 87 (FIG. 14) attached to the subject k by the wireless transmission device built in the capsule endoscope 80 . The recorder 87 is connected to a sensor 88 attached to the subject k's body. In the recorder 87, the moving image calculated by the control section 12 (see FIG. 1) is output to the monitor of the display section 13 in the same manner as in the above embodiment. The operator can view the image sent to the recorder 87 from the outside of the subject's body through the display unit 13 .
At this time, the position information from the sensor 88 may be associated with the captured moving image to specify the position inside the body.

<Dealing with missing images>
In the omnidirectional video stabilization method of the embodiment proposed by the inventors, the rotation component is removed from the omnidirectional video captured by the camera attached to the ball, and the viewpoint is fixed.
The ball camera of the above embodiment and the capsule endoscope 80 of this embodiment have in common that they move while rotating.
Therefore, if the stabilization method of the embodiment is applied to the capsule endoscope 80, the image of the capsule endoscope 80 that moves while rotating can be stabilized so that the operator can easily see it.

As shown in FIG. 13, the capsule endoscope 80 of this embodiment has two small cameras 82, 82 on the axis L that face 180 degrees in opposite directions. The omnidirectional image is created by transforming the fisheye image captured by these two cameras 82, 82 using the equirectangular projection method, the cube map method, or the like.
However, as shown in FIG. 15, the image of the capsule endoscope 80 is not a perfect circular fisheye image (circumferential fisheye image) but a diagonal fisheye image. ~d is cut off.
Also, the angle of view ang of each camera is approximately 172 degrees, and the portions of 0 to 4 degrees and 176 to 180 degrees are not photographed. Therefore, simply combining the images of the two cameras 82, 82 cannot obtain a 360-degree photographed image.

Furthermore, the respective cameras 82, 82 built into the capsule endoscope 80 do not always take images while synchronizing, but each takes images independently. That is, the time stamps of the images captured by the two cameras 82, 82 do not match. Therefore, it is not possible to easily create an omnidirectional image.

The capsule endoscope 80 used intermittently takes pictures in consideration of the capacity of the battery, or takes pictures while changing the frame rate according to the movement speed as described above. For example, if the movement of the capsule endoscope 80 in the digestive tract is slow, 4 sheets per second, and if it moves quickly, 35 sheets per second. be filmed.
As a result, some frames may be lost.

In addition, the capsule endoscope 80 captures images in a narrow space of internal organs while being in contact with the inner wall surface of the internal organs. Therefore, the image is dark and not sharp as a whole. Therefore, the shot image does not become a completely continuous image from the start of shooting to the end of shooting.
By the way, in the fixed viewpoint algorithm, feature point matching is performed between frames. Therefore, whether or not the characteristic points of the image can be obtained in creating an omnidirectional moving image has a great influence on the stability of the image.

For example, as shown in FIG. 16, as the image number, the frame of the video imaged by the front camera 82 is F1. F3. F5 . . . are odd numbers, and F2. F4. They are even numbers such as F6. FIG. 16 shows that some frames are lost when the frame rate is not constant.

In the stabilization method using the algorithm of the above-described embodiment, feature point matching between frames is performed, and after the posture change is obtained, it is decomposed into a rotation matrix and a translation vector.
The image is then modified by multiplying the current frame by the cumulative rotation matrix from the reference frame. By performing this operation for all frames, it becomes possible to fix the viewpoint of the omnidirectional video.
However, since this algorithm corrects the posture change from the reference frame using the cumulative rotation matrix, if the posture change estimation fails in the middle of the frame, matching of feature points cannot be performed thereafter. There was a possibility that the image could not be corrected.
For example, in the image generation device of the above-described embodiment, it is determined that pose estimation has failed when the number of matchings in feature point matching between frames is equal to or less than a certain threshold.

Assuming that Rij is the change in posture between frame i and frame j, it is expressed by the following equation (1).

formula 1

図１７は、特徴点マッチングが成功した一例を示す模式的な概念図である。また、図１８は、特徴点マッチングが失敗した一例を示す模式的な概念図である。

FIG. 17 is a schematic conceptual diagram showing an example of successful feature point matching. Also, FIG. 18 is a schematic conceptual diagram showing an example in which feature point matching fails.

In this embodiment, in order to solve the problem of missing frames and asynchronization of the two cameras 82, 82, the intermediate frames are interpolated and the number of frames captured by the cameras 82, 82 on both sides is the same to simulate a pseudo image. A continuous fisheye image (pseudo circular fisheye image). That is, replace the missing frame with the previous frame. As a result, the missing frame portions are smoothly connected to obtain a pseudo-continuous fisheye image.
At this time, when the control unit 12 specifically determines that the time stamps of the images captured by the respective cameras 82, 82 do not match and it is difficult to match the time stamps, the frames with the same number are It is configured to perform processing assuming that the images were taken at the same time.

Referring to FIG. 15, in order to convert an imperfect fisheye image into an omnidirectional image, the angle of view of one camera 82 is smaller than 180 degrees in the image of FIG. for example, 172 degrees), and has portions a to d cut vertically and horizontally.
Therefore, as shown in FIG. 19, by treating the image as a circle having a virtual angle of view of 180 degrees, it is possible to convert using the equirectangular projection method.

FIG. 19 is a schematic conceptual diagram for explaining how missing frames are complemented.
In FIG. 19, the area from the boundary line of parts a to d of the photographed image to the outer circle is treated as a black image with blind spots. Therefore, in the image of FIG. 15, the missing upper, lower, left, and right portions a to d are complemented with black images. The black band that appears around the image when converted to an omnidirectional image is the result of this conversion. FIG. 20 is a schematic conceptual diagram illustrating how missing frames are complemented.
FIG. 21 is a schematic conceptual diagram illustrating conversion from an incomplete fisheye image to an omnidirectional image. As a specific procedure for creating an omnidirectional image from the raw data captured by the camera 82, (1) cut out unnecessary portions such as time information from the raw data. (2) After interpolating missing portions of two fish-eye images assumed to have been taken at the same time, the two images are arranged side by side to form one image. (3) Transform into an omnidirectional image using equirectangular projection. The processes (1) to (3) are performed for all frames. Then, by linking the frames, it can be converted into an omnidirectional moving image.

<Feature point matching process>
If there are missing images when linking and connecting feature points, it is difficult to obtain a smooth moving image. Therefore, the missing portion of the frame is restored by correction.
FIG. 22 is a schematic conceptual diagram for explaining how a frame is recovered by correction when an error frame occurs in the frame.
The algorithm of the omnidirectional video stabilization method has a problem that if the feature point matching fails, the meaningful video ends there (see the upper right part of Fig. 22). frame is called an error frame).
In order to solve this problem, the image generating apparatus of this embodiment initializes the cumulative rotation matrix with a unit matrix and starts the process again with the next frame as the base point if the feature point matching fails. The ball camera of the embodiment is improved and applied to the capsule endoscope 80. FIG.
Since the rotation matrix between the error frame and the previous frame is also initialized with the unit matrix, the viewpoint of the base frame is different from the viewpoint of the frame when the program is started. That is, every time matching fails, the viewpoint is fixed to the viewpoint of the error frame.

Next, an example of an image generating apparatus using the capsule endoscope 80 of this embodiment will be described.
The capsule endoscope 80 used had an angle of view of 172×2 degrees and a frame rate that varied according to the moving speed, and was capable of imaging the large intestine mucosa at 4 to 35 frames per second. Using. Problems such as the desynchronization of the cameras 82 and 82 used in the capsule endoscope 80 and the lack of frames may be solved by making the frame rate constant.

Since the capsule endoscope 80 has an angle of view of 172×2 degrees, there is a blind spot. It is difficult to understand because it was divided into three parts when converted to spherical video, but I judged that the viewpoint was fixed to some extent.
However, it cannot be said to be complete because there are some parts that have been corrected unnaturally due to clearly incorrect assumptions. Also, if the feature point matching failed and the rotation matrix could not be obtained, the process could be re-executed using the next frame as the base point.

The results are shown in FIG. FIG. 23 shows the image of the comparative example on the left. In this comparative example, meaningless images flow after an error in the existing method. The image on the right side of FIG. 23 is a schematic conceptual diagram showing how the image generation program of the image generation device using the capsule endoscope 80 of the present embodiment is improved so that the rotation can be estimated even after an error. is.
However, even with this correction, the matching is judged to have failed only when the rotation matrix is not obtained, and the rotation matrix obtained by erroneous estimation causes an incorrect rotation. It cannot be detected when it is broken.
For this reason, the process may proceed with an erroneous rotation.

As a result of applying the fixed viewpoint algorithm to the capsule endoscope 80 in this manner, several problems were found.
The first is that there are many blind spots and it is difficult to see the video itself. This time, there were many blind spots because the omnidirectional image was created from an imperfect fisheye image. If two capsule endoscopes 80 with an angle of view of 180 degrees or more can be used by improving the performance of the capsule endoscope 80, the range that can be photographed will be widened, and a complete omnidirectional image without blind spots can be created. That is, if the image is a complete omnidirectional image, the connection between the two fisheye images can be seen, making it easy to visually recognize.

On the other hand, in the image generating apparatus of this embodiment, as shown in FIGS. 15 and 19, the upper, lower, left, and right blind spots a to d are complemented with black images.
This makes it easier to understand the connection between two fisheye images, even if they are not perfect omnidirectional images.

The second point is that rotation cannot be estimated due to erroneous estimation due to feature point matching and a small number of matchings. Feature point matching is more likely to succeed as the number of feature points in the image increases. However, an image taken in close contact with the internal organs, such as the image of the capsule endoscope 80, is a dark image with few feature points. In such a case, there is a high possibility that the feature points cannot be detected, resulting in erroneous estimation or failure.
In addition, in the case of an image in which the surrounding situation is constantly changing, such as internal organs, it is determined that the camera has rotated, which may cause erroneous estimation.

Next, processing of the image generation device will be described with reference to the flowchart of FIG. 24 and FIG. Note that the description of the same or equivalent parts as those in FIG. 7 of the above embodiment will be omitted.
In the flowchart of FIG. 24, steps S00 to S12 extract feature points from the frame (step S10) of the acquired image (step S00) (step S11 ), and the next frame and feature points are compared (step S12) (see the first and second lines in FIG. 25).

In step S16, it is determined whether collation has failed or succeeded. If the collation succeeds in step S16, the process proceeds to the next step S13 (NO in step S16), and if the collation fails in step S16, the process returns to step S10 (YES in step S16) to acquire the frame. conduct.

In steps S13 to S15, as in steps S3 to S5 of the above embodiment, posture change estimation is performed (see FIG. 10), and the results are used to determine the posture obtained from two frames by an 8-point algorithm. Using the transformation matrix R, posture correction is performed in step S14.

In the conventional stabilization method shown in the third line of FIG. 25, if the matching fails, the frames cannot be linked, and the meaningless images E1, E2, . . . It is continuously generated.
In contrast, in the end of stabilization of this embodiment, shown in the fourth row of FIG. Complement the
If the acquired image G1 of the next frame can be matched, the feature points can be matched and the stabilization process for correcting the posture can be continued. If the collation is not successful, step S10 is repeated until the collation is successful.
Therefore, even if there is a point of discontinuity in the acquired image, the next frame is immediately assigned a frame for which matching is successful, thereby recovering from the error. Therefore, even if collation fails, a stable and easy-to-view image with inconspicuous frame breaks can be obtained.

In step S15, it is determined whether or not there is a next frame, as in step S5 of the above embodiment. If there is a next frame in step S15 (Yes in step S15), the process returns to step S10, and extraction of feature points is continued (steps S11 and S12). Also, in step S15, if there is no next frame (No in step S15), the control unit 12 terminates the process as in the above embodiment.

Next, the effects of the image generation device of this embodiment will be described. In the image generating apparatus of this embodiment, the image captured by the capsule endoscope 80 is converted into an omnidirectional image, and a viewpoint-fixing algorithm is applied.
In the original algorithm, there was a problem that if the feature point matching failed, it could not be corrected thereafter (see the upper right part in FIG. 22). By updating, even after a failure, it was possible to fix the viewpoint, recover from the error, and continue the image (see the lower right part in FIG. 22).
It is more desirable to estimate the rotation matrix using a Kalman filter or the like in order to prevent the viewpoint from changing every time it fails.

Also, if there is a blind spot in the endoscopic image, by using two capsule endoscopes with an angle of view of 180 degrees or more by improving the performance of the capsule endoscope 80, the range that can be photographed is widened and the blind spot is eliminated. Alternatively, a complete omnidirectional image may be created.

Furthermore, if the image itself captured by the capsule endoscope 80 is dark, the capsule endoscope 80 can use auxiliary light from an LED illumination device, or change the ISO sensitivity to a higher one. It is even better if the performance of the camera 82 is improved.

<Modification 1>
FIG. 26 shows a crane truck 50 of Modification 1 of the embodiment. The crane truck 50 has a hook member 53 suspended from one end of a wire 52 extending from the tip of an arm portion 51 . The hook member 53 engages the ball 3 or the like of the above-described embodiment, and the ball 3 can be lowered from the tip of the arm portion 51 extended to a position separated from the driver's seat 54 .
As a result, the ball 3 can enter a space where people cannot enter due to a disaster site or the like, and the state of the interior can be photographed with a stable image.

<Modification 2>
FIG. 27 shows a drone (unmanned aerial vehicle) 60 of Modification 2 of the embodiment. A camera 62 equivalent to the ball 3 of the above-described embodiment or the capsule endoscope 80 of another embodiment is attached to the lower surface 61 of the drone 60 at the center of the drone 60, and the drone 60 is positioned at a distance from the operator. It is possible to shoot a stable image of the surroundings.
Since the drone 60 can be remotely controlled, the operator can take detailed pictures of the site even if the operator is away from the disaster site, thereby preventing secondary disasters.

<Modification 3>
FIG. 28 shows a fishing rod 70 of modification 3 of the embodiment. A fishing line 74 is let out through a plurality of guides 73 from a reel 72 fixed to the hand of a rod 71 . A camera 75 equivalent to the ball 3 of the embodiment or the capsule endoscope 80 of another embodiment is attached to the tip of the fishing line 74 .
Then, the camera 75 at the tip of the fishing line 74 is lowered at a position away from the person on the ground holding the rod 71, and the state of the water can be photographed with a stable image.

Although the image generation device S and the image generation program according to the present embodiment have been described in detail above, the present invention is not limited to these embodiments, and can be changed as appropriate without departing from the scope of the present invention. It goes without saying that

For example, in the present embodiment, six rotated images 42a to 47a are generated by rotating a photographed omnidirectional image 41 by π/6. However, it is not particularly limited to this. For example, one or more images with different rotation angles, such as 4 or 8 images rotated by π/4 or π/8, may be used at any angular interval in the direction of rotation. It is sufficient if the feature point can be extracted using the low latitude portion of the image with little distortion.

Furthermore, in the embodiment, six cameras 4 constitute the imaging unit 1, and in the modified example, the omnidirectional camera 14 having two lenses 14c corresponding to two units constitutes the imaging unit 1. FIG. However, it is not particularly limited to this. For example, the imaging unit 1 may be configured using a plurality of cameras, one or two or more. The number, shape and combination of cameras are not particularly limited as long as they can

For example, in the other embodiment, as shown in FIG. 13, a capsule endoscope 80 having fisheye lenses 83, 83 and built-in cameras 82, 82 capable of photographing a wide range at both ends is used. Yes, but not limited to this. For example, a wide-angle lens other than the fisheye lens 83 may be used as long as the camera 82 is provided on at least one of the one side and the other side and the cameras 82, 82 are capable of photographing a wide range. The shape, quantity, and material of 82 and lenses are not particularly limited.

Also, in the embodiment and the modified example, the balls 3 and 30 are used as the objects on which the imaging unit 1 is mounted, but the objects are not limited to this. For example, an omnidirectional camera may be attached to each surface of a cubic frame that can rotate and move. Alternatively, it may be an oval ball such as a rugby ball. Thus, the shape of an object that moves with rotational motion is not limited to a spherical shape.

In the image generation device S of the embodiment, the case where the ball simultaneously performs rotational motion and translational motion has been described, but the present invention is not limited to this, and at least one of rotational motion and translational motion is performed. If it is

1 imaging unit 3, 30 ball 4 camera 5, 14c lens 10 communication interface unit 11 storage unit 12 control unit 13 display unit 14 omnidirectional camera (camera)
14a Housing 15 Circular plate for fixing 16 Mounting hole 20 Input unit 31 Dome 80 Capsule endoscope 82 Small camera (imaging unit)
100 network

Claims (11)

an imaging unit attached to a moving object;
When generating a moving image in the same viewpoint direction based on the images acquired by the imaging unit, the By matching the feature points in the image of the frame (t+1) after the movement of the imaging unit, the amount of movement is calculated, and the posture correction is repeated to return the image by the amount of movement. a control unit that consecutively frames moving images;
An image generation device comprising:
The control unit rotates a high latitude region to a low latitude region in one step for a frame at each time, extracts a feature point from the low latitude region, and reversely rotates the coordinates of the feature point. to the coordinates in the original image of the high latitude region, and then in the next step of the first step, the feature points extracted from the previous frame and the feature points extracted from the current frame are compared with each other. 1. An image generating apparatus comprising: a calculation unit for estimating a change in posture; and a generation unit for correcting the posture based on the estimated change in posture. The calculation unit uses the two-dimensional rotated image generated from the omnidirectional image in the feature point collation, and if the feature point is located in a high latitude region with much distortion of the image periphery, the distortion of the periphery in the same viewpoint direction 2. The image generating apparatus according to claim 1, wherein after moving the image to a low latitude area, the feature points are matched. 2. The image generating apparatus according to claim 1, wherein the object moves with rotational motion. 2. The image generating apparatus according to claim 1, wherein the object moves with translational motion. 2. The image generating apparatus according to claim 1, wherein said imaging unit is provided in an omnidirectional camera. 2. The image generating apparatus according to claim 1, wherein the imaging unit is provided on at least one of one side and the other side of the capsule endoscope. 2. The image generation apparatus according to claim 1, wherein the control unit replaces the missing error frame with a frame preceding the error frame to complement the missing error frame. 8. The image generating apparatus according to claim 7, wherein the control unit complements a missing peripheral portion of the image of the error frame having a missing peripheral portion with a black image. an imaging unit provided on a moving object for capturing an image of the surroundings of the moving object;
an image generation program for causing an image generation device to execute , comprising: a control unit that calculates a movement amount when the imaging unit moves from a moving image acquired by the imaging unit, and continues frames of the moving image; hand,
a feature point extracting step of catching feature points present in an image of a moving image captured by the imaging unit in the image generation device ;
A high latitude area is rotated to a low latitude area for a frame at each time, a feature point is extracted from the low latitude area, and the coordinates of the feature point are reversely rotated to coordinates in the original image of the high latitude area. a correction step that corrects to
a feature point matching step of matching feature points extracted from a previous frame with feature points extracted from a current frame;
An image generation program for executing a step of calculating a movement amount and repeating moving images back by the movement amount to continue frames of a moving image. An image generating program to be executed by an image generating device for calculating the amount of movement of a moving object from a moving image of the surroundings of the moving object and making the frames of the moving image continuous,
a feature point extracting step of catching feature points present in the image of the moving image in the image generation device ;
A high latitude area is rotated to a low latitude area for a frame at each time, a feature point is extracted from the low latitude area, and the coordinates of the feature point are reversely rotated to coordinates in the original image of the high latitude area. a correction step that corrects to
a feature point matching step of matching feature points extracted from a previous frame with feature points extracted from a current frame;
An image generation program for executing a step of calculating a movement amount and repeating moving images back by the movement amount to continue frames of a moving image. a collation determination step of determining whether collation has failed or succeeded in the image generation device ;
a frame acquisition step of acquiring a frame for which matching is successful if matching fails;
11. The image generating program according to claim 9 or 10, for executing

Images