JP7393179B2 - Photography equipment - Google Patents

Description

The present invention relates to a photographing device suitable for narrow spaces.

When you want to photograph the surrounding area or internal structure (hereinafter referred to as the subject) in a narrow space such as a tunnel, a narrow alley, a room, a structure with a narrow recess, or a space between two closely spaced walls. be. In a narrow space, it is not possible to maintain a sufficient distance between the camera and the subject, so if the camera and subject are directly facing each other, even if a wide-angle lens is used, part of the subject may not be captured by the camera. Situations may arise.

Patent Document 8 discloses that while moving a camera along the subject, the subject is photographed from a close angle, the portion including the subject is cropped from the photographed image, the image is converted to an image photographed from the front, and the converted image is Disclosed is a technique for sequentially connecting and synthesizing the images in the scanning direction. According to this method, the field of view can be made wider than when the camera is directly facing the subject.

Japanese Patent Application Publication No. 2004-312549 International Publication WO08/102898 Publication Japanese Patent Application Publication No. 2007-323616 Japanese Patent Application Publication No. 2010-073035 JP2012-022652A Japanese Patent Application Publication No. 2012-109737 Japanese Patent Application Publication No. 2013-250891 Japanese Patent Application Publication No. 2011-126989

R. Timofte, V.D. Smet, and L. V. Gool., "A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution. ACCV, 2014" C. Dong, C. C. Loy, K. He, and X. Tang. "Image Super-Resolution Using Deep Convolutional Networks", TPAMI, 2016

Even with the technique described in Patent Document 8, a situation may arise where the photographing range in the height direction is insufficient. One possible approach to solving this problem is to take pictures while tilting the camera in the height direction (pitching direction) and combine multiple images. However, this method requires the camera to be tilted, which makes the entire photographing device structurally complex. Furthermore, it is time-consuming because it requires multiple shootings.

Another approach to solving this problem is to use multiple cameras facing different directions or placed at different heights to take pictures. However, in this case, multiple cameras are required, which complicates the system and increases cost.

The present invention has been made in view of the above problems, and one exemplary objective of a certain aspect of the present invention is to provide an imaging device suitable for narrow spaces.

One aspect of the present invention relates to an imaging device that generates a lateral front-view image of a narrow area. The imaging device has a camera that takes pictures while moving in the depth direction of a narrow space, and an image of the focused area in the camera image when the focused area and the camera are relatively far, and an image of the focused area when the focused area and the camera are relatively close. An image processing unit that has a trained model derived from the correlation with the image of the gazed part in the camera image, and generates a front-view image based on the camera images sequentially output from the camera during actual operation. and.

The image processing unit crops a predetermined area from the camera image, inputs the cropped image to a trained model to generate a high-resolution image, and synthesizes the high-resolution images that are sequentially generated as the camera moves to create a front-view image. An image may also be generated.

The image processing unit crops a first region included in one of the pair of camera images taken at different times but not included in the other, and inputs the cropped image to the trained model to generate a high-resolution partial image. Then, the high-resolution partial image is combined with the image of the second region of the other camera image pair to generate a high-resolution image, and the high-resolution images sequentially generated as the camera moves are combined to create a front-view image. An image may also be generated.

The image processing unit may generate a high-resolution image based on an image set including a plurality of camera images taken at different times.

A predetermined region is defined at a different position for each of the plurality of camera images included in the image set, and the image processing unit crops the corresponding predetermined region from each camera image and inputs the cropped image to the trained model. A high-resolution intermediate image may be generated by combining a plurality of high-resolution intermediate images obtained for a plurality of camera images to generate a high-resolution image.

The high resolution image may be vector data.

Note that arbitrary combinations of the above-mentioned constituent elements and mutual substitution of constituent elements and expressions of the present invention among methods, devices, systems, etc. are also effective as aspects of the present invention.

According to the present invention, it is possible to provide a photographing device suitable for narrow spaces.

FIG. 1(a) is a diagram illustrating an object to be photographed by a photographing device according to an embodiment, and FIG. 1(b) is a diagram showing a final image that is an output of the photographing device. FIG. 1 is a block diagram of an imaging device according to an embodiment. FIG. 3(a) is a diagram explaining the movement of the camera, and FIGS. 3(b) and 3(c) are two images IMG_A and IMG_B obtained at camera viewpoints A and B in FIG. 3(a). FIG. FIGS. 4A and 4B are diagrams showing front-view images obtained in the first comparative technique and the second comparative technique. It is a figure explaining learning. FIGS. 6A and 6B are diagrams illustrating super-resolution processing. FIGS. 7A to 7C are diagrams illustrating the operation of the photographing device. FIG. 1 is a block diagram of an imaging device according to an embodiment. FIGS. 9A and 9B are diagrams illustrating the processing of the front-view image composition section. FIG. 3 is a diagram showing a user interface of a display. 7 is a diagram illustrating super-resolution processing according to Modification 1. FIG. FIG. 7 is a diagram illustrating super-resolution processing according to Modification 2.

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below based on preferred embodiments with reference to the drawings. Identical or equivalent components, members, and processes shown in each drawing are designated by the same reference numerals, and redundant explanations will be omitted as appropriate. Further, the embodiments are illustrative rather than limiting the invention, and all features and combinations thereof described in the embodiments are not necessarily essential to the invention.

FIG. 1A is a diagram illustrating an object to be photographed by the photographing apparatus 100 according to the embodiment. The object to be photographed by the photographing device 100 is the side of the narrow area 2 (in this example, a road). The photographing device 100 photographs both sides or one side of the narrow portion 2 while moving in the depth direction. In the following, for the sake of brevity and ease of understanding, the explanation will focus on the left side of the photographing device 100.

FIG. 1(b) is a diagram showing the final image IMGf that is the output of the imaging device 100. The final image IMGf is a front view image of the side (left side in this case) of the narrow part, and includes a plurality of objects OBJ1 to OBJ3 existing on the left side of the narrow part 2.

FIG. 2 is a block diagram of the imaging device 100 according to the embodiment. The photographing device 100 includes a camera 110 and an image processing section 120. The camera 110 is directed toward the front in the direction in which the photographing device 100 moves. Alternatively, when measuring only the left side, the camera 110 may be placed slightly tilted to the left.

The camera 110 continuously takes pictures while moving in the depth direction of the narrow part. Camera 110 may be a still camera or a video camera.

FIG. 3(a) is a diagram illustrating movement of the camera. FIG. 3(a) shows the narrow portion 2 viewed from the side. FIGS. 3(b) and 3(c) are diagrams showing two camera images IMG_A and IMG_B obtained at camera viewpoints A and B in FIG. 3(a).

Focus is on the central object OBJ2 (a building in this example). The image IMG_A taken from the viewpoint A on the near side shows the entire front of the object OBJ2, but since its size is small, it can be said that the resolution of the object OBJ2 is low.

In the image IMG_B taken from a viewpoint B closer to the object OBJ2, which is the subject, the object OBJ2 is larger than the image IMG_A, and its details are shown in high resolution. However, the entire front of object OBJ2 is not captured.

Before explaining the image processing of the imaging device 100 according to the embodiment, some comparative techniques will be explained.

In the first comparison technique, from an image taken at a position close to the subject (attention point), a region including the subject (region RGN1 in FIGS. 3(b) and 3(c)) is cropped and converted into a front view. and synthesize. FIG. 4(a) is a diagram showing a front view image obtained in the first comparison technique. With the first comparison technique, a high-resolution front-view image can be obtained, but the upper side of the subject is missing.

In the second comparison technique, an area where the entire object is captured (region RGN2 in FIGS. 3(b) and (c)) is cropped from an image taken at a far position from the object, and it is converted to a front view. , synthesize. FIG. 4(b) is a diagram showing a front view image obtained in the second comparison technique. With the second comparison technique, the entire front of the subject can be observed, but the image is stretched by front view conversion, so the resolution decreases and edges and contours appear blurred, as if out of focus.

Returning to FIG. 2, image processing in this embodiment will be described. The image processing unit 120 generates a final image IMGf based on the output of the camera 110. The image processing unit 120 can be implemented by a combination of arithmetic processing means such as a CPU (Central Processing Unit) and a software program.

The image processing unit 120 is equipped with a trained model 122 that is generated in advance by machine learning. Machine learning of the trained model 122 will be explained. The trained model 122 is used for super-resolution processing of images.

Super-resolution processing may use classical methods that do not require training data (eg, bilinear method, Lanczos method), but in that case, accuracy becomes a problem. Therefore, in this embodiment, a learning-based method such as the A+ method disclosed in Non-Patent Document 1 and the SRCNN method disclosed in Non-Patent Document 2 is adopted. The A+ method is a method of constructing a learning model called a sparse dictionary from learning data. The SRCNN method is a method using deep learning. As the learning images, a general-purpose corpus may be used, or one may be prepared independently. A pair of low-resolution images and high-resolution images is required for training the trained model 122, but often a high-resolution image and an image obtained by lowering the resolution using the classical method described above are used. be. A predetermined value (temporarily assumed to be r) is used as the enlargement ratio of the high resolution image corresponding to the low resolution image.

On the other hand, images obtained in the environment in which the photographing device 100 is used or in an environment similar thereto can also be used as the learning data. FIG. 5 is a diagram explaining learning. In the learning stage, a plurality of sample images SI are taken while moving the camera 110. This sample image SI is used as learning data. A certain object or part (referred to as a gaze part) existing in a narrow area appears in different sizes and at different positions in a plurality of sample images taken from different positions.

In FIG. 5, X _# and Y _# (#=1, 2, . . . ) represent regions including the same gaze portion. When focusing on a certain gazing area, a sample image obtained when the gazing area and the camera are relatively far apart is referred to as a first image. Further, a sample image obtained when the same gaze point and the camera are relatively far apart is referred to as a second image. The trained model is derived based on the correlation between the image X _# of the part of interest in the first image and the image Y _# of the part of interest in the second image. The image X _# is a wide-range, low-resolution image of the gazed area, and the image Y _# is a high-resolution image of a narrow-gazing area. The learned model 122 can be understood to execute a conversion process from X to Y. This conversion process is represented by a mapping f.
Y _# =f(X _# )
As mentioned above, the pair of images X _# , Y _# may be a one-dimensional image (vector) with a width of one pixel. In this case, the height (number of pixels) of image Y _# is r times the height (number of pixels) of image X _# .

When the pair of images X _# and Y _# is a two-dimensional image with a width of 2 pixels or more, their shapes may be different. This is because the camera's optical system creates a perspective depending on the distance and position to the same subject, so the shapes of the subjects are photographed differently.

By inputting a pair of images X _# and Y _# from a plurality of sample images to a learning device for each of a plurality of gaze parts, a transformer Y=f(X) can be obtained as a trained model. The image processing unit 120 may perform denoising processing in addition to super-resolution processing.

Return to Figure 2. The image processing unit 120 crops a predetermined region from the output image of the camera 110 during operation of the photographing device 100, and inputs a low-resolution image included in the predetermined region to a converter that is a trained model. As a result, an image (high-resolution image) in which the entire predetermined area is subjected to super-resolution processing is obtained. FIGS. 6A and 6B are diagrams illustrating super-resolution processing. FIG. 6(a) shows a case where a one-dimensional low resolution image IMGl is input. The high-resolution image IMGh can be understood as an estimated image obtained when the camera position approaches the gaze area, and its height h' is r times the height h of the low-resolution image IMGl (r> 1).

FIG. 6B shows a case where a two-dimensional low resolution image is input, and the height h' of the high resolution image IMGh is r times the height h of the low resolution image IMGl. The width w' of the high resolution image IMGh may be equal to or larger than the width w of the low resolution image IMGl.

Return to Figure 2. The image processing unit 120 combines high-resolution images that are sequentially generated as the camera 110 moves, and generates a final image IMGf that is a front-view image.

The above is the configuration of the photographing device 100. Next, its operation will be explained. FIGS. 7A to 7C are diagrams illustrating the operation of the photographing device 100. FIG. 7(a) shows an output image of the camera 110 obtained at a certain time. The image processing unit 120 extracts a predetermined region RGNc from the image of FIG. 7(a). In this example, the width of the predetermined region RGNc is 2 pixels or more, and the low-resolution image to be cropped is two-dimensional image data. You can also use it as

FIG. 7B shows a high-resolution image IMGh obtained by inputting the image data in the region RGNc to the learned model 122. For comparison, the frame of image IMG_B in FIG. 3(c) is indicated by a chain line. This high-resolution image IMGh has rich details even in the portions that protrude from the frame of image IMG_B.

FIG. 7(c) shows an image IMGg obtained by converting the high-resolution image IMGh of FIG. 7(b) in front view. A plurality of front-view high-resolution images IMGg are generated by repeating the processes in FIGS. 7A to 7C for camera images obtained at different times while changing the camera position. By combining them, the final image IMGf shown in FIG. 1(b) can be obtained.

Next, a specific configuration of the photographing device 100 will be described with reference to examples.

FIG. 8 is a block diagram of the imaging device 200 according to the embodiment. The photographing device 200 includes an image photographing section 202, an input image determining section 204, a sharpening processing section 206, a front view image combining section 208, and an image output section 210.

The input image determination unit 204 is an image acquisition unit and corresponds to the camera 110 in FIG. 2. The input image determining unit 204 may include a camera, means for moving the camera, means for acquiring the position or movement distance of the camera, means for controlling the attitude of the camera, and the like. The image captured by the input image determining unit 204 is input to the input image determining unit 204 together with position information.

The input image determination unit 204, the sharpening processing unit 206, and the front-view image synthesis unit 208 can be associated with the image processing unit 120 in FIG. 2.

The input image determining unit 204 receives the image and position information obtained by the image capturing unit 202, and selects a part of the input image (corresponding to the predetermined area RGNc in FIG. 7(b), hereinafter referred to as the selected area). , print it. The selected area is an area located far from the camera and photographed over a relatively wide range, and is an area that is too far from the camera and has low clarity (effective resolution). The selected area may be linear (vector) or may be a wide area (image). The image data of the selected area is supplied to the sharpening processing unit 206 together with position information.

The sharpening processing unit 206 receives the selected area from the input image determining unit 204 and outputs an image with increased sharpness. "Sharpening" includes a function to enlarge an image, such as super-resolution processing, and a function to remove noise (denosing). In the case of super-resolution processing, if the selected area is vector data, a vector whose size is the number of elements multiplied by a constant is output. If the selected area is two-dimensional image data with a width, an image whose size is a constant multiplication of the number of pixels in the vertical and horizontal directions is output. The output of the sharpening processing unit 206 includes a high-resolution image (one-dimensional vector or two-dimensional image) and its position information. The learned model 122 in FIG. 2 can be used as a means for generating a high-resolution image.

FIGS. 9A and 9B are diagrams illustrating the processing of the front-view image composition unit 208. The front-view image synthesis unit 208 combines the high-resolution images output from the sharpening processing unit 206 using the position information output together with the high-resolution images to generate one large final image IMGf. If the high-resolution image is a one-dimensional image (vector), a plurality of high-resolution images L1 to Ln may be combined in the horizontal direction, as shown in FIG. 9(a). When the high-resolution images are two-dimensional images, as shown in FIG. 9(b), a margin 4 may be provided and the images may be pasted together. In this case, as the pixel value in the overlap portion 4, the average value of the corresponding pixels may be used, or the maximum value may be used. Through this processing, a front-view image can be obtained. If an unusual edge or the like occurs at the joint as a result of the bonding process, post-processing may be performed to smooth it, for example, a Gaussian filter or a Wiener filter may be used.

Note that as pre-processing or post-processing for pasting, conversion processing such as camera position information correction may be performed in order to approximate the image viewed from the front.

Returning to FIG. 8, the image output unit 210 outputs the final image IMGf obtained by the front-view image synthesis unit 208. The image output unit 210 may include a display and visually present the final image IMGf to the user. The image output unit 210 may have a scaling function. Furthermore, if there are multiple final images, a function to display them simultaneously or a function to display them selectively may be provided. If the size of the final image is large, the display portion may be changed using a GUI (Graphical User Interface) such as a slide bar.

Further, the image output unit 210 may have a function of attaching a marker to and highlighting a portion of interest to the user (for example, an abnormal portion or a damaged portion). It may also have a function of calculating positional information of a portion of interest to the user and displaying the value. Note that the image output unit 210 may save the data of the final image IMGf in a storage medium.

FIG. 10 is a diagram showing the user interface of the display. In area 302 of display 300, an output image of the camera is displayed. Accompanying this area 302 are play, stop, and rewind buttons 306 so that the image (or frame) displayed in the area 302 can be controlled.

A marking 308 can be displayed in the area 302. The marking 308 is an area designated by the user, and whether or not the marking 308 is displayed can be controlled by a marking button 310. For example, when the imaging device 200 is a diagnostic device, the area surrounded by the markings 308 is the target of the abnormality detection process, and abnormal locations and damaged locations within the area are detected.

In area 310, the final image IMGf obtained by image processing is displayed. Region 310 may be divided into multiple columns 312. For example, different portions of the final image IMGf may be selectively displayed in the plurality of columns 312. Furthermore, when generating front-view images of both the right and left sides of the narrow portion, the final image of the left side surface may be displayed on one of the plurality of columns 312, and the final image of the right side surface may be displayed on the other column. If the entire final image cannot be displayed in the column 312, a slide bar 314 may be displayed to allow the user to select the display range. In addition to or in place of the slide bar 314, buttons for enlarging or reducing the image may be added.

A marking 316 can also be displayed in the area 310, and the display portion of the marking 316 (that is, 310) can be enlarged or reduced by buttons 320 and 322.

The above is the configuration of the photographing device 200. According to the photographing device 200, it is possible to generate a front-view image in which a wide range is photographed, and it becomes easy for a person to visually discover what exists where in the image. Furthermore, since there is almost no distortion, it becomes easy to automatically find out what exists where by image recognition processing. Therefore, the accuracy of visual processing and automatic image recognition processing using the output of the photographing device 200 can be improved.

The use of the photographing device 100 and the photographing device 200 is not particularly limited, but it is widely applicable to tunnels, clay pipes, coke ovens, etc. where the measurement target is long and narrow and the camera cannot be directly facing the wall, or where it is difficult to install. can.

The present invention has been described above based on examples. It will be understood by those skilled in the art that the present invention is not limited to the above embodiments, and that various design changes and modifications are possible, and that such modifications also fall within the scope of the present invention. By the way. Hereinafter, such modified examples will be explained.

(Modification 1)
In previous super-resolution processing, a high-resolution image of a single gaze area was generated using a single image, but in modification example 1, a set of multiple camera images taken at different distances was used. Then, a high-resolution image of a single gaze area is generated. FIG. 11 is a diagram illustrating super-resolution processing according to Modification 1. FIG. 11 shows a plurality of (three in this example) camera images obtained at different times, in other words, photographing the same gazed area from different distances. A unique region is defined for each of the plurality of camera images so as to include the same gazed portion. From each camera image, an image of a unique region is cropped to generate three low-resolution images IMGl1 to IMGl3. By performing super-resolution processing on the three low-resolution images IMGl1-IMGl3, three high-resolution intermediate images IMGh1-IMGh3 are obtained. Three high-resolution intermediate images IMGh1 to IMGh3 are combined to generate one high-resolution image IMGh.

(Modification 2)
FIG. 12 is a diagram illustrating super-resolution processing according to Modification 2. In the super-resolution processing shown in FIGS. 6A and 6B, the region to be cropped was determined to include the entire gaze portion. On the other hand, in the modified example shown in FIG. 12, a part of the gazed area is cropped and super-resolution processing is performed. The left side of FIG. 12 shows an image obtained when the object is far from the gaze area 400, and the right side of FIG. 12 shows an image that is obtained when the image is close to the gaze area 400. In the image of FIG. 12, reference numeral 402 represents a portion that goes out of frame when the camera approaches the focused portion. In this modification, this portion 402 is cropped, and the cropped image is subjected to super-resolution processing. As a result, a high-resolution image portion (high-resolution partial image) 404 of a portion that goes out of frame when the camera approaches the focused portion is obtained. Furthermore, a range 406 of the gazed portion that is framed in the image obtained when the camera approaches the gazed portion is selected, and by combining the two portions 404 and 406, a high-resolution image IMGh can be generated.

2 Narrow area 100 Photographing device 110 Camera 120 Image processing section 122 Learned model 200 Photographing device 202 Image photographing section 204 Input image determining section 206 Sharpening processing section 208 Front-view image composition section 210 Image output section

Claims (6)

An imaging device that generates a lateral front view image of a narrow space,
a camera that photographs the narrow area while moving in the depth direction;
an image processing unit that has a converter based on the trained model derived in the learning stage and generates the front-view image based on camera images sequentially output from the camera during actual operation;
Equipped with
In the trained model, a gazed area, which is an object or part existing in a narrow area, is captured in a predetermined first part of a plurality of sample images taken while moving the camera in the depth direction during the learning stage. Using a pair of a first image and a second image photographed on the back side of the first image in which the gazed area is reflected in a predetermined second part, the gazed area of the first image is is derived from the correlation between the image X of configured,
The image processing unit crops a predetermined region from the camera image during the actual operation, inputs the cropped image to the converter, and generates a high-resolution image in which the predetermined region is subjected to high-resolution processing in the height direction. The photographing device is characterized in that the plurality of high-resolution images generated as the camera moves are combined to generate the front-view image . The predetermined area includes the first portion and is longer than the first portion in the height direction,
The photographing device according to claim 1 , wherein the image processing unit laterally combines the high-resolution images that are sequentially generated as the camera moves to generate the front-view image . The predetermined area is adjacent to the upper side of the first portion and is defined in correspondence with a portion of the second image where the gazed portion goes out of frame in the height direction,
The image processing unit includes:
clipping the predetermined area from one of the camera images taken while the camera is moving and inputting it to the converter to generate a high-resolution first image portion;
generating a second image portion by selecting a range of the gazed portion from another camera image in which the gazed portion is captured in the second portion among camera images taken while the camera is moving;
generating the high-resolution image by connecting the first image portion and the second image portion in the height direction;
The imaging device according to claim 1, wherein the front-view image is generated by combining a plurality of the high-resolution images generated as the camera moves. The photographing device according to claim 1, wherein the image processing unit generates a high-resolution image based on an image set including a plurality of the camera images taken at different times. For each of the plurality of camera images included in the image set, predetermined areas are defined at different positions in which the same object should be captured ,
The image processing unit crops a corresponding predetermined region from each camera image, inputs the cropped image to the trained model, and generates a high-resolution intermediate image;
The photographing device according to claim 4, wherein the high-resolution image is generated by combining a plurality of high-resolution intermediate images obtained for the plurality of camera images. 6. The photographing device according to claim 2, wherein the high-resolution image is vector data.

Images