JP2020182048A - Imaging device, image processing device, and image processing method - Google Patents

Abstract

To provide an imaging device, an image processing device, and an image processing method that can acquire a panoramic image of a shooting target wider than a shooting range of a camera with appropriate resolution.SOLUTION: An imaging device includes: a shooting device 21 capable of shooting a part of a shooting target range 11 as a shooting range; a moving body 22 that moves the shooting device 21 along a shooting target range 11 such that an un-shooting portion of the shooting target range 11 is included in a new shooting range; and an image processing device 35 that performs image processing on each shot image obtained by shooting. The shooting device 21 creates a new shot image including the un-shooting portion by moving the moving body 22. The image processing device 35 includes a target extraction unit 40 that extracts a portion of each shot image which includes a predetermined velocity vector as a part of the shooting target, a relative position calculation unit 41 that calculates the relative position of each shot image, and a creation unit 42 that arranges the respective shot images on the basis of the relative position and creates a wide area image including a range wider than the shooting range.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image photographing apparatus for photographing an image to be photographed, an image processing device for processing the captured image, and an image processing method.

Conventionally, there is a horizontally long or vertically long wide-angle image, a so-called panoramic image, as compared with an image having an aspect ratio of 3: 4 or the like captured by a digital camera. Such a panoramic image is taken by combining images taken sequentially while changing the direction of the lens of the digital camera with respect to a shooting target wider than the shooting range of the digital camera (for example, Patent Document 1).

In the technique described in Patent Document 1, the user moves the digital camera so as to rotate it in the horizontal direction while maintaining the state of pressing the shutter switch while keeping the digital camera substantially fixed in the vertical direction. After that, when a predetermined condition such as the elapse of a predetermined time or the movement of the image processing device by a predetermined amount is satisfied, the image pickup control unit ends the continuous imaging operation in the image pickup unit.

Japanese Unexamined Patent Publication No. 2016-186797

In panoramic shooting, a range wider than the shooting range of the camera is shot by changing only the direction of the lens of the camera while maintaining the position of the camera. At this time, if the camera and the shooting target are close to each other, the camera lens is shaken greatly to the left and right to shoot the shooting target, but the distance to the shooting target becomes longer and the resolution decreases depending on the magnitude of the shake angle. Therefore, it becomes difficult to take a panoramic image having a resolution with a small change. On the other hand, if the camera and the shooting target are far apart, the shooting target can be shot by shaking the camera lens slightly to the left and right, so the change in resolution is suppressed as the distance to the shooting target changes, and a panoramic image with a small change in resolution is shot. However, because the distance between the camera and the subject is long, only low-resolution images can be obtained.

The present invention has been made in view of such circumstances, and an object of the present invention is an image capturing device or an image processing device capable of acquiring a panoramic image of a shooting target wider than the shooting range of a camera with an appropriate resolution. And to provide an image processing method.

The image capturing device that solves the above problems includes a photographing device capable of photographing a part of the photographing target range as a photographing range, and an unphotographed portion of the photographing target range in the new photographing range of the photographing device. The imaging device includes a moving body that moves the imaging device along the imaging target range, and an image processing device that performs image processing on each captured image obtained by photographing the imaging device. A new photographed image is created in which the unphotographed portion is included in a new imaging range that is moved by the body, and the image processing device captures a portion of each captured image that includes a predetermined speed vector. The target extraction unit to be extracted as a part of the image, the relative position calculation unit for calculating the relative position of each photographed image captured by the imaging device, and the relative position calculation unit for calculating the relative position of each captured image captured by the imaging device, and the captured images arranged based on the relative position to be larger than the photographing range. It is equipped with a creation unit that creates a wide area image including a wide range.

The image processing device that solves the above problems is a portion including a predetermined speed vector in each captured image captured by the imaging device based on the speed vector of the imaging device and the distance from the imaging device to the imaging target range. A target extraction unit that extracts the image as a part of the image to be photographed, a relative position calculation unit that calculates the relative position of each photographed image, and an arrangement of each photographed image based on the relative position to exceed the image pickup range of the image pickup device. It is equipped with a creation unit that creates a wide area image including a wide range.

The image processing method for solving the above problem is a portion including a predetermined speed vector in each captured image captured by the imaging device based on the speed vector of the imaging device and the distance from the imaging device to the imaging target range. A target extraction step for extracting as a part of a shooting target, a relative position calculation step for calculating the relative position of each shot image, and an arrangement of each shot image based on the relative position, which is larger than the shooting range of the shooting device. It includes a creation step to create a wide area image that includes a wide range.

According to such a configuration or method, when the photographing device is moved by a moving body, images in a shooting target range wider than the shooting range can be obtained as each shot image from a distance where an appropriate resolution is secured. .. Then, by arranging the captured images captured in this way, it is possible to acquire a panoramic image which is a wide area image having an appropriate resolution.

Further, from each of the captured images captured by the photographing device moved by the moving body, the target imaging target is extracted based on having a specific velocity vector in the image. As a result, it is possible to extract information necessary for specifying the relative position from the captured image without performing image processing for separating the imaging target from the background with high accuracy.

As a preferred configuration, the target extraction unit calculates a speed vector of each shot image based on the speed vector of the shooting device and the distance from the shooting device to the shooting target.
According to such a configuration, information necessary for specifying the relative position is extracted based on the speed vector of the photographing device and the speed vector calculated based on the distance between the photographing target and the photographing device.

As a preferred configuration, the relative position calculation unit identifies the relative position of each captured image based on the position and velocity vector of the photographing device and the distance from the photographing device to the imaging target range.

According to such a configuration, the moving route of the photographing device is specified based on the position and speed vector of the photographing device and the distance between the photographing target and the photographing device, so that each image taken on the moving route is taken. The relative position of the image can be calculated.

As a preferred configuration, the relative position calculation unit further identifies the relative position of each captured image based on the result of the image recognition processing of each captured image.
According to such a configuration, it is possible to add an image recognition result for each captured image to create a panoramic image with higher accuracy.

As a preferred configuration, the moving body scans the imaging target range within the imaging range of the imaging device.
According to such a configuration, it is possible to acquire each captured image in which the imaging target range is captured without interruption, so that a seamless panoramic image can be created. Further, since each shot image in the shooting target range is shot in order, it is easy to calculate the relative position of each shot image and to create a panoramic image by synthesizing each shot image.

As a preferred configuration, the scan includes a horizontal scan and a vertical scan.
According to such a configuration, each shot image in the shooting target range is created in a predetermined order, so that it is possible to calculate the relative position of each shot image or combine the shot images to create a panoramic image. It will be easier.

As a preferable configuration, a pattern learning unit that learns an image pattern to be detected from a shooting target range and a pattern detecting unit that collates with the learning result by the pattern learning unit and detects an image pattern corresponding to the learning result from the captured image. Further prepare.

According to such a configuration, a characteristic image pattern to be detected and a panoramic image thereof, such as cracks, scratches, rust, and other parts having a color or pattern different from the normal color or pattern, from each photographed image. The upper occupied position can be detected.

As a preferred configuration, the pattern detection unit highlights the detected image pattern.
According to such a configuration, in a panoramic image, a portion including a characteristic image pattern to be detected can be easily recognized by highlighting.

As a preferred configuration, the moving body is capable of three-dimensional orbitless movement including the vertical direction.
According to such a configuration, even if the moving object has a shooting target range that is larger in all directions than the shooting range, the entire view can be viewed in the horizontal direction, the vertical direction, and the direction of turning around the moving object in a free three-dimensional manner without orbit. You can shoot on the move.

According to the present invention, it is possible to acquire a panoramic image of a shooting target wider than the shooting range of the camera with an appropriate resolution.

The block diagram which shows the 1st Embodiment of an image taking apparatus, an image processing apparatus and an image processing method. The schematic diagram of the shooting operation of the same embodiment. The flowchart which shows the processing procedure of the shooting operation of the same embodiment. The flowchart which shows the shooting preparation process of the same embodiment. The flowchart which shows the shooting process of the same embodiment. The flowchart which shows the photographing target extraction processing of the same embodiment. The flowchart which shows the image creation process of the same embodiment. The flowchart which shows the pattern detection process of the same embodiment. The schematic diagram which shows the 2nd Embodiment of an image taking apparatus, an image processing apparatus and an image processing method. The schematic diagram which shows the result of the image composition and the pattern detection processing of the same embodiment. The schematic diagram which shows the image taking apparatus, the image processing apparatus and another embodiment of an image processing method.

(First Embodiment)
An embodiment of an image capturing apparatus, an image processing apparatus, and an image processing method will be described with reference to FIGS. 1 to 8. The image capturing device creates a panoramic image in which the shooting target is a position image from a plurality of shooting images obtained by shooting a shooting target that is wider than the shooting range. The panoramic image may be a printed image, or may be an image that can be displayed while scrolling all or part of it on a display screen of a computer or the like.

As shown in FIG. 1, the image capturing device is a device capable of capturing a shooting target range 11 of the shooting target 10 and creating a panoramic image of the shooting target 10.
The image capturing device includes a moving photographing device 20 that moves along the moving route 12 and a processing device 30 that acquires a photographed image of the moving photographing device 20 and creates a panoramic image.

As shown in FIG. 2, for example, the photographing device 21 transports a moving body 22 along a moving path R1 (moving route 12) defined at a predetermined distance L from the photographing target 10 at a high place of the photographing target 10 such as a building. Moving. In the present embodiment, the moving route R1 (moving route 12) is set with reference to the focal position of the photographing device 21. The photographing device 21 has a field of view E in the direction of the photographing target 10, and each time it reaches each photographing point Pm3 to Pm1, Pn, Pn1 to Pn3 on the movement path R1, the photographed image Fm3 to included in the field of view E Fm1, Fn, Fn1 to Fn3 are created. The moving body 22 is a flying body, a transport device, or the like capable of three-dimensional orbital movement or non-orbital movement in the front-rear direction such as the horizontal direction and the vertical direction. In the present embodiment, a panoramic image used for visual inspection or the like is created from the captured images Fm3 to Fm1, Fn and Fn1 to Fn3 thus created.

As shown in FIG. 1, the mobile photographing device 20 includes a photographing device 21 for photographing an image and a moving body 22 for moving the photographing device 21.
The photographing device 21 is a photographing device including a so-called digital camera such as a CCD camera or a CMOS camera. The photographing device 21 has a field of view E (see FIG. 2) capable of photographing at least a part of the photographing target range 11. The photographing device 21 has a communication device capable of various information communication with the processing device 30, can transmit a photographed image to the processing device 30, and can receive shooting conditions and the like from the processing device 30. The photographing device 21 is moved by the moving body 22 so that the unphotographed portion of the photographing target range 11 is included in the new photographing range, and a new image including the unphotographed portion is included. Can be photographed.

The moving body 22 can move while holding the photographing device 21. The moving body 22 is a flying body or a transport device, and can autonomously move on a predetermined route or an automatically determined route. The moving body 22 can move the photographing device 21 along the photographing target range 11. The moving body 22 moves the photographing device 21 so that the photographing range of the photographing device 21 includes an unphotographed portion of the photographing target range 11. The mobile body 22 has a communication device capable of various information communication with the processing device 30, and can transmit three-dimensional position information, time, movement route information, and control information to the processing device 30 from the processing device 30. It is possible to receive movement route information, movement start / stop commands, and the like.

The moving body 22 can move in three dimensions, and the moving direction includes the horizontal direction, the vertical direction, the diagonal direction which is a combination thereof, the front-rear direction orthogonal to the horizontal direction and the vertical direction, and the turning direction. At least one may be included. In the case of the moving body 22, even in a place where it is difficult to support the photographing device 21 at a predetermined distance from the photographing target by a support or the like, the distance to the photographing target is maintained at an appropriate distance to obtain a photographed image having a predetermined resolution. You will be able to create it. For example, examples of places that are difficult to support with columns or the like include high places such as buildings, columns, steel towers, slopes, and cliffs. Further, it is difficult to take a picture at an appropriate distance in a place where the distance changes frequently such as an uneven surface and a curved surface.

The processing device 30 includes an arithmetic device, a storage device, a communication device, an information input / output device, and the like. The arithmetic unit is a so-called computer, and performs various types of information processing based on the execution of each program stored in the storage device or the like. The storage device is composed of a RAM that temporarily holds the results of operations and the like, a ROM that holds programs and the like, a flash memory, and the like.

The processing device 30 includes a moving body control unit 31, an image acquisition unit 32, a shooting preparation unit 33, a shooting processing unit 34, an image processing device 35, a storage unit 36, an operation unit 37, and a display unit 38. Further, the processing device 30 includes a pattern learning unit 50 and a pattern detection unit 51.

The storage unit 36 is composed of all or a part of a storage device or the like.
The operation unit 37 is an interface for inputting setting information and operation information.
The display unit 38 is an interface that outputs image information such as a display.

The mobile body control unit 31 sets the movement path of the mobile body 22 and various information necessary for the movement, and performs various information communication with the mobile body 22 via the communication device. The mobile body control unit 31 can receive three-dimensional position information, time, movement route information, and control information from the moving body 22, and can transmit movement route information, movement start / stop commands, and the like to the moving body 22. .. The mobile control unit 31 can manage the schedule information and the actual result information. The schedule information is a list of a certain time and three-dimensional position information at that time, which is estimated based on the movement route of the moving body 22, various information necessary for the movement, and the like. The actual information is a list of the time and the three-dimensional position information at that time, which is specified based on the three-dimensional position information received from the moving body 22 and the time and the like.

The image acquisition unit 32 acquires a captured image such as a digital image captured by the photographing device 21. The image acquisition unit 32 can receive the captured image from the photographing device 21 via the communication device, and can transmit the photographing conditions and the like to the photographing device 21.

The shooting preparation unit 33 prepares data that needs to be set in the moving shooting device 20 in order to create a panoramic image to be shot. Specifically, the shooting preparation unit 33 prepares data set in the shooting device 21, data set in the moving body 22, and data necessary for the processing device 30 to be used for creating a panoramic image.

Specifically, the shooting preparation unit 33 sets various initial values required for creating a panoramic image. As various initial values, values stored in the storage unit 36 and the like are used.
Further, the shooting preparation unit 33 specifies the shooting target range 11 from the width and height of the shooting target. Further, the shooting preparation unit 33 calculates the distance L (see FIG. 2) to the shooting target range 11 where the shooting device 21 can shoot at a resolution for which the shooting target is intended. Further, the shooting preparation unit 33 calculates a shooting range from the field of view E (see FIG. 2) of the shooting device 21 and the distance to the shooting target. Then, the shooting preparation unit 33 calculates the moving route 12 so that the shooting target range 11 can be scanned within the shooting range of the shooting device 21, and also calculates the shooting point Pn or the like for executing shooting on the moving route 12. Therefore, in the moving shooting device 20, the moving body 22 moves according to the moving route 12 set by the shooting preparation unit 33, and the shooting device 21 shoots at the shooting position, so that the shooting target range is within the shooting range of the shooting device 21. 11 is scanned. As a result, each photographed image Fn or the like captured without interruption can be acquired, and a panoramic image can be created by synthesizing the acquired captured images Fn or the like. Further, since each photographed image in the photographing target range 11 is sequentially created by scanning, it is easy to calculate the relative position of each photographed image and to create a panoramic image by synthesizing each photographed image.

In addition, the shooting preparation unit 33 prepares shooting conditions such as a shutter speed and exposure to be set in the shooting device 21.
The movement path is determined to correspond to the shape of the shooting target range 11 of the shooting target, but even if the shooting target range 11 has a rectangular shape or a complicated shape, shooting during horizontal movement is performed. By combining the horizontal scanning, which is the above-mentioned horizontal scanning, and the vertical scanning, which is the photographing during the vertical movement, it is possible to correspond to various shapes of the imaging target range 11.

The shooting processing unit 34 moves the moving shooting device 20 to the start position of the moving route 12, instructs the start of shooting from the start position, and instructs the stop of shooting. The photographing processing unit 34 confirms the moving position, the photographing position, and the like with the three-dimensional position information via communication with the moving photographing device 20. Further, the photographing processing unit 34 stores the acquired three-dimensional position information at the time of creation in association with each photographed image.

The image processing device 35 performs image processing on each captured image obtained by photographing the photographing range of the photographing device 21. It should be noted that each photographed image obtained by photographing in the photographing range of the photographing apparatus 21 includes a photographed image adjusted by the photographing processing unit 34. Further, the image processing device 35 creates a panoramic image based on each captured image that has undergone image processing.

The image processing device 35 includes a target extraction unit 40 that calculates a shooting target from each shot image, a relative position calculation unit 41 that calculates the relative relationship of the shooting targets in a plurality of processed shot images, and a panoramic image of the shooting target. It is provided with a creation unit 42 for creating an image.

First, the image processing device 35 performs predetermined processing on each captured image that is sequentially selected. At this time, the image processing device 35 acquires the actual information, the schedule information, the speed vector, and the like of the moving body 22 corresponding to each captured image. For example, the image processing device 35 adjusts each photographed image based on the three-dimensional position information of the photographed image so that the distance between the imaged object and the image processing device corresponds to a predetermined predetermined distance L.

The target extraction unit 40 extracts a portion including a predetermined velocity vector from each captured image as a shooting target (target extraction step). The target extraction unit 40 calculates the speed vector of each captured image based on the velocity vector of the moving body 22 and the distance from the imaging device 21 to the imaging target. More specifically, the velocity vector of the image included in the captured image is calculated. Then, the target extraction unit 40 extracts an image in which the calculated speed vector is in the range corresponding to the speed vector estimated for the shooting target as the shooting target. That is, the target extraction unit 40 can extract the information necessary for specifying the relative position from the captured image without performing the image processing for separating the captured object 10 from each captured image Fn including the background with high accuracy. it can.

The velocity vector of the moving body 22 can be calculated based on the three-dimensional position information and the elapsed time at the two points of the moving body 22. In particular, if the change in distance to the imaged object is small, the velocity vector of the moving body 22 can be calculated from the amount of movement of the feature points included in the two captured images over the elapsed time.

The relative position calculation unit 41 calculates the relative position of each photographed image captured by the photographing device 21 (relative position calculation step).
As the first method, the relative position calculation unit 41 calculates the relative positional relationship with the adjacent captured image based on the three-dimensional position information of the captured image, and aligns the adjacent captured images. This makes it possible to combine adjacent captured images. At this time, by using the actual information, it is possible to align the adjacent captured images with high accuracy. In addition, when the actual information cannot be obtained, by using the schedule information, it is possible to align the adjacent captured images with the available accuracy, although it is lower than when the actual information is used.

Further, the relative position calculation unit 41 can align the adjacent photographed images with appropriate accuracy based on the relative positional relationship of the imaged objects extracted in each photographed image. By considering the relative positional relationship of the shooting target, the accuracy of the alignment can be further improved as compared with the alignment using only the actual information and the schedule information.

As a second method, the relative position calculation unit 41 determines the captured image in the photographing target range 11 based on the three-dimensional position of the moving body 22, the velocity vector, and the distance L from the photographing device 21 to the photographing target range 11. Identify the relative position of. That is, by specifying the moving route 12 of the photographing device 21, the relative position of each photographed image photographed on the moving route 12 can be calculated.

The creation unit 42 arranges each photographed image based on the relative position, combines them, and creates a panoramic image as a wide area image including a range wider than the photographing range (creation step). The creation unit 42 combines two adjacent captured images into one panoramic image based on the alignment performed by the relative position calculation unit 41. In addition, the creation unit 42 repeatedly synthesizes the created panoramic image and one adjacent shot image into one panoramic image, and corresponds to a size including the shooting target 10, for example, a shooting target range 11. Create a panoramic image of the size you want. The creation unit 42 may create one panoramic image from three or more captured images, or may create one panoramic image from the created panoramic image and two or more adjacent captured images. Good.

The pattern learning unit 50 learns an image pattern to be detected from the shooting target range 11. For example, image pattern learning is performed by a machine learning program. The pattern learning unit 50 acquires a large amount of teacher data and creates an image recognition engine capable of recognizing a characteristic image pattern based on the data. Here, the characteristic image pattern is a portion having a color, a shape, and a pattern that the object to be photographed usually has. Specifically, a portion having a color or pattern different from the normal color or pattern such as cracks, scratches, or rust is detected as a characteristic image pattern. Therefore, the characteristic image pattern can be extracted as a portion where it is preferable to perform a precise inspection rather than leaving it unattended in the inspection.

The pattern detection unit 51 collates the panoramic image composed of each captured image with the learning result by the pattern learning unit 50, and detects the panoramic image as the collation result together with the position information occupying the image range corresponding to the learning result. In addition, the pattern detection unit 51 creates a list of characteristic image patterns detected from each captured image and a list of positions on the panoramic image. For example, the image pattern is detected by using an image recognition program that uses the learning result of the machine learning program. The pattern detection unit 51 associates the positions of characteristic image patterns with the panoramic image based on the list of acquired collation results. In addition, the pattern detection unit 51 adds information for enhancing the recognizability of the detected image region of the characteristic image pattern. For example, the display unit 38 may be highlighted such that the image pattern corresponding to the image pattern that is the learning result is surrounded by a rectangle.

(Operation of image capturing device)
The operation of the image capturing apparatus will be described with reference to FIGS. 3 to 8.
As shown in FIG. 3, when creating a panoramic image, the processing device 30 includes a shooting preparation process (step S10), a shooting process (step S20), a shooting target identification process (step S30), and an image composition process (step S30). Step S40) and the pattern detection process (step S50) are executed.

In the shooting preparation process (step S10 in FIG. 3), the processing device 30 prepares the shooting target 10 for shooting with the mobile shooting device 20.
In the photographing process (step S20 of FIG. 3), the processing device 30 acquires and processes the photographed image. As a result, a captured image adjusted for differences in imaging conditions can be obtained.

In the shooting target extraction process (step S30 in FIG. 3), the processing device 30 extracts the shooting target from the shot image. This makes it possible to create a panoramic image to be photographed.
In the panoramic image creation process (step S40 of FIG. 3), the processing device 30 creates one or a plurality of panoramic images from the plurality of captured images, which is a number smaller than the number of captured images.

In the pattern detection process (step S50 in FIG. 3), the processing device 30 detects a characteristic image pattern included in the panoramic image and enables the display unit 38 to highlight the detected position.

More specifically, as shown in FIG. 4, when the shooting preparation process (step S10 in FIG. 3) is started, the processing device 30 sets the initial value (step S11), calculates the movement route (step S12), and sets the initial value. The shooting position calculation (step S13) and the data preparation (step S14) are executed.

In the initial value setting (step S11 in FIG. 4), the shooting preparation unit 33 of the processing device 30 prepares shooting conditions such as shutter speed and exposure.
In the movement route calculation (step S12 in FIG. 4), the shooting preparation unit 33 of the processing device 30 calculates the movement route 12 (movement route R1).

In the shooting position calculation (step S13 in FIG. 4), the shooting preparation unit 33 of the processing device 30 places the shooting points Pm3 to Pm1, Pn, Pn1 to Pn3 (see FIG. 2) on the moving route 12 (moving path R1). To set.

In the data preparation (step S14 of FIG. 4), the shooting preparation unit 33 of the processing device 30 prepares the data to be transmitted to the mobile shooting device 20 for shooting. Further, the shooting preparation unit 33 prepares to transmit the moving route 12 (moving path R1) and each shooting point Pm3 to Pm1, Pn, Pn1 to Pn3 to the moving body 22. Further, the shooting preparation unit 33 prepares for transmitting shooting conditions such as shutter speed and exposure to the shooting device 21.

Then, as shown in FIG. 5, in the processing device 30, when the shooting process (step S20 in FIG. 3) is started, the data setting (step S21), the movement start (step S22), and the shooting start (step S23) are started. ), The shooting position movement (step S24), and the shooting (step S25) are executed. Further, in the processing device 30, image reception (step S26) and end determination (step S27) are executed.

In the data setting (step S21 of FIG. 5), the shooting preparation unit 33 of the processing device 30 transmits data necessary for shooting to the mobile shooting device 20 and sets the data. The shooting preparation unit 33 of the processing device 30 transmits the moving route 12 (moving path R1) and each shooting point Pm3 to Pm1, Pn, Pn1 to Pn3 to the moving body 22 for setting. Further, the shooting preparation unit 33 of the processing device 30 transmits and sets shooting conditions such as shutter speed and exposure to the shooting device 21.

At the start of movement (step S22 in FIG. 5), the photographing processing unit 34 of the processing device 30 moves the moving photographing device 20 to the start position of the moving route 12.
At the start of shooting (step S23 in FIG. 5), the shooting processing unit 34 of the processing device 30 instructs the mobile shooting device 20 to start shooting the shooting target.

In the shooting position movement (step S24 in FIG. 5), the shooting processing unit 34 of the processing device 30 confirms that the moving shooting device 20 moves along the moving route 12 and shoots at the shooting point. At this time, the moving shooting device 20 autonomously moves based on the set moving route 12 and the shooting point. That is, the mobile photographing device 20 is moved by the moving body 22 according to the moving route 12, and the focus of the photographing device 21 is sequentially moved to the photographing point.

In shooting (step S25 in FIG. 5), the shooting processing unit 34 of the processing device 30 confirms that the shooting target is shot each time the focus of the shooting device 21 reaches the shooting point. At this time, the mobile photographing device 20 autonomously photographs an image. Further, when shooting an image, the shooting device 21 does not recognize the image to be shot, so the direction specified at each shooting point is shot with a focus corresponding to a predetermined distance L.

In the image reception (step S26 of FIG. 5), the photographing processing unit 34 of the processing device 30 acquires each photographed image photographed at each photographing point transmitted from the photographing device 21 by reception, and at the same time, obtains the three-dimensional position at the time of creation. Correspond to information.

Then, the processing device 30 determines whether or not the photographing is completed (step S27 in FIG. 5). When the moving photographing device 20 reaches the end point of the moving route 12, it is determined that the photographing is completed, and when it has not reached the end point, it is determined that the photographing is not completed.

When it is determined that the shooting is not completed (NO in step S27 in FIG. 5), the processing device 30 returns the processing to step S24 and re-executes the processing after the shooting position movement (step S24). On the other hand, when it is determined that the shooting is completed (YES in step S27 in FIG. 5), the processing device 30 ends the shooting process.

Subsequently, as shown in FIG. 6, when the photographing target extraction process (step S30 in FIG. 3) is started, the processing device 30 selects the target image (step S31) and acquires the moving object state (step S32). , The object speed calculation (step S33), the target speed image extraction (step S34), and the end determination (step S34) are executed.

In the target image selection (step S31 of FIG. 6), the image processing device 35 of the processing device 30 selects the captured image to be processed. The captured images may be selected in the order of the moving route 12, or may be selected in a different order from the moving route 12, but the captured images are selected so that the speed vector of the shooting target can be easily calculated. Is preferable.

In the moving body state acquisition (step S32 in FIG. 6), the image processing device 35 of the processing device 30 acquires the actual information, the schedule information, the speed vector, and the like of the moving body 22 corresponding to the selected captured image.

In the object speed calculation (step S33 in FIG. 6), the target extraction unit 40 of the processing device 30 includes a shooting target included in each shot image based on the acquired actual information, schedule information, speed vector, etc. of the moving body 22. Calculate the velocity vector and the like that are estimated to be possessed by a part of the captured image.

In the target speed image extraction (step S34 in FIG. 6), the target extraction unit 40 of the processing device 30 uses an image region having an estimated velocity vector from each captured image as an image region to be captured. Extract.

When the shooting target is extracted, the processing device 30 determines whether or not the target image is completed (step S35 in FIG. 6). Then, when it is determined that the target image has not been completed (NO in step S35 of FIG. 6), the processing device 30 returns the process to step S31 and re-executes the processes after the target image selection (step S31). To do. On the other hand, when it is determined that the target image is completed (YES in step S35 of FIG. 6), the processing device 30 ends the photographing target extraction process.

Further, as shown in FIG. 7, when the panoramic image creation process (step S40 in FIG. 3) is started, the processing device 30 performs target image selection (step S41), relative position calculation (step S42), and synthesis. The position calculation (step S43), the image composition (step S44), and the end determination (step S45) are executed.

In the target image selection (step S41 in FIG. 7), the relative position calculation unit 41 of the processing device 30 selects the captured image to be processed. The captured images may be selected in the order of the moving route 12, or may be selected in a different order from the moving route 12, but at first two adjacent captured images are selected, and thereafter, they are being created. It is preferable that a captured image adjacent to the panoramic image is selected.

In the relative position calculation (step S42 in FIG. 7), the relative position calculation unit 41 of the processing device 30 calculates the relative position of the selected captured image.
In the composite position calculation (step S43 in FIG. 7), the relative position calculation unit 41 of the processing device 30 aligns the adjacent captured images.

In image composition (step S44 of FIG. 7), the creating unit 42 of the processing device 30 arranges each captured image according to the alignment and creates a panoramic image as a wide area image including a range wider than the photographing range of the photographing device 21. ..

Then, the image processing device 35 determines whether or not the target image selected from the acquired captured images has been completed (step S45 in FIG. 7). When it is determined that the target image has not been completed (NO in step S45 in FIG. 7), the image processing device 35 returns the process to step S41 and re-executes the processes after the target image selection (step S41). .. On the other hand, when it is determined that the target image is completed (YES in step S45 of FIG. 7), the image processing device 35 ends the panoramic image creation process.

As a result, a panoramic image of the shooting target 10 is created from an image shot from a short distance and having a predetermined resolution.
Further, as shown in FIG. 8, when the pattern detection process (step S50 in FIG. 3) is started, the processing device 30 acquires a collation range (step S51), a collation process (step S52), and a collation result. (Step S53), collation result association (step S54), and end determination (step S55 in FIG. 8) are executed.

In the collation range acquisition (step S51 in FIG. 8), the pattern detection unit 51 of the processing device 30 acquires one or a plurality of panoramic images as the collation range.
In the collation process (step S52 in FIG. 8), the pattern detection unit 51 of the processing device 30 detects a characteristic image pattern included in the panoramic image by a program for image recognition. Characteristic image patterns are patterns such as cracks, scratches, and rust that may require detailed inspection. Further, the image pattern is specified to be in the collation range in the panoramic image, and the occupied position in the collation range is also specified.

In the collation result acquisition (step S53 of FIG. 8), the pattern detection unit 51 of the processing device 30 acquires a list of characteristic image pattern collation results from the image recognition program.
In the collation result mapping (step S54 of FIG. 8), the pattern detection unit 51 of the processing device 30 associates the position of the characteristic image pattern with the panoramic image based on the acquired list.

Then, the processing device 30 determines whether or not the collation range has been completed (step S55 in FIG. 8). When it is determined that the collation range has not been completed (NO in step S55 of FIG. 8), the pattern detection unit 51 of the processing device 30 returns the process to step S51, and processes after the collation range acquisition (step S51). Is re-executed. On the other hand, when it is determined that the collation range is completed (YES in step S55 of FIG. 8), the pattern detection unit 51 of the processing device 30 ends the panoramic image creation process.

With the moving body 22 capable of three-dimensional movement, the movement is arranged at a distance L at which a photographed image having a predetermined resolution can be obtained even in a high place to be photographed and which cannot be observed at a high resolution from the ground. It is possible to preferably observe based on a panoramic image based on the captured image of the photographing device 20. As a result, the labor required for the inspection can be reduced. In addition, proper inspection makes it possible to properly maintain the subject to be photographed. In addition, the highlighting makes it easier to recognize the detected portion and reduces the risk of oversight.

According to this embodiment, the following effects can be obtained.
(1) Since the photographing device 21 is moved by the moving body 22, images of the photographing target 10 wider than the photographing range can be obtained as each photographed image from a distance L in which an appropriate resolution is secured. Then, by arranging the captured images Fn and the like captured in this way, a panoramic image which is a wide area image having an appropriate resolution can be acquired.

Further, from each of the captured images Fn and the like captured by the photographing device 21 moved by the moving body 22, the target imaging target 10 is extracted based on having a specific velocity vector in each captured image. To. As a result, information necessary for specifying the relative position can be extracted from the photographed image without performing image processing for separating the imaged object 10 from each photographed image Fn or the like including the background with high accuracy.

(2) Information necessary for specifying the relative position is extracted based on the speed vector calculated based on the speed vector of the photographing device 21 and the distance L between the photographing target 10 and the photographing device 21.
(3) Since the moving route 12 of the photographing device 21 is specified based on the position and speed vector of the photographing device 21 and the distance L between the photographing target 10 and the photographing device 21, the image was taken on the moving route 12. The relative position of each captured image Fn or the like can be calculated.

(4) Since the shooting target range 11 is scanned in the shooting range of the shooting device 21, each shot image Fn or the like obtained by shooting the shooting target range 11 without interruption can be acquired, so that a seamless panoramic image can be created. Further, since each photographed image Fn or the like in the photographing target range 11 is photographed in order, it is easy to calculate the relative position of each photographed image Fn or the like and to create a panoramic image by synthesizing each photographed image.

(5) Since the captured images Fn and the like in the shooting target range 11 are created in a predetermined order, it is easier to calculate the relative position of the captured images and to create a panoramic image by synthesizing the captured images Fn and the like. become.

(6) By having the pattern learning unit 50 and the pattern detection unit 51, there are cracks, scratches, rusts, and other parts having a color or pattern different from the normal color or pattern in each captured image Fn or the like. The image pattern to be detected and the occupied position on the panoramic image can be detected.

(7) In a panoramic image, a portion including an image pattern to be detected can be easily recognized by highlighting.
(8) The moving body 22 captures the entire view in the horizontal direction, the vertical direction, the direction of turning around, etc. by free three-dimensional movement without orbit even if the imaging target range is larger than the imaging range in all directions. be able to.

(Second Embodiment)
With reference to FIGS. 9 and 10, a second embodiment of the image capturing apparatus, the image processing apparatus, and the image processing method will be illustrated. The present embodiment is different from the first embodiment in that the object to be photographed is the wall surface of the building. Hereinafter, the differences will be mainly described.

As shown in FIG. 9, the mobile photographing apparatus 20 has a field of view E when arranged at a distance L at which a photographed image of the wall surface 101 of the building 100 can be obtained with a predetermined resolution. The wall surface 101 of the building 100 is wider than 1 times in the horizontal direction and wider than 1 times in the vertical direction with respect to the range of the photographed image that can be taken in the field of view E. Therefore, the moving photographing device 20 moves a position at a distance L from the wall surface 101 of the building 100, and creates a plurality of captured images so that an image of the wall surface 101 of the building 100 can be obtained with a predetermined resolution. R2 is set. For example, the moving path R2 is created by the shooting preparation unit 33, and is set in the moving shooting device 20 by the moving body control unit 31.

In the movement path R2, shooting points P21, P22, P23, P24, ... Are set so that a part of the vertically adjacent captured images overlaps and the unphotographed portion is included. As a result, the wall surface 101 of the building 100 is vertically scanned in the field of view E. Further, by performing this scanning sequentially in the horizontal direction, a photographed image of the entire wall surface 101 of the building 100 is acquired, and a panoramic image of the wall surface 101 of the building 100 is created. It should be noted that the amount of movement in the horizontal direction is such that a part of adjacent captured images overlaps and an unphotographed portion is included.

That is, as shown in FIG. 10, the movement path R2 is a path that repeats vertical movement with respect to the wall surface 101 of the building 100. Then, the image processing device 35 creates a panoramic image of the wall surface 101 of the building 100 based on the photographed image taken by moving along the movement path R2.

In the panoramic image, the captured image includes an entrance structure 102, a window structure 103, and the like, in addition to the wall surface 101 of the building. Further, the panoramic image includes cracks C1, cracks C2 and cracks C3.

The pattern learning unit 50 learns the normal wall surface 101, the entrance structure 102, and the window structure 103 of the building 100 as normal patterns. On the contrary, cracks C1, cracks C2, cracks C3 and the like that may require inspection are learned as characteristic image patterns.

The pattern detection unit 51 collates the learning result of the pattern learning unit 50 with the panoramic image of the wall surface 101 of the building 100 to determine whether or not the panoramic image contains a characteristic image pattern. For example, it is detected that the panoramic image contains cracks C1, cracks C2, and cracks C3.

Further, the pattern detection unit 51 is surrounded by frames CB1, CB2, and CB3 for highlighting the detected cracks C1, cracks C2, and cracks C3, respectively. As a result, the visibility of the portion detected as a characteristic image pattern can be enhanced.

That is, the display unit 38 can display the panoramic image including the frames CB1, CB2, and CB3 that highlight the portion having the characteristic image pattern. In other words, the position of the characteristic image pattern is specified, and the specified position is highlighted by the frames CB1, CB2, and CB3 that are highlighted on the panoramic image.

Even if the wall surface 101 of the building 100 is located in a portion that cannot be observed with a high resolution from the ground, the moving photographing device 20 arranged at a distance L capable of obtaining a photographed image of the wall surface 101 of the building 100 with a predetermined resolution is photographed. It can be preferably observed by the image. As a result, the labor required for the inspection can be reduced. In addition, proper inspection makes it possible to properly maintain the subject to be photographed. In addition, the highlighting surrounded by a frame facilitates the recognition of the detected portion, reduces the risk of oversight, and improves the inspection accuracy.

According to the present embodiment, in addition to the effects described in (1) to (8) above, the following effects can be obtained.
(9) By having the pattern learning unit 50 and the pattern detection unit 51, there are characteristics such as cracks, scratches, rust, and other parts having a color or pattern different from the normal color or pattern from the captured image Fn or the like. Image pattern and its position on the panoramic image can be detected. In addition, the position of a characteristic image pattern can be specified and highlighted on the panoramic image.

Each of the above embodiments can be modified and implemented as follows. Each of the above embodiments and the following modified examples can be implemented in combination with each other within a technically consistent range.

-As shown in FIG. 11, the movement path R3 when the imaging target 110 is hemispherical will be described.
The movement path R3 is set to a distance L from the surface of the image target 110 so that each photographed image of the surface of the image target 110 can be obtained at a predetermined resolution. That is, the movement path R3 is provided in a circumferential shape along the surface of the photographing target 110.

More specifically, the surface of the photographing target 110 is wider than 1 times in the horizontal direction and also in the vertical direction with respect to the range of the photographed image that can be photographed in the field of view E when the moving photographing device 20 is arranged at the distance L. Wider than 1x.

Therefore, the mobile photographing device 20 has a circumferential movement path R3 for creating each photographed image so that an image of the photographing object 110 can be obtained with a predetermined resolution while moving a position at a distance L from the photographing object 110. Set.

In the movement path R3, shooting points P31, ..., P3n, ... Are set so that a part of the horizontally adjacent captured images overlaps and the unphotographed portion is included. As a result, the imaging target 110 is scanned horizontally in the field of view E. Further, by performing this scanning sequentially in the vertical direction, each captured image of the entire imaged object 110 is acquired, and a panoramic image of the surface of the imaged object 110 is created.

-In each of the above embodiments, a case where the moving body 22 is capable of autonomously free three-dimensional movement without an orbit is illustrated. However, the present invention is not limited to this, and the moving body may be able to autonomously move freely in two dimensions without an orbit. Further, the moving body may be capable of three-dimensional movement or two-dimensional movement of a defined path. The defined route may be a contact type guidance path such as a rail or a guide, or a non-contact type guidance path such as a laser or radio.

-In each of the above embodiments, the case where the moving body is a flying body or a transport device has been illustrated, but the moving body may include a belt conveyor or the like capable of moving the photographing device 21.
-In each of the above embodiments, the case where the processing device is provided with the operation unit 37 and the display unit 38 has been illustrated, but the present invention is not limited to this, and the operation unit 37 and the display unit 38 may be provided outside.

-In each of the above embodiments, the moving body 22 may move along an existing rail, wire, or the like.
-In the first embodiment, the target extraction unit 40 illustrates a case where a portion including a predetermined velocity vector is extracted as a shooting target from each shot image. However, the present invention is not limited to this, and image recognition may be performed for each captured image so as to extract a capture target. Then, the relative position calculation unit may add the result of image recognition to the identification of the relative position of the adjacent captured image. As a result, the accuracy of the relative position is improved, so that a panoramic image with higher accuracy can be created.

-In each of the above embodiments, the case where the photographing target 10 is an artificial object has been illustrated, but the present invention is not limited to this, and the photographing object may be a natural object.
-In each of the above embodiments, the case where the object 10 to be photographed is an example is illustrated, but the present invention is not limited to this, and any pattern, color, or the like that can be photographed by the photographing device 21 may be used.

10 ... Shooting target, 11 ... Shooting target range, 12 ... Moving route, 20 ... Moving shooting device, 21 ... Shooting device, 22 ... Moving body, 30 ... Processing device, 31 ... Moving body control unit, 32 ... Image acquisition unit, 33 ... Shooting preparation unit, 34 ... Shooting processing unit, 35 ... Image processing device, 36 ... Storage unit, 37 ... Operation unit, 38 ... Display unit, 40 ... Target extraction unit, 41 ... Relative position calculation unit, 42 ... Creation unit , 50 ... Pattern learning unit, 51 ... Pattern detection unit, 100 ... Building, 101 ... Wall surface, 102 ... Entrance structure, 103 ... Window structure, 110 ... Shooting target, C1, C2, C3 ... Crack.

Claims (11)

A shooting device that can shoot a part of the shooting target range as the shooting range,
A moving body that moves the photographing device along the photographing target range so that the unphotographed portion of the photographing target range is included in the new photographing range of the photographing device.
An image processing device that performs image processing on each shot image obtained by shooting with the shooting device, and
With
The photographing device is moved by the moving body to create a new photographed image in which the unphotographed portion is included in the new photographing range.
The image processing device includes a target extraction unit that extracts a portion of each captured image including a predetermined velocity vector as a part of the imaging target.
A relative position calculation unit that calculates the relative position of each captured image captured by the imaging device,
An image capturing apparatus including a creating unit that arranges each captured image based on the relative position and creates a wide area image including a range wider than the photographing range. The image capturing device according to claim 1, wherein the target extraction unit calculates a speed vector of each captured image based on the speed vector of the photographing device and the distance from the photographing device to the photographing target. The image capture according to claim 1 or 2, wherein the relative position calculation unit specifies a relative position of each captured image based on the position and speed vector of the imaging device and the distance from the imaging device to the imaging target range. apparatus. The image capturing apparatus according to any one of claims 1 to 3, wherein the relative position calculation unit further specifies a relative position of each captured image based on the result of image recognition processing of each captured image. The image capturing device according to any one of claims 1 to 4, wherein the moving body scans the photographing target range within the photographing range of the photographing device. The imaging apparatus according to claim 5, wherein the scanning includes horizontal scanning and vertical scanning. A pattern learning unit that learns the image pattern you want to detect from the shooting target range,
The image capturing apparatus according to any one of claims 1 to 6, further comprising a pattern detecting unit that detects an image pattern corresponding to the learning result from the captured image by collating with the learning result by the pattern learning unit. The image capturing apparatus according to claim 7, wherein the pattern detection unit highlights the detected image pattern. The imaging device according to any one of claims 1 to 8, wherein the moving body is capable of three-dimensional non-orbital movement including a vertical direction. An object for extracting a portion including a predetermined speed vector in each photographed image photographed by the photographing apparatus as a part of the photographing object based on the speed vector of the photographing device and the distance from the photographing device to the photographing target range. Extractor and
A relative position calculation unit that calculates the relative position of each captured image,
An image processing device including a creation unit that arranges each photographed image based on the relative position and creates a wide area image including a range wider than the photographing range of the photographing apparatus. An object for extracting a portion including a predetermined speed vector in each photographed image photographed by the photographing apparatus as a part of the photographing object based on the speed vector of the photographing device and the distance from the photographing device to the photographing target range. Extraction step and
A relative position calculation step for calculating the relative position of each captured image, and
An image processing method including a creation step of arranging each captured image based on the relative position to create a wide-area image including a range wider than the imaging range of the imaging device.