WO2018180214A1

WO2018180214A1 - Image processing device, camera device, and image processing method

Info

Publication number: WO2018180214A1
Application number: PCT/JP2018/008125
Authority: WO
Inventors: 浩明菊池
Original assignee: 富士フイルム株式会社
Priority date: 2017-03-28
Filing date: 2018-03-02
Publication date: 2018-10-04
Also published as: JP6712358B2; JPWO2018180214A1

Abstract

Provided are an image processing device, a camera device, and an image processing method, whereby time and costs of calculation required for corresponding point detection when synthesizing can be suppressed and a plurality of captured images efficiently stored, by calculating a spatial arrangement for a plurality of captured images on the basis of position information indicating overlapping areas without measuring imaging position by sensor type. The image processing device 400 comprises: a captured image input unit 402; an overlapping area measurement unit 404 that measures overlapping areas; a storage control unit 406 that stores position information indicating overlapping areas associated to a second image and measured, storing same in a storage unit; a spatial arrangement unit 410 that spatially arranges captured images; a corresponding point detection unit 412 that detects overlapping areas in captured images and detects corresponding points between images in the detected overlapping areas; and a synthesizing unit 414 that synthesizes a plurality of captured images on the basis of the detected corresponding points and generates a composite image.

Description

Image processing apparatus, camera apparatus, and image processing method

The present invention relates to an image processing device, a camera device, and an image processing method, and more particularly, to an image processing device, a camera device, and an image processing method in a case where a composite image is generated by synthesizing images having continuous and overlapping regions.

Conventionally, a technique for panoramic synthesis of a plurality of continuously shot images has been proposed. As one technique for efficiently synthesizing panoramic images, a technique has been used in which a photographing position where each photographed image is acquired is measured by a GPS (Global Positioning System) or a rangefinder.

For example, Patent Document 1 describes a technique for capturing a captured image for panoramic synthesis using a camera with a gyro sensor. In Patent Document 1, when the camera is moved from the first shooting scene to the adjacent second shooting scene, the movement of the camera is detected by the gyro sensor, and the correct shooting position of the second scene is indicated to the user by the indicator. A camera with a gyro sensor intended to give instructions is described.

Also, a method for efficiently performing panorama synthesis without providing a camera with position sensors and gyro sensors has been proposed.

For example, Patent Document 2 describes a technique for panoramic synthesis of a plurality of captured images. In the panoramic image synthesis method described in Patent Document 2, the similarity is evaluated based on the spatial frequency or the number of feature points for the synthesis position of the Nth and N + 1th images. When the movement amount between the first image and the (N + 1) th image cannot be calculated, the movement amount calculated from two consecutive images before the Nth image or after the N + 1th image is calculated. To perform panorama composition.

Japanese Patent Application Laid-Open No. 06-105214 JP 2011-188035 A

Here, in the case where a plurality of captured images overlap with each other, if there is information on how each captured image forms an overlapping region, the overlap of the image group composed of the plurality of captured images is spatially determined. Can be calculated.

Also, when compositing captured images having overlapping areas in time series order, corresponding points are detected in the overlapping areas, and a combining process is performed based on the result of the corresponding point detection. Therefore, if the spatial arrangement of each captured image is known, efficient image processing can be performed when combining. That is, when performing the synthesis process, if the overlapping area of each captured image is known, the corresponding points need only be detected in the overlapping area, and the processing time and the calculation cost can be suppressed.

Patent Documents

1 and 2 do not mention storing information indicating an overlapping area between captured images.

The present invention has been made in view of such circumstances, and the object thereof is not to measure the photographing position with sensors, and to spatially arrange a plurality of photographed images based on position information indicating an overlapping region. Provided are an image processing device, a camera device, and an image processing method capable of efficiently storing a plurality of captured images while suppressing the time and calculation cost required for detecting corresponding points when performing synthesis processing. It is to be.

In order to achieve the above object, an image processing apparatus according to an aspect of the present invention is a captured image input unit that sequentially inputs a plurality of captured images in time-series order used for image synthesis, and a plurality of captured images The first image and the second image before and after the chronological order of the captured image input unit for inputting a plurality of captured images having overlapping areas, and an overlapping area between the input first image and the second image An overlapping area measuring unit for measuring, a storage control unit for storing a plurality of input captured images in the storage unit, and storing in the storage unit positional information indicating the overlapping region measured in association with the second image, and a storage unit A spatial arrangement unit that spatially arranges a plurality of captured images based on a plurality of stored captured images and position information, and an overlapping region of the plurality of spatially arranged captured images is detected, and correspondence between the detected overlapping region images A corresponding point detection unit for detecting points; And a synthesis processing unit that generates a composite image by combining a plurality of photographed images based on corresponding points out.

According to this aspect, the overlap region between the first image and the second image to be input is measured by the overlap region measurement unit, and the storage controller stores position information indicating the overlap region in association with the second image. Is remembered. Thereby, the spatial arrangement of a plurality of photographed images input to the photographed image input unit is calculated without measuring the photographing position with sensors. Then, the corresponding point detection unit detects corresponding points between the images in the overlapping region based on the calculated spatial arrangement of the plurality of captured images. Therefore, according to the present aspect, the corresponding point detection processing is performed only for a limited area, so that the time required for detecting the corresponding points and the calculation cost required for detecting the corresponding points can be suppressed. Further, according to this aspect, the position information indicating the overlapping area is also stored in the storage unit, so that the captured image can be efficiently stored in the storage unit.

An image processing apparatus according to another aspect of the present invention is a photographed image input unit that sequentially inputs a plurality of photographed images in time-series order used for image synthesis, and includes a plurality of photographed images in time-series order. A captured image input unit that inputs a plurality of captured images in which the front and back first images and the second image have overlapping areas; a first image cutout unit that extracts the first image from the plurality of input captured images; When an overlap region measurement unit that measures an overlap region between the cut out first image and the plurality of input captured images, and when the measurement result of the overlap region measurement unit becomes a default value, the second is extracted from the plurality of captured images. A second image cutout unit that cuts out an image; a storage control unit that stores a plurality of input captured images in a storage unit; and a storage control unit that stores position information indicating an overlapping region measured in association with the second image; Multiple captured images and positions stored in the storage unit A spatial arrangement unit that spatially arranges a plurality of captured images based on the information, a corresponding point detection unit that detects an overlapping region of the plurality of spatially arranged captured images, and detects corresponding points between the images of the detected overlapping region; A combining processing unit that combines a plurality of captured images based on the detected corresponding points to generate a combined image.

According to this aspect, the overlapping area measurement unit measures the overlapping area between the cut-out first image and the plurality of input captured images, and the storage control unit indicates the overlapping area in association with the second image. Information is stored in the storage unit. Thereby, the spatial arrangement of a plurality of photographed images input to the photographed image input unit is calculated without measuring the photographing position with sensors. Then, the corresponding point detection unit detects corresponding points between the images in the overlapping region based on the calculated spatial arrangement of the plurality of captured images. Therefore, according to the present aspect, the corresponding point detection processing is performed only for a limited area, so that the time required for detecting the corresponding points and the calculation cost required for detecting the corresponding points can be suppressed. Further, according to this aspect, the position information indicating the overlapping area is also stored in the storage unit, so that the captured image can be efficiently stored in the storage unit.

An image processing apparatus according to another aspect of the present invention is a photographed image input unit that sequentially inputs a plurality of photographed images in time-series order used for image synthesis, and includes a plurality of photographed images in time-series order. A captured image input unit that inputs a plurality of captured images in which the preceding and following first images and the second image have overlapping regions; an overlapping region measuring unit that measures an overlapping region between the input first image and second image; When the measurement result of the overlapping area measurement unit becomes a default value, the image extraction unit that extracts the first image and the second image, the extracted first image and the second image are stored in the storage unit, and A storage control unit that stores position information indicating an overlap area measured in association with the second image in the storage unit, and a space in which a plurality of captured images are spatially arranged based on the plurality of captured images and position information stored in the storage unit Overlapping between the arrangement part and multiple shot images arranged in space A corresponding point detecting unit that detects a corresponding area between images of the detected overlapping region, and a combining processing unit that combines a plurality of captured images based on the detected corresponding points to generate a combined image, Prepare.

According to this aspect, the overlap region between the first image and the second image to be input is measured by the overlap region measurement unit, and the position information indicating the overlap region measured in association with the second image is stored by the storage control unit. Stored in the department. Thereby, the spatial arrangement of a plurality of photographed images input to the photographed image input unit is calculated without measuring the photographing position with sensors. Then, the corresponding point detection unit detects corresponding points between the images in the overlapping region based on the calculated spatial arrangement of the plurality of captured images. Therefore, according to the present aspect, the corresponding point detection processing is performed only for a limited area, so that the time required for detecting the corresponding points and the calculation cost required for detecting the corresponding points can be suppressed. Further, according to this aspect, the position information indicating the overlapping area is also stored in the storage unit, so that the captured image can be efficiently stored in the storage unit.

Preferably, the compositing processing unit generates an ortho image by compositing.

According to this aspect, since the composition processing unit generates an ortho image by compositing, it is possible to suppress time and calculation cost for detecting corresponding points of overlapping regions in the generation of the ortho image.

Preferably, the storage control unit records position information in an EXIF tag attached to the second image.

According to this aspect, since the storage controller records the position information on the EXIF tag attached to the second image, the position information can be efficiently stored in association with the second image.

Preferably, the corresponding point detection unit also detects an overlapping area of the first image with an image other than the second image, and detects a corresponding point between the images of the detected overlapping area.

According to this aspect, the corresponding point detection unit also detects an overlapping area of the first image with an image other than the second image, and detects a corresponding point between the images of the detected overlapping area. As a result, in this aspect, since the combining process is performed with more corresponding points than the combining process using only the corresponding points of the overlapping region of the first image and the second image, it is possible to perform more accurate combining. .

Preferably, the position information indicating the overlapping area is the first coordinates of the first image and the second coordinates of the second image corresponding to the first image.

According to this aspect, since the position information indicating the overlapping area is the first coordinates of the first image and the second coordinates of the second image corresponding to the first image, more accurate position information of the overlapping area is stored in the storage unit. Can be memorized.

Preferably, the position information indicating the overlapping region is two or less sets of first coordinates and second coordinates.

According to this aspect, since the position information indicating the overlapping area is the first coordinate and the second coordinate of two sets or less, the information amount of the position information is suppressed, and the storage capacity of the storage unit is effectively reduced. Can be used.

A camera apparatus according to another aspect of the present invention is a captured image acquisition unit that sequentially acquires a plurality of captured images in time-series order used for image synthesis, and before and after a time-series order among a plurality of captured images. The first image and the second image of the first image and the second image overlap region measurement that measures an overlap region between a captured image acquisition unit that acquires a plurality of captured images having overlapping regions and a moving image captured by the first image and the captured image acquisition unit. And when the measurement result of the overlapping area measurement unit becomes a default value, the shooting control unit that causes the captured image acquisition unit to acquire the second image, the acquired plurality of captured images to be stored in the storage unit, and the first A storage control unit that stores position information indicating an overlap area measured in association with two images in a storage unit, and a spatial arrangement that spatially arranges a plurality of captured images based on the plurality of captured images and position information stored in the storage unit And a plurality of spatially arranged Corresponding point detection unit that detects overlapping regions of shadow images, detects corresponding points between images of the detected overlapping regions, and a combining process that generates a combined image by combining a plurality of captured images based on the detected corresponding points A section.

According to this aspect, the overlapping region measurement unit measures the overlapping region between the first image and the moving image captured by the captured image acquisition unit, and the storage control unit indicates the overlapping region measured in association with the second image. Information is stored in the storage unit. Thereby, the spatial arrangement of a plurality of photographed images input to the photographed image input unit is calculated without measuring the photographing position with sensors. Then, the corresponding point detection unit detects corresponding points between the images in the overlapping region based on the calculated spatial arrangement of the plurality of captured images. Therefore, according to the present aspect, the corresponding point detection processing is performed only for a limited area, so that the time required for detecting the corresponding points and the calculation cost required for detecting the corresponding points can be suppressed. Further, according to this aspect, the position information indicating the overlapping area is also stored in the storage unit, so that the captured image can be efficiently stored in the storage unit.

An image processing method according to another aspect of the present invention is a step of sequentially inputting a plurality of photographed images in time-series order used for image synthesis, wherein The step of inputting a plurality of captured images having an overlapping area between one image and the second image, the step of measuring an overlapping area between the first image and the second image to be input, and storing the plurality of input captured images A step of storing in the storage unit position information indicating an overlap region measured in association with the second image and a plurality of photographed images based on the plurality of photographed images and position information stored in the storage unit. A step of spatial arrangement, a step of detecting overlapping regions of a plurality of captured images arranged in space, a step of detecting corresponding points between the images of the detected overlapping regions, and a combination of the plurality of captured images based on the detected corresponding points Together And generating an image.

An image processing method according to another aspect of the present invention is a step of sequentially inputting a plurality of photographed images in time-series order used for image synthesis, wherein The step of inputting a plurality of captured images having an overlapping area between the one image and the second image, the step of cutting out the first image from the plurality of input captured images, and the plurality of the input of the cut out first image A step of measuring an overlapping area with the captured image, a step of cutting out the second image from the plurality of captured images when the measurement result of the step of measuring the overlapping area is a default value, and the plurality of input captured images Is stored in the storage unit, and the storage unit stores position information indicating the overlap region measured in association with the second image, and a plurality of captured images and a plurality of position information are stored based on the plurality of captured images stored in the storage unit. A step of spatially arranging shadow images, a step of detecting overlapping regions of a plurality of spatially arranged captured images, a step of detecting corresponding points between the images of the detected overlapping regions, and a plurality of photographing based on the detected corresponding points Synthesizing images to generate a composite image.

An image processing method according to another aspect of the present invention is a step of sequentially inputting a plurality of photographed images in time-series order used for image synthesis, wherein Measurement of a step of inputting a plurality of captured images having an overlapping area between one image and a second image, a step of measuring an overlapping area between the input first image and the second image, and a step of measuring the overlapping area A step of extracting the first image and the second image when the result has become a predetermined value, and storing the extracted first image and the second image in the storage unit and measuring in association with the second image A step of storing position information indicating a region in the storage unit, a step of spatially arranging a plurality of photographed images based on the plurality of photographed images stored in the storage unit and the position information, and a plurality of photographed images arranged in space Detect overlapping areas And detecting corresponding points between the images of the detected overlapped area, the steps of combining a plurality of captured images to generate a composite image based on the detected corresponding points, a.

According to the present invention, the overlapping area between the input first image and the second image is measured by the overlapping area measuring unit, and the storage control unit stores the positional information indicating the overlapping area in association with the second image. Thus, the spatial arrangement of the plurality of photographed images input to the photographed image input unit is calculated without measuring the photographing position with sensors, and the corresponding point detection unit calculates Corresponding points between images in the overlapping region are detected based on the spatial arrangement of the plurality of captured images. Therefore, in this aspect, the corresponding point detection process is performed only in a limited area, so that the time required for detecting the corresponding point and the calculation cost required for detecting the corresponding point can be suppressed, and the position indicating the overlapping area Since the information is also stored in the storage unit, the captured image can be efficiently stored in the storage unit.

It is a perspective view which shows the structure of the bridge which is one of the structures which are imaging | photography objects. It is a conceptual diagram which shows an imaging | photography system. It is a perspective view which shows the external appearance of a robot apparatus. It is principal part sectional drawing of the robot apparatus shown in FIG. It is a figure which shows a camera. It is a block diagram which shows the function structural example of a robot apparatus. It is a block diagram which shows the system configuration | structure of a computer. It is a figure which shows the function structural example of an image processing apparatus. It is a figure which shows notionally the image input into a picked-up image input part. It is a figure explaining the measurement of an overlap area. The data structural example of a memory | storage part is shown. It is a figure explaining that the positional information on an overlap area | region is memorize | stored in an EXIF tag. It is a conceptual diagram when the image is spatially arranged based on the position information of the overlapping area. It is the figure which showed the overlap of the image after space arrangement | positioning. 3 is a flowchart illustrating an operation of the image processing apparatus and an image processing method. It is a figure which shows the function structural example of an image processing apparatus. 3 is a flowchart illustrating an operation of the image processing apparatus and an image processing method. It is a figure which shows the function structural example of an image processing apparatus. 3 is a flowchart illustrating an operation of the image processing apparatus and an image processing method. It is a figure which shows the function structural example of an image processing apparatus.

Hereinafter, preferred embodiments of an image processing device, a camera device, and an image processing method according to the present invention will be described with reference to the accompanying drawings.

FIG. 1 is a perspective view showing the structure of a bridge, which is one of the structures to be photographed, and is a perspective view of the bridge as viewed from below. The object to be photographed used in the present invention is not particularly limited. For example, the present invention is applied to the case of synthesizing a photographed image that is used when a structure damage detection inspection is performed. Is done. Specifically, in the case where the damage detection inspection of the floor slab 6 is performed in the space unit, the present invention is applied to the case where the floor slab 6 in the space is dividedly photographed and the captured images obtained by the divided photographing are combined. .

The bridge 1 shown in FIG. 1 has a main girder 2, a cross girder 3, a counter-tilt 4 and a horizontal 5 which are connected by bolts, rivets or welding. In addition, a floor slab 6 for driving a vehicle or the like is provided on the upper portion of the main beam 2 or the like. The floor slab 6 is generally made of reinforced concrete.

The main girder 2 is a member that is passed between the abutment or the pier and supports the load of the vehicle on the floor slab 6. The cross beam 3 is a member that connects the main beams 2 in order to support the load by the plurality of main beams 2. The anti-tilt frame 4 and the horizontal frame 5 are members for connecting the main girders 2 to each other in order to resist lateral loads of wind and earthquake, respectively.

FIG. 2 is a conceptual diagram showing a photographing system.

The imaging system 500 includes a robot apparatus 100 and a computer 300.

The robot apparatus 100 includes an imaging apparatus 200, and moves and captures images with the imaging apparatus 200 under the control of the computer 300. The robot apparatus 100 is not particularly limited as long as it can move and shoot under the control of the computer 300. Other examples of the robot apparatus 100 include an apparatus called a traveling robot, a small helicopter, a multicopter, a drone, or UAV (Unmanned Aerial Vehicles).

The computer 300 and the robot apparatus 100 can communicate with each other, and the computer 300 can remotely control the movement of the robot apparatus 100 and the imaging control of the imaging apparatus 200.

FIG. 3 is a perspective view showing an appearance of the robot apparatus 100 including the camera which is an embodiment of the imaging apparatus 200, and shows a state where the robot apparatus 100 is installed between the main girders 2 of the bridge 1. FIG. FIG. 4 is a cross-sectional view of a main part of the robot apparatus 100 shown in FIG.

As shown in FIGS. 3 and 4, the robot apparatus 100 includes an imaging apparatus 200, controls the position (imaging position) of the imaging apparatus 200 in the three-dimensional space, controls the imaging direction of the imaging apparatus 200, and bridges An arbitrary inspection member or the like of the bridge 1 composed of a plurality of members at the time of the inspection of 1 is photographed.

As will be described in detail later, the robot apparatus 100 includes a main frame 102, a vertical extension arm 104, a casing 106 in which a drive unit and various control units for the vertical extension arm 104 are disposed, and a casing 106. The X direction driving unit 108 (FIG. 6) moves in the longitudinal direction of the frame 102 (direction orthogonal to the longitudinal direction of the main beam 2) (X direction), and the longitudinal direction of the main beam 2 (Y direction). And a Y-direction drive unit 110 (FIG. 6) that moves in the vertical direction and a Z-direction drive unit 112 (FIG. 6) that expands and contracts the vertical extension arm 104 in the vertical direction (Z direction).

The X-direction drive unit 108 includes a ball screw 108A disposed in the longitudinal direction (X direction) of the main frame 102, a ball nut 108B disposed in the housing 106, and a motor 108C that rotates the ball screw 108A. The casing 106 is moved in the X direction by rotating the ball screw 108A forward or backward by the motor 108C.

The Y-direction drive unit 110 includes

tires

110A and 110B disposed at both ends of the main frame 102, and motors (not shown) disposed in the

tires

110A and 110B. By driving the motor, the entire robot apparatus 100 is moved in the Y direction.

The robot apparatus 100 is installed in such a manner that the

tires

110A and 110B at both ends of the main frame 102 are placed on the lower flanges of the two main girders 2 and the main girders 2 are sandwiched between them. Thereby, the robot apparatus 100 can be suspended (suspended) along the main girder 2 by being suspended from the lower flange of the main girder 2. Although not shown, the main frame 102 is configured such that its length can be adjusted according to the interval between the main girders 2.

The vertical extension arm 104 is disposed in the housing 106 of the robot apparatus 100 and moves in the X direction and the Y direction together with the housing 106. Further, the vertical extension arm 104 is expanded and contracted in the Z direction by a Z direction driving unit 112 (FIG. 6) provided in the housing 106.

As shown in FIG. 5, a camera installation unit 104A is provided at the tip of the vertical extension arm 104, and a camera 202 that can be rotated in the pan direction and the tilt direction by the pan / tilt mechanism 120 is installed in the camera installation unit 104A. ing.

* Acquire the image (photographed image) to be captured as an “inspection image” attached to the inspection report.

Further, the camera 202 is rotated around a pan axis P coaxial with the vertical extension arm 104 by a pan / tilt mechanism 120 to which a driving force is applied from a pan / tilt driving unit 206 (FIG. 6) or centered on a horizontal tilt axis T. Rotate. As a result, the camera 202 can shoot in an arbitrary posture (shoot in an arbitrary shooting direction).

FIG. 6 is a block diagram illustrating a functional configuration example of the robot apparatus 100.

As shown in FIG. 6, the robot apparatus 100 includes a robot control unit 130, an X-direction drive unit 108, a Y-direction drive unit 110, and a Z-direction drive unit 112 on the robot apparatus 100 side, a camera 202 on the imaging apparatus 200 side, and imaging control. The unit 204, the pan / tilt control unit 210, the pan / tilt driving unit 206, and the robot side communication unit 230 are configured.

The robot-side communication unit 230 performs two-way wireless communication with the computer 300, a movement command for controlling movement of the robot apparatus 100 transmitted from the computer 300, a pan / tilt command for controlling the pan / tilt mechanism 120, and a camera 202. Various commands such as a shooting command to be controlled are received, and the received commands are output to corresponding control units. Details of the computer 300 will be described later.

The robot control unit 130 controls the X direction driving unit 108, the Y direction driving unit 110, and the Z direction driving unit 112 based on the movement command input from the robot side communication unit 230, and the robot device 100 performs the X direction and Y direction. And the vertical extension arm 104 is expanded and contracted in the Z direction (see FIG. 3).

The pan / tilt control unit 210 operates the pan / tilt mechanism 120 in the pan direction and the tilt direction via the pan / tilt driving unit 206 based on the pan / tilt command input from the robot side communication unit 230 to pan / tilt the camera 202 in a desired direction (see FIG. 5).

The imaging control unit 204 causes the imaging unit 202A of the camera 202 to capture a live view image or an inspection image based on the imaging command input from the robot side communication unit 230. As the captured image of the present invention, a moving image including a captured image or a live view image acquired by the imaging unit 202A is used.

Image data indicating an image K taken by the imaging unit 202A of the camera 202 when the bridge 1 is inspected is transmitted to the computer 300 via the robot side communication unit 230.

The computer 300 is composed of a tablet computer as shown in FIG. The computer 300 includes a flat casing having a rectangular outline, and includes a touch panel display 302 that also serves as a display unit 326 (FIG. 6) and an input unit 328 (FIG. 6).

FIG. 7 is a block diagram showing a system configuration of the computer 300.

As shown in FIG. 7, the computer 300 includes a CPU (Central Processing Unit) 310 that controls the overall operation of the computer 300, and a main memory 314, a nonvolatile memory 316, a mobile device via the system bus 312. A communication unit 318, a wireless LAN communication unit 320, a short-range wireless communication unit 322, a wired communication unit 324, a display unit 326, an input unit 328, a key input unit 330, an audio processing unit 332, an image processing unit 334, and the like are connected. Is done.

The CPU 310 reads out an operation program (OS (Operating System) and application program that operates on the OS), standard data, and the like stored in the non-volatile memory 316, expands them in the main memory 314, and performs this operation. By executing the program, it functions as a control unit that controls the operation of the entire computer.

The main memory 314 is composed of, for example, a RAM (Random Access Memory) and functions as a work memory for the CPU 310.

The non-volatile memory 316 is composed of, for example, a flash EEPROM (EEPROM: Electrically Erasable Programmable Read Only Memory), and stores the above-described operation program and various fixed data. The nonvolatile memory 316 functions as a storage unit of the computer 300 and stores various data.

The mobile communication unit 318 is a third-generation mobile communication system compliant with the IMT-2000 standard (International Mobile Telecommunication-2000) and a fourth-generation mobile communication system compliant with the IMT-Advanced standard (International Mobile Telecommunications-Advanced). Based on this, data is transmitted / received to / from the nearest base station (not shown) via the antenna 318A.

The wireless LAN communication unit 320 uses a predetermined wireless LAN communication standard (for example, IEEE802.11a / b / g / n standard) between the wireless LAN access point and an external device capable of wireless LAN communication via the antenna 320A. ) To perform wireless LAN communication.

The short-range wireless communication unit 322 transmits / receives data to / from other Bluetooth (registered trademark) equipment within the range of, for example, class 2 (within a radius of about 10 m) via the antenna 322A.

The wired communication unit 324 performs communication according to a predetermined communication standard with an external device connected by a cable via the external connection terminal 306. For example, USB (Universal Serial Bus) communication is performed.

The display unit 326 includes a color LCD (liquid crystal display) panel that constitutes a display portion of the touch panel display 302 and a driving circuit thereof, and displays various images.

The input unit 328 constitutes a touch panel portion of the touch panel display 302. The input unit 328 is configured integrally with the color LCD panel using a transparent electrode.

The key input unit 330 includes a plurality of operation buttons provided on the casing of the computer 300 and a drive circuit thereof.

The audio processing unit 332 converts the digital audio data provided via the system bus 312 into an analog signal and outputs it from the speaker 304.

The image processing unit 334 digitizes an analog image signal output from the built-in camera 305 including a photographing lens and an image sensor, performs necessary signal processing, and outputs it.

<First Embodiment>
FIG. 8 is a diagram illustrating a functional configuration example of the image processing apparatus 400 according to the first embodiment. Note that the image processing apparatus 400 is provided in the computer 300.

The image processing apparatus 400 includes a captured image input unit 402, an overlapping area measurement unit 404, a storage control unit 406, a storage unit 408, a space arrangement unit 410, a corresponding point detection unit 412, and a synthesis processing unit 414.

The photographed image input unit 402 is sequentially input with a plurality of photographed images in time series order used for image synthesis, and before and after the first image and the next image (first image) before and after the time series order among the plurality of photographed images. With the second image), a plurality of captured images having overlapping areas are input. In other words, the photographed image input unit 402 receives a plurality of photographed images constituting a composite image, and a photographed image having an overlapping area between the previous image and the subsequent image.

FIG. 9 is a diagram conceptually showing a captured image input to the captured image input unit 402. The eight captured images from the image 601 to the image 608 constitute a composite image and are captured from the image 601 to the image 608 in chronological order. Further, the image 601 to the image 608 have overlapping areas with each other, and a part of the front image and a part of the rear image overlap each other. That is, in the image 601 and the image 602, the image 601 is the previous image, the image 602 is the subsequent image, the image 601 and the image 602 have overlapping areas, and in the image 602 and the image 603, the image 602 is the previous image, Reference numeral 603 denotes a rear image, and the image 602 and the image 603 have a relationship having an overlapping area. A similar relationship is established from image 604 to image 608. In addition, the area | region of the dotted line in a figure is an overlapping area | region, and the arrow has shown the overlapping relationship.

The overlapping area measurement unit 404 measures an overlapping area between the previous image and the subsequent image input to the captured image input unit 402. Specifically, the overlapping area measurement unit 404 measures information indicating the size of the overlapping area, such as the area of the overlapping area with the previous image or the length of the overlapping area in the subsequent image. In addition, the overlapping area measurement unit 404 calculates position information indicating the overlapping area together with the measurement of the overlapping area.

FIG. 10 is a diagram for explaining the measurement of the overlapping area of the front image U and the rear image V. FIG.

The front image U and the rear image V have an overlapping area. In the front image U, the overlapping area P1 overlaps the rear image V, and in the rear image V, the overlapping area P2 overlaps the front image U. In the front image U and the rear image V, the subject H, the subject I, and the subject J are shown as a common subject. The overlapping area measuring unit 404 detects a pair of feature point coordinates between the previous image U and the subsequent image V, for example, and determines overlapping areas, and measures the overlapping areas P1 and P2. Specifically, the overlapping area measurement unit 404 extracts feature points O5 to O8 in the subsequent image, and detects feature points O1 to O4 corresponding to the feature points O5 to O8, respectively, from the previous image. Among the detected overlapping portions, for example, overlapping portions that are equal to or greater than the threshold may be adopted by template matching. The overlapping area measuring unit 404 measures the size (area, length) of the overlapping area based on the feature points detected in this way, or obtains information (coordinates) indicating the position of the overlapping area in each image. Or measure.

The storage control unit 406 stores a plurality of input captured images in the storage unit 408, and causes the storage unit 408 to store position information indicating an overlapping area measured in association with the second image. Specifically, the image 608 to the image 608 shown in FIG. 9 are stored in the storage unit 408, and the position information indicating the overlapping area is stored in association with the subsequent image.

FIG. 11 shows a data configuration example of the storage unit 408. Note that FIG. 11 describes the image 601, the image 602, and the image 603, and other images are also stored in the same manner.

FIG. 11 (A) shows an example of the storage data structure of the shooting file name. For example, the image 601 is stored in the storage unit 408 with the shooting file name “filename1”, the image 602 is “filename2”, and the image 603 is “filename3”.

FIG. 11B shows an example of the storage data structure of the previous image index having an overlapping area. Since the image 601 is the foremost image, there is no previous image with an overlapping area, and “NONE” is stored, and since the image 602 has an overlapping area with the image 601, “1” indicating the image 601 is stored. Since the image 603 has an overlapping area with the image 602, “2” indicating the image 602 is stored in the storage unit 408.

FIG. 11C shows an example of the storage data configuration of the pixels of the previous image. Since the image 601 is the foremost image, the information of the pixels of the previous image having the overlapping area is stored as “NONE”, and the image 602 has the overlapping area with the image 601, so the pixels of the previous image (image 601). The information “(X11, Y11),... (X1n, Y1n)” is stored. Since the image 603 has an overlapping area with the image 602, the pixel information “(X21, Y21),... (X2n, Y2n)” of the previous image (image 602) is stored.

FIG. 11 (D) shows a configuration example of storage data of a corresponding pixel. Since the image 601 is the foremost image, the corresponding pixel information is “NONE”, and the image 602 is the pixel information “(X21-I) corresponding to the pixel shown in the part (C) of FIG. , Y21-I),... (X2n-I, Y2n-I) "is stored, and the image 603 contains the corresponding pixels" (X31-I, Y31-I), ... (X3n-I, Y3n-I) "is stored.

In the example shown in FIG. 11, as position information indicating the overlapping area, information on the pixels in the previous image (part (C) in FIG. 11) and information on the pixels corresponding to the pixels in the previous image (part (D) in FIG. 11). Is remembered. That is, position information first coordinates (part (C) in FIG. 11) indicating the overlapping area and second coordinates (part (D) in FIG. 11) of the rear image corresponding to the previous image are stored. In addition, it is preferable that the positional information which shows an overlapping area | region is 1st coordinate and 2nd coordinate of 2 sets or less. That is, it is sufficient that the position information indicating the overlapping area has a role as outline information. Here, the outline information is rough information on overlapping images, and information having a small capacity and easy calculation is preferable. . It should be noted that information forms other than the information forms described above are also employed as position information indicating overlapping areas. In this way, by storing the positional information indicating the overlapping area in association with the rear image and the rear image in the storage unit, the captured images constituting the image group are associated with each other, and the management of the files of the image group is made more efficient. To do. Further, by storing the position information indicating the overlapping area in association with the rear image and the rear image in this manner, it is not necessary to measure the shooting position with the GPS or the rangefinder.

FIG. 12 is a diagram for explaining a case where the position information of the overlapping area is stored in an EXIF (Exchangeable image file format) tag attached to the subsequent image.

As shown in FIG. 12, for example, the location information of the overlapping area is stored in the “MakerNote” part of the “EXIF IFD” of the EXIF tag. That is, the previous image file name having the overlapping area is stored in the place of “MakerNote”, and the coordinates of the pixel of the previous image and the coordinates of the corresponding pixel are stored. Specifically, the name of the previous image file with the overlapping area, the pixel (coordinate) of the previous image, and the corresponding pixel (coordinate) are stored in the “MakerNote” portion of the EXIF tag.

The space arrangement unit 410 spatially arranges a plurality of photographed images based on the plurality of photographed images and position information stored in the storage unit 408. That is, the space arrangement unit 410 arranges the captured image in space based on the position information of the overlapping area that is the outline information.

FIG. 13 is a conceptual diagram when images 601 to 608 are spatially arranged based on the position information of the overlapping area.

Each image from image 601 to image 608 is overlapped by overlapping region 480, overlapping region 482, overlapping region 484, overlapping region 486, overlapping region 488, overlapping region 490, and overlapping region 492. In the storage unit 408, positional information regarding the overlapping area 480, the overlapping area 482, the overlapping area 484, the overlapping area 486, the overlapping area 488, the overlapping area 490, and the overlapping area 492 is stored in association with the image to be the subsequent image. . The spatial arrangement unit 410 spatially arranges each captured image based on the position information indicating each overlapping area stored in the storage unit 408. Specifically, the spatial arrangement unit 410 spatially arranges the image 601 and the image 602 based on position information indicating the overlapping area 480 of the image 602. Similarly, the spatial arrangement unit 410 spatially arranges the image 602 and the image 603, the image 603 and the image 604, the image 604 and the image 605, the image 605 and the image 606, the image 606 and the image 607, and the image 607 and the image 608. . The spatial arrangement unit 410 can calculate the spatial arrangement of the entire photographed image input to the photographed image input unit 402 by sequentially arranging each image spatially.

Corresponding point detection unit 412 detects overlapping areas of a plurality of spatially arranged captured images, and detects corresponding points between the images of the detected overlapping areas. That is, the corresponding point detection unit 412 detects corresponding points in the overlapping area based on the position information indicating the overlapping area or the overlapping area newly generated as a result of spatial arrangement with the overlapping area based on the position information indicating the overlapping area. . Here, a known method is employed as the method for detecting the corresponding points. For example, as a method for detecting corresponding points, robust local features that are robust to scaling, rotation, and illumination changes between images, SIFT (Scale-invariant feature transform) features, SURF (Speed-Upped Robust Feature) features Those using quantities and AKAZE (Accelerated KAZE) features are known.

FIG. 14 is a diagram showing the overlap of the image 601 to the image 608 after the spatial arrangement, and shows that a new overlapping area occurs after the spatial arrangement is performed using the position information of the overlapping area. FIG. That is, the corresponding point detection unit 412 detects an overlapping area between the previous image and the subsequent image and an overlapping area between the previous image and an image other than the subsequent image, and detects corresponding points between the images in the detected overlapping area.

Specifically, the corresponding point detection unit 412 not only performs corresponding point detection on the overlapping area 480 with the image 601 and the overlapping area 482 with the image 603 in the image 602, but also with the overlapping area 494 with the image 607, and the image Corresponding point detection is also performed for the overlapping region 495 with 608. As a result, more corresponding points can be detected than the corresponding points detected between the image 601 and the image 602 and between the image 602 and the image 603.

Also, the corresponding point detection unit 412 not only performs corresponding point detection on the overlapping region 480 with the image 602 in the image 601, but also performs corresponding point detection on the overlapping region 496 with the image 608. As a result, more corresponding points can be detected than the corresponding points between the image 601 and the image 602.

The composition processing unit 414 generates a composite image by combining a plurality of captured images based on the detected corresponding points. The composition processing unit 414 can generate various composite images by combining a plurality of images. For example, the composition processing unit 414 generates a panorama composite image and an ortho image. Here, the ortho image is an image obtained by converting an aerial photograph into an image having no positional deviation and giving correct positional information.

FIG. 15 is a flowchart showing the operation and the image processing method of the image processing apparatus 400 of the present embodiment.

First, a plurality of photographed images in time series used for composition are input by the photographed image input unit 402 (step S10). For example, a captured image input unit 402 realized by the CPU 310 of the computer 300 via the mobile communication unit 318, the wireless LAN communication unit 320, or the short-range wireless communication unit 322 of the computer 300 for the image group captured by the robot apparatus 100. Is input.

Next, the overlapping area of each image is measured by the overlapping area measuring unit 404 (step S11). For example, the overlapping area measurement unit 404 is realized by the CPU 310 of the computer 300. Thereafter, the captured image input by the captured image input unit 402 and the position information measured by the overlapping region measuring unit 404 are realized in the main memory 314 of the computer 300 by the storage control unit 406 realized by the CPU 310 of the computer 300. Is stored in the storage unit 408 (step S12).

Next, the spatial arrangement unit 410 realized by the CPU 310 of the computer 300 arranges the captured image in a spatial manner based on the position information of the overlapping area, which is the outline information (step S13). Thereafter, the corresponding point detection unit 412 realized by the CPU 310 of the computer 300 detects the corresponding point in the overlapping region (step S14). The overlapping area after the captured image is spatially arranged is not only the overlapping area of the relationship between the previous image and the subsequent image, but also the overlapping area with an image that is not in the context of the image (for example, the overlapping area 494 to the overlapping area shown in FIG. 14). Corresponding points can also be detected in the region 496).

After that, the composite processing unit 414 realized by the CPU 310 of the computer 300 generates a composite image based on the detected corresponding points (step S15).

In the above-described embodiment, the hardware structure of a processing unit (processing unit) that executes various processes is the following various processors. The various processors are programmable processors that can change the circuit configuration after manufacturing, such as a CPU or FPGA (Field Programmable Gate Gate Array) that is a general-purpose processor that functions as various processing units by executing software (programs). Examples include a dedicated electric circuit which is a processor having a circuit configuration specifically designed to execute a specific process such as a logic device (Programmable Logic Device: PLD) and an ASIC (Application Specific Integrated Circuit).

One processing unit may be configured by one of these various processors, or may be configured by two or more processors of the same type or different types (for example, a plurality of FPGAs or a combination of CPU and FPGA). May be. Further, the plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with one processor, first, as represented by a computer such as a client or server, one processor is configured with a combination of one or more CPUs and software. There is a form in which the processor functions as a plurality of processing units. Second, as represented by a system-on-chip (SoC), a form of using a processor that realizes the functions of the entire system including a plurality of processing units with a single IC (integrated circuit) chip. is there. As described above, various processing units are configured using one or more of the various processors as a hardware structure.

Further, the hardware structure of these various processors is more specifically an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

The above-described configurations and functions can be appropriately realized by arbitrary hardware, software, or a combination of both. For example, for a program that causes a computer to execute the above-described processing steps (processing procedure), a computer-readable recording medium (non-transitory recording medium) that records such a program, or a computer that can install such a program However, the present invention can be applied.

<Second Embodiment>
Next, a second embodiment will be described. In the present embodiment, a moving image is input to the captured image input unit 402.

FIG. 16 is a diagram illustrating a functional configuration example of the image processing apparatus 400 according to the second embodiment. In addition, the part already demonstrated in FIG. 8 attaches | subjects the same code | symbol, and abbreviate | omits description.

The image processing apparatus 400 according to the present embodiment includes a captured image input unit 402, a first image cutout unit 421, an overlapping area measurement unit 404, a second image cutout unit 423, a storage control unit 406, a storage unit 408, and a space arrangement unit. 410, a corresponding point detection unit 412, and a synthesis processing unit 414.

The moving image is input to the photographed image input unit 402. That is, a plurality of time-sequential captured images used for image synthesis are sequentially input, and the first image and the second image before and after the time-sequential order among the plurality of input captured images have overlapping areas. A moving image composed of a plurality of captured images is input.

The first image cutout unit 421 cuts out the first image from the plurality of shot images input to the shot image input unit 402. In other words, in the present embodiment, a moving image is input to the captured image input unit 402, and one of a plurality of frame images constituting the input moving image is cut out as a previous image.

The overlapping area measuring unit 404 measures an overlapping area between the cut out first image and the plurality of input captured images. That is, the overlapping area measuring unit 404 measures the overlapping area with the clipped previous image for the frame image of the input moving image. For example, the overlapping area measurement unit 404 may measure the overlapping area of the frame image that is in chronological order from the extracted previous image.

The second image cutout unit 423 cuts out the second image from the plurality of captured images when the measurement result of the overlapping area measurement unit 404 becomes a predetermined value. That is, the second image cutout unit 423 cuts out the frame image as the second image when the overlapped region is measured by the overlapped region measuring unit 404 and the result becomes a predetermined value. For example, the overlapping region measuring unit 404 measures the area of the overlapping region, and the second image cutout unit 423 cuts out the frame image in which the area to be measured is a predetermined value as the second image.

The storage control unit 406 stores the input moving image and the position information of the overlapping area measured by the overlapping area measuring unit 404 in the storage unit 408. Alternatively, the storage control unit 406 may cause the storage unit 408 to store the front image, the rear image, and the position information of the overlapping area.

FIG. 17 is a flowchart showing the operation and the image processing method of the image processing apparatus 400 of the present embodiment.

First, a plurality of photographed images in time series used for composition are input by the photographed image input unit 402 (step S20). For example, an image group (moving image) captured by the robot apparatus 100 is captured by the CPU 310 of the computer 300 via the mobile communication unit 318, the wireless LAN communication unit 320, or the short-range wireless communication unit 322 of the computer 300. Input to the input unit 402. In the case of this embodiment, a moving image is input to the captured image input unit 402.

Thereafter, the first image cutout unit 421 cuts out a previous image from a plurality of frame images constituting the moving image (step S21). For example, the first image cutout unit 421 is realized by the CPU 310 of the computer 300.

Next, the overlapping area measuring unit 404 measures an overlapping area between a plurality of frame images constituting the moving image and the previous image (step S22). For example, the overlapping area measurement unit 404 is realized by the CPU 310 of the computer 300. Then, the second image cutout unit 423 cuts out a frame image that is a subsequent image from the moving image when the measurement result of the overlapping area measurement unit 404 becomes a threshold value (step S23). For example, the second image cutout unit 423 is realized by the CPU 310 of the computer 300.

Thereafter, the moving image input by the photographed image input unit 402 and the position information measured by the overlapping area measuring unit 404 are realized in the main memory 314 of the computer 300 by the storage control unit 406 realized by the CPU 310 of the computer 300. It is stored in the storage unit 408 (step S24).

Next, the spatial arrangement unit 410 realized by the CPU 310 of the computer 300 arranges the captured image in a spatial arrangement based on the position information of the overlapping area that is the outline information (step S25). Thereafter, the corresponding point detection unit 412 realized by the CPU 310 of the computer 300 detects the corresponding point in the overlapping region (step S26). The overlapping area after the captured image is spatially arranged is not only the overlapping area of the relationship between the previous image and the subsequent image, but also the overlapping area with an image not in the context of the image (for example, the overlapping

areas

494 and 496 shown in FIG. 14). ) Can also detect corresponding points.

After that, the composite processing unit 414 realized by the CPU 310 of the computer 300 generates a composite image based on the detected corresponding points (step S27).

<Third Embodiment>
Next, a third embodiment will be described. In the present embodiment, a group of still images is input to the photographed image input unit 402, and a front image and a rear image are extracted from the image group and stored in the storage unit 408.

FIG. 18 is a diagram illustrating a functional configuration example of the image processing apparatus 400 according to the third embodiment. In addition, the part already demonstrated in FIG. 8 attaches | subjects the same code | symbol, and abbreviate | omits description.

The image processing apparatus 400 according to the present embodiment includes a captured image input unit 402, an overlapping area measurement unit 404, an image extraction unit 431, a storage control unit 406, a storage unit 408, a space arrangement unit 410, a corresponding point detection unit 412, and a synthesis process. Part 414.

The photographed image input unit 402 receives a group of still images. That is, a captured image input unit 402 that sequentially inputs a plurality of captured images in time series used for image synthesis, and includes a first image and a second image before and after the time sequence in the plurality of captured images. A plurality of captured images having overlapping areas are input.

The image extraction unit 431 extracts the previous image and the rear image when the measurement result of the overlapping area measurement unit 404 becomes a default value. That is, the image extraction unit 431 extracts the previous image and the subsequent image from the still image group input by the captured image input unit 402 when the measurement result of the overlapping area measurement unit 404 becomes a default value. For example, the overlapping area measurement unit 404 measures the area of the overlapping area, and extracts the front image and the rear image when the area to be measured becomes a predetermined value.

The storage control unit 406 stores the extracted previous image and subsequent image in the storage unit 408, and causes the storage unit 408 to store position information indicating the overlapping area measured in association with the subsequent image.

FIG. 19 is a flowchart showing an operation and an image processing method of the image processing apparatus 400 of the present embodiment.

First, the photographed image input unit 402 inputs a plurality of photographed images (groups of still images) in time series used for composition (step S30). For example, a captured image input unit 402 realized by the CPU 310 of the computer 300 via the mobile communication unit 318, the wireless LAN communication unit 320, or the short-range wireless communication unit 322 of the computer 300 for the image group captured by the robot apparatus 100. Is input.

Next, the overlapping area of each image is measured by the overlapping area measuring unit 404 (step S31). For example, the overlapping area measurement unit 404 is realized by the CPU 310 of the computer 300.

Then, when the measurement result of the overlapping area measurement unit 404 becomes a default value, the image extraction unit 431 extracts the previous image and the rear image (step S32). For example, the image extraction unit 431 is realized by the CPU 310 of the computer 300.

Thereafter, the captured image input by the captured image input unit 402 and the position information measured by the overlapping region measuring unit 404 are realized in the main memory 314 of the computer 300 by the storage control unit 406 realized by the CPU 310 of the computer 300. Is stored in the storage unit 408 (step S33).

Next, the spatial arrangement unit 410 realized by the CPU 310 of the computer 300 arranges the photographed images in a spatial arrangement based on the position information of the overlapping area that is the outline information (step S34). Thereafter, the corresponding point detection unit 412 realized by the CPU 310 of the computer 300 detects the corresponding point in the overlapping region (step S35). The overlapping area after the captured image is spatially arranged is not only the overlapping area of the relationship between the previous image and the subsequent image, but also the overlapping area with an image not in the context of the image (for example, the overlapping

areas

494 and 496 shown in FIG. 14). ) Can also detect corresponding points.

After that, the composite processing unit 414 realized by the CPU 310 of the computer 300 generates a composite image based on the detected corresponding points (step S36).

<Fourth Embodiment>
Next, a fourth embodiment will be described. In the present embodiment, the first image and the second image are automatically acquired by the captured image acquisition unit 401.

FIG. 20 is a diagram illustrating a functional configuration example of the image processing apparatus 400 according to the second embodiment. In addition, the part already demonstrated in FIG. 8 attaches | subjects the same code | symbol, and abbreviate | omits description.

The camera device 403 includes a captured image acquisition unit 401, an overlapping area measurement unit 404, an imaging control unit 441, a storage control unit 406, a storage unit 408, a space arrangement unit 410, a corresponding point detection unit 412, and a composition processing unit 414. For example, the camera device 403 is provided in the robot device 100. For example, the captured image acquisition unit 401 is realized by the imaging device 200 of the robot apparatus 100, and includes an overlapping area measurement unit 404, an imaging control unit 441, a storage control unit 406, a storage unit 408, a space arrangement unit 410, and a corresponding point detection unit 412. , And the synthesis processing unit 414 is realized by the imaging control unit 204 of the robot apparatus 100, for example.

The captured image acquisition unit 401 is a captured image acquisition unit 401 that sequentially acquires a plurality of time-sequential captured images used for image synthesis, and includes a previous image before and after a time-sequential order among the plurality of captured images. A plurality of captured images having overlapping areas with the subsequent image are acquired.

The overlapping area measurement unit 404 measures an overlapping area between the previous image and the moving image captured by the captured image acquisition unit 401. For example, the overlapping area measurement unit 404 measures an overlapping area between the acquired previous image and the live view image acquired by the camera.

The imaging control unit 441 causes the captured image acquisition unit 401 to acquire the second image when the measurement result of the overlapping area measurement unit 404 becomes a default value. That is, the imaging control unit 441 measures an overlapping area between the previous image and the live view image, and automatically acquires the subsequent image when the overlapping area becomes a predetermined value. For example, the overlapping area measuring unit 404 measures the area of the overlapping area, and when the area to be measured becomes a predetermined value, the imaging control unit 441 causes the captured image acquisition unit 401 to acquire the second image.

The examples of the present invention have been described above, but the present invention is not limited to the above-described embodiments, and it goes without saying that various modifications can be made without departing from the spirit of the present invention.

DESCRIPTION OF SYMBOLS 1 Bridge 2 Main girder 3 Horizontal girder 4 Counter-tilt 5 Horizontal structure 6 Floor slab 100 Robot apparatus 102 Main frame 104 Vertical extension arm 104A Camera installation part 106 Case 108 X direction drive part

108A Ball screw

108B Ball nut 108C Motor 110 Y direction drive Unit 110A tire 110B tire 112 Z-direction drive unit 120 pan-tilt mechanism 130 robot control unit 200 imaging device 202 camera 202A imaging unit 204 imaging control unit 206 pan-tilt drive unit 210 pan-tilt control unit 230 robot-side communication unit 300 computer 302 touch panel display 304 speaker 305 Built-in camera 306 External connection terminal 310 CPU
312 System bus 314 Main memory 316 Non-volatile memory 318 Mobile communication unit 318A Antenna 320 Wireless LAN communication unit 320A Antenna 322 Short-range wireless communication unit 322A Antenna 324 Wired communication unit 326 Display unit 328 Input unit 330 Key input unit 332 Audio processing unit 334 Image processing unit 400 Image processing device 401 Captured image acquisition unit 402 Captured image input unit 403 Camera device 404 Overlap area measurement unit 406 Storage control unit 408 Storage unit 410 Spatial arrangement unit 412 Corresponding point detection unit 414 Composite processing unit 421 First image cut Output unit 423 Second image extraction unit 431 Image extraction unit 441 Imaging control unit 500 Imaging system S10-S15 Steps S20-S27 of the image processing method of the first embodiment Step S30- of the image processing method of the second embodiment S3 Step of the image processing method of the third embodiment

Claims

A photographed image input unit that sequentially inputs a plurality of photographed images in time series used for image synthesis, wherein the first image and the second image before and after the time series of the plurality of photographed images are mutually A captured image input unit for inputting the plurality of captured images having an overlapping area;
An overlapping area measuring unit that measures the overlapping area between the first image and the second image to be input;
A storage control unit for storing the input plurality of captured images in a storage unit and storing in the storage unit position information indicating the measured overlap area in association with the second image;
A spatial arrangement unit that spatially arranges the plurality of captured images based on the plurality of captured images and the position information stored in the storage unit;
A corresponding point detection unit that detects the overlapping regions of the plurality of captured images arranged in space and detects corresponding points between the images of the detected overlapping regions;
A combining processing unit that combines the plurality of captured images based on the detected corresponding points to generate a combined image;
An image processing apparatus.
A photographed image input unit that sequentially inputs a plurality of photographed images in time series used for image synthesis, wherein the first image and the second image before and after the time series of the plurality of photographed images are mutually A captured image input unit for inputting the plurality of captured images having an overlapping area;
A first image cutout unit that cuts out the first image from the input plurality of captured images;
An overlapping area measuring unit that measures the overlapping area between the cut out first image and the input plurality of captured images;
A second image cutout unit that cuts out the second image from the plurality of captured images when the measurement result of the overlapping region measurement unit is a default value;
A storage control unit for storing the input plurality of captured images in a storage unit and storing in the storage unit position information indicating the measured overlap area in association with the second image;
A spatial arrangement unit that spatially arranges the plurality of captured images based on the plurality of captured images and the position information stored in the storage unit;
A corresponding point detection unit that detects the overlapping regions of the plurality of captured images arranged in space and detects corresponding points between the images of the detected overlapping regions;
A combining processing unit that combines the plurality of captured images based on the detected corresponding points to generate a combined image;
An image processing apparatus.
A photographed image input unit that sequentially inputs a plurality of photographed images in time series used for image synthesis, wherein the first image and the second image before and after the time series of the plurality of photographed images are mutually A captured image input unit for inputting the plurality of captured images having an overlapping area;
An overlapping area measuring unit that measures the overlapping area between the first image and the second image to be input;
An image extraction unit that extracts the first image and the second image when the measurement result of the overlapping region measurement unit is a default value;
A storage control unit for storing the extracted first image and the second image in a storage unit, and storing in the storage unit position information indicating the measured overlap area in association with the second image;
A spatial arrangement unit that spatially arranges the plurality of captured images based on the plurality of captured images and the position information stored in the storage unit;
A corresponding point detection unit that detects the overlapping regions of the plurality of captured images arranged in space and detects corresponding points between the images of the detected overlapping regions;
A combining processing unit that combines the plurality of captured images based on the detected corresponding points to generate a combined image;
An image processing apparatus.
The image processing apparatus according to any one of claims 1 to 3, wherein the composition processing unit generates an ortho image by compositing.
The image processing apparatus according to any one of claims 1 to 4, wherein the storage control unit records the position information in an EXIF tag attached to the second image.
The said corresponding point detection part also detects the said overlapping area | region with images other than the said 2nd image in the said 1st image, and detects the corresponding point between the images of the detected said overlapping area. The image processing apparatus according to claim 1.
The image according to any one of claims 1 to 6, wherein the position information indicating the overlapping region is a first coordinate of the first image and a second coordinate of the second image corresponding to the first image. Processing equipment.
The image processing apparatus according to claim 7, wherein the position information indicating the overlapping area is two or less sets of first coordinates and second coordinates.
A captured image acquisition unit for sequentially acquiring a plurality of captured images in time series used for image composition, wherein the first image and the second image before and after the time sequence in the plurality of captured images are mutually A captured image acquisition unit that acquires the plurality of captured images having overlapping areas;
An overlapping region measuring unit that measures an overlapping region between the first image and the moving image captured by the captured image acquisition unit;
When the measurement result of the overlapping area measurement unit becomes a default value, the imaging control unit that causes the captured image acquisition unit to acquire the second image;
A storage control unit that stores the acquired plurality of captured images in a storage unit, and stores, in the storage unit, position information indicating the measured overlap area in association with the second image;
A spatial arrangement unit that spatially arranges the plurality of captured images based on the plurality of captured images and the position information stored in the storage unit;
A corresponding point detection unit that detects the overlapping regions of the plurality of captured images arranged in space and detects corresponding points between the images of the detected overlapping regions;
A combining processing unit that combines the plurality of captured images based on the detected corresponding points to generate a combined image;
A camera device comprising:
A step of sequentially inputting a plurality of time-sequential captured images to be used for image synthesis, wherein the first image and the second image before and after the time-sequential order of the plurality of photographed images have overlapping areas; Inputting the plurality of photographed images,
Measuring the overlapping area between the first image and the second image to be input;
Storing the input plurality of captured images in a storage unit, and storing in the storage unit position information indicating the measured overlap area in association with the second image;
Spatially arranging the plurality of photographed images based on the plurality of photographed images stored in the storage unit and the position information;
Detecting the overlapping regions of the plurality of captured images arranged in space, and detecting corresponding points between the images of the detected overlapping regions;
Combining the plurality of captured images based on the detected corresponding points to generate a composite image;
An image processing method including:
A step of sequentially inputting a plurality of time-sequential captured images to be used for image synthesis, wherein the first image and the second image before and after the time-sequential order of the plurality of photographed images have overlapping areas; Inputting the plurality of photographed images,
Cutting out the first image from the input plurality of captured images;
Measuring the overlap area between the cut out first image and the input plurality of captured images;
Cutting out the second image from the plurality of captured images when the measurement result in the step of measuring the overlap region is a default value;
Storing the input plurality of captured images in a storage unit, and storing in the storage unit position information indicating the measured overlap area in association with the second image;
Spatially arranging the plurality of photographed images based on the plurality of photographed images stored in the storage unit and the position information;
Detecting the overlapping regions of the plurality of captured images arranged in space, and detecting corresponding points between the images of the detected overlapping regions;
Combining the plurality of captured images based on the detected corresponding points to generate a composite image;
An image processing method including:
A step of sequentially inputting a plurality of time-sequential captured images to be used for image synthesis, wherein the first image and the second image before and after the time-sequential order of the plurality of photographed images have overlapping areas; Inputting the plurality of photographed images,
Measuring the overlapping area between the first image and the second image to be input;
Extracting the first image and the second image when the measurement result of the step of measuring the overlap region is a predetermined value;
Storing the extracted first image and the second image in a storage unit, and storing, in the storage unit, positional information indicating the measured overlap area in association with the second image;
Spatially arranging the plurality of photographed images based on the plurality of photographed images stored in the storage unit and the position information;
Detecting the overlapping regions of the plurality of captured images arranged in space, and detecting corresponding points between the images of the detected overlapping regions;
Combining the plurality of captured images based on the detected corresponding points to generate a composite image;
An image processing method including: