JP6665917B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus, and more particularly, to an image processing apparatus for joining a plurality of input images.

  2. Description of the Related Art An omnidirectional imaging system that uses a plurality of wide-angle lenses such as a fish-eye lens and an ultra-wide-angle lens to capture images in all directions (hereinafter, omnidirectional) at once is known. In the omnidirectional imaging system, an omnidirectional image is generated by projecting an image from each lens onto a sensor surface and combining the obtained images by image processing. For example, a spherical image can be generated using two wide-angle lenses having an angle of view exceeding 180 degrees.

  In the above-described image processing, distortion correction and projection transformation are performed on a partial image captured by each lens optical system based on a predetermined projection model and in consideration of distortion from an ideal model. Then, the partial images are connected using the overlapping portion included in the partial images, and a process of forming one omnidirectional image is performed. In the process of joining images, in an overlapping portion between partial images, a joining position at which a subject overlaps is detected using pattern matching or the like.

  However, in the splicing position detection technology based on the conventional pattern matching, an appropriate splicing is performed when the target area to be matched is a flat image or an area where the same pattern is repeated and has few features. It was difficult to detect the position. For this reason, the partial images cannot be satisfactorily stitched, and the quality of the obtained omnidirectional image may be reduced.

  Various techniques for joining a plurality of partial images captured using a plurality of cameras are known. For example, Japanese Patent Application Laid-Open No. 2001-148779 (Patent Literature 1) discloses that when performing pattern matching, it is possible to avoid an error in pattern matching caused by the use of an inappropriate matching area, and to synthesize images with high accuracy. An image processing apparatus for performing the above is disclosed. In the related art of Patent Literature 1, it is configured to determine whether or not the matching region extracted by the matching region extracting unit is appropriate, and to re-extract the inappropriate matching region. . In addition, the matching result of the pattern matching unit is also determined to be appropriate, and if the matching result is not appropriate, the matching area is extracted again.

  The prior art disclosed in Patent Document 1 removes in advance areas that are not suitable as matching areas, such as areas of white pixels and black pixels on the entire surface, and areas in which straight lines extending in the vertical, horizontal, and oblique directions exist. By performing pattern matching, the accuracy of image synthesis is improved.

  However, the conventional technique of Patent Document 1 is a technique of re-extracting a matching area when the matching area is inappropriate. For this reason, it was still difficult to determine an appropriate connection position for a matching area determined to be inappropriate.

  The present invention has been made in view of the above-described inadequacy of the related art, and the present invention has been made to combine a plurality of input images at an appropriate joint position even in a partial region having a small feature of the input image. It is an object of the present invention to provide an image processing apparatus capable of performing the following.

  The present invention provides an image processing apparatus for synthesizing a plurality of input images, having the following features, in order to solve the above-mentioned problems. The image processing apparatus includes: a feature amount obtaining unit configured to obtain a feature amount of a plurality of target images in the first input image; The image processing apparatus further includes a combining unit that combines the first input image and the second input image based on the connection position detected using the connection position with respect to the second input image.

  According to the above configuration, it is possible to combine a plurality of input images at an appropriate connection position even in a partial region having few features of the input image.

FIG. 1 is a cross-sectional view showing a spherical imaging system according to the embodiment. FIG. 2 is a hardware configuration diagram of the celestial sphere imaging system according to the embodiment. FIG. 2 is a diagram showing a flow of entire image processing in the celestial sphere imaging system according to the embodiment. FIG. 3 is a main functional block diagram of a omnidirectional image combining process realized on the omnidirectional imaging system according to the embodiment. 6 is a flowchart illustrating an overall flow of a celestial sphere image synthesis process executed by the celestial sphere imaging system according to the embodiment. FIG. 3 is a diagram illustrating a projection relationship in a celestial sphere imaging system using a fisheye lens. FIG. 2 is a diagram illustrating a data structure of image data in a spherical image format used in the embodiment. FIG. 7 is a diagram illustrating conversion data referred to by a position detection distortion correction unit and an image synthesis distortion correction unit. FIG. 7 is a diagram for explaining mapping of two partial images captured by two fisheye lenses to a spherical coordinate system in a position detection process. FIG. 3 is a functional block diagram of a connection position detection unit according to the embodiment. FIG. 4 is a block diagram of a template feature amount calculation unit according to a specific embodiment. 5 is a flowchart illustrating a connection position detection process executed by the celestial sphere imaging system according to the embodiment. FIG. 4 is a diagram illustrating a method for generating a template image by a template generation unit according to the embodiment. 6A and 6B are diagrams illustrating a method of calculating a temporary position of a template image based on the ranking performed by the temporary position calculation unit according to the embodiment. FIG. 4 is a diagram illustrating a search range setting method performed by a search range setting unit according to the embodiment. FIG. 9A is a diagram illustrating a template image, and FIG. 9B is a diagram illustrating a process of searching for a template image in a search range by pattern matching. (A) A graph of the offset function, and (B, C) a graph in which the score before correction and the score after correction are plotted at the search position are shown together with the matching position based on the score before correction and the matching position based on the score after correction. FIG. FIG. 4 is a diagram illustrating a data structure of detection result data generated by a connection position detection unit in the embodiment. FIG. 4 is a diagram for describing processing for generating detection result data by a connection position detection unit according to the embodiment. 9 is a flowchart illustrating a process of generating an image synthesis conversion table, which is performed by the omnidirectional imaging system according to the embodiment. FIG. 8 is a diagram illustrating mapping of two partial images captured by two fisheye lenses to a spherical coordinate system during image synthesis processing. The schematic diagram of the celestial sphere imaging system in other embodiments. FIG. 11 is a flowchart showing an overall flow of a celestial sphere image synthesis process in a celestial sphere imaging system according to another embodiment.

  Hereinafter, embodiments of the present invention will be described, but embodiments of the present invention are not limited to the embodiments described below. In the following embodiments, as an example of an image processing apparatus and an imaging system, an image pickup body including two fish-eye lenses in an optical system is provided, and distortion correction and projection are performed on two partial images captured by the two fish-eye lenses. A description will be given using an omnidirectional imaging system 10 having an image processing function of performing conversion, connecting images, and generating an omnidirectional image.

[overall structure]
Hereinafter, the overall configuration of the omnidirectional imaging system according to the present embodiment will be described with reference to FIGS. FIG. 1 is a cross-sectional view showing a spherical imaging system (hereinafter, simply referred to as an imaging system) 10 according to the present embodiment. An imaging system 10 shown in FIG. 1 includes an imaging body 12, a housing 14 that holds the imaging body 12, components such as a controller and a battery, and a shutter button 18 provided on the housing 14. The imaging body 12 shown in FIG. 1 includes two imaging optical systems 20A and 20B, and two solid-state imaging devices 22A and 22B such as a charge coupled device (CCD) sensor and a complementary metal oxide semiconductor (CMOS) sensor. A combination of the imaging optical system 20 and the solid-state imaging device 22 one by one is referred to as an imaging optical system. Each of the imaging optical systems 20 can be configured as, for example, seven groups of seven fish-eye lenses. In the embodiment shown in FIG. 1, the fisheye lens has a total angle of view larger than 180 degrees (= 360 degrees / n; n = 2), preferably has an angle of view of 185 degrees or more, and is more preferably. Has an angle of view of 190 degrees or more.

  The optical elements (lens, prism, filter, and aperture stop) of the two imaging optical systems 20A and 20B have a positional relationship with respect to the solid-state imaging elements 22A and 22B. The positioning is performed so that the optical axes of the optical elements of the imaging optical systems 20A and 20B are orthogonal to the center of the light receiving area of the corresponding solid-state imaging device 22, and the light receiving area is formed by the corresponding fisheye lens. This is performed so as to be on the image plane. Each of the solid-state imaging devices 22 is a two-dimensional solid-state imaging device in which a light receiving region forms an area, and converts light collected by the imaging optical system 20 to be combined into an image signal.

  In the embodiment shown in FIG. 1, the imaging optical systems 20A and 20B are of the same specification, and are combined in opposite directions so that their optical axes match. The solid-state imaging devices 22A and 22B convert the received light distribution into an image signal and output the image signal to an image processing unit on a controller (not shown). Although the image processing means will be described in detail later, the partial images input from the solid-state imaging devices 22A and 22B are connected and combined to form an image having a solid angle of 4π radians (hereinafter, referred to as an “omnidirectional image”). Generate The celestial sphere image is obtained by shooting in all directions that can be seen from the shooting location. Here, in the embodiment shown in FIG. 1, a spherical image is generated, but a so-called panoramic image obtained by photographing only the horizontal plane at 360 degrees may be used.

  Here, by making the scanning directions of the solid-state imaging devices 22A and 22B coincide with each other, it is possible to easily join the captured images. In other words, by making the scanning directions and the order of the solid-state imaging devices 22 coincide with each other at the portions where they are connected to each other, an effect can be obtained for connecting objects at the boundary between the cameras, particularly moving objects. For example, when the upper left portion of the image captured by the solid-state imaging device 22A and the lower left portion of the image captured by the solid-state imaging device 22B match as a portion to be joined, the solid-state imaging device 22A The scanning is performed from right to left from top to bottom of the solid-state imaging device. On the other hand, the solid-state imaging device 22B scans from right to left from bottom to top of the solid-state imaging device. As described above, by controlling the scanning directions of the respective solid-state imaging devices to coincide based on the portion to be joined of the images, an effect that joining is easy can be obtained.

  As described above, since the fisheye lens has a full angle of view exceeding 180 degrees, when composing a celestial sphere image, in an image captured by each imaging optical system, an overlapping image portion is a reference data representing the same image. It is used as a reference for image stitching. The generated celestial sphere image is stored in an external storage such as a display device, a printing device, an SD (registered trademark) card, or a compact flash (registered trademark) provided in or connected to the imaging body 12. Output to a medium or the like.

  FIG. 2 shows a hardware configuration of the imaging system 10 according to the present embodiment. The imaging system 10 includes a digital still camera processor (hereinafter, simply referred to as a processor) 100, a lens barrel unit 102, and various components connected to the processor 100. The lens barrel unit 102 has the above-described two sets of lens optical systems 20A and 20B, and the solid-state imaging devices 22A and 22B. The solid-state imaging device 22 is controlled by a control command from a CPU 130 described later in the processor 100.

  The processor 100 includes an ISP (Image Signal Processor) 108, a DMAC (Direct Memory Access Controller) 110, an arbiter (ARBMEMC) 112 for arbitrating memory access, an MEMC (Memory Controller) 114 for controlling memory access, And a distortion correction / image synthesis block 118. The ISPs 108A and 108B perform white balance setting and gamma setting on image data input through signal processing of the solid-state imaging devices 22A and 22B, respectively. The SDRAM 116 is connected to the MEMC 114. The SDRAM 116 temporarily stores data when the ISPs 108A and 180B and the distortion correction / image synthesis block 118 perform processing. The distortion correction / image synthesis block 118 performs distortion correction and top / bottom correction on two partial images obtained from the two imaging optical systems using information from the three-axis acceleration sensor 120, and synthesizes images.

  The processor 100 further includes a DMAC 122, an image processing block 124, a CPU 130, an image data transfer unit 126, an SDRAM 128, a memory card control block 140, a USB block 146, a peripheral block 150, and an audio unit 152. , A serial block 158, an LCD (Liquid Crystal Display) driver 162, and a bridge 168.

  The CPU 130 controls the operation of each unit of the imaging system 10. The image processing block 124 includes a resize block 132, a JPEG block 134, Various image processing is performed on the image data using the H.264 block 136 and the like. The resize block 132 is a block for enlarging or reducing the size of image data by interpolation processing. The JPEG block 134 is a codec block that performs JPEG compression and decompression. H. The H.264 block 136 is a H.264 block. This is a codec block for compressing and expanding moving images such as H.264. The image data transfer unit 126 transfers the image processed by the image processing block 124. The SDRAM 128 controls the SDRAM 138 connected to the processor 100, and temporarily stores the image data when performing various processes on the image data in the processor 100.

  The memory card control block 140 controls reading from and writing to the memory card inserted into the memory card slot 142 and the flash ROM 144. The memory card slot 142 is a slot for detachably mounting a memory card on the imaging system 10. The USB block 146 controls USB communication with an external device such as a personal computer connected via the USB connector 148. A power switch 166 is connected to the peripheral block 150. The audio unit 152 is connected to a microphone 156 to which the user inputs an audio signal and a speaker 154 to output a recorded audio signal, and controls audio input / output. The serial block 158 controls serial communication with an external device such as a personal computer, and is connected to a wireless NIC (Network Interface Card) 160. The LCD driver 162 is a drive circuit for driving the LCD monitor 164, and converts the signal into a signal for displaying various states on the LCD monitor 164.

  The flash ROM 144 stores a control program and various parameters described in codes readable by the CPU 130. When the power is turned on by operating the power switch 166, the control program is loaded into the main memory. The CPU 130 controls the operation of each unit of the apparatus according to the program read into the main memory, and temporarily stores data necessary for the control in the SDRAM 138 and a local SRAM (not shown).

  FIG. 3 is a diagram illustrating a flow of the entire image processing in the imaging system 10 according to the present embodiment. First, in steps S101A and 101B, an image is captured by each of the solid-state imaging devices 22A and 22B. In steps S102A and 102B, the ISP 108 shown in FIG. 2 performs an optical black correction process, a defective pixel correction process, a linear correction process, a shading process, and the like on the Bayer RAW image output from each of the solid-state imaging devices 22A and 22B. An area division averaging process is performed. In steps S103A and 103B, the data is stored in the memory. In steps S104A and 104B, the ISP 108 shown in FIG. 2 further performs white balance processing, gamma correction processing, Bayer interpolation processing, YUV conversion processing, edge enhancement processing, and color correction processing. In steps S105A and 105B, Stored in memory.

  When the above-described processing is completed for each of the two solid-state imaging devices 22A and 22B, in step S106, distortion correction and synthesis processing are performed on each of the partial images subjected to the above processing. In step S107, the celestial sphere image is appropriately filed and saved as a file in the internal memory or the external storage. Further, in the process of the distortion correction and the synthesis processing, the tilt and the vertical correction may be performed by appropriately obtaining information from the three-axis acceleration sensor 120. Further, the image file to be stored may be subjected to an appropriate compression process.

[Spherical image composition function]
Hereinafter, the omnidirectional image combining function of the imaging system 10 according to the present embodiment will be described in detail with reference to FIGS. FIG. 4 shows main functional blocks 200 of the spherical image synthesis processing realized on the imaging system 10 according to the present embodiment. As shown in FIG. 4, the distortion correction / image synthesis block 118 includes a position detection distortion correction unit 202, a joint position detection unit 204, a table correction unit 206, a table generation unit 208, and an image synthesis distortion correction unit. 210 and an image synthesizing unit 212.

  In addition, two partial images are input to the distortion correction / image synthesis block 118 from the two solid-state imaging devices 22A and 22B through image signal processing by the ISPs 108A and 108B. Here, reference is made to the solid-state imaging devices 22A and 22B by attaching numbers “0” and “1”, and an image using the solid-state imaging device 22A as a source is referred to as “partial image 0”. An image using the element 22B as a source is referred to as “partial image 1”. Further, the distortion correction / image synthesis block 118 is provided with a position detection conversion table 220 created in advance by a manufacturer or the like according to a predetermined projection model based on design data of each lens optical system.

  The position detection distortion correction unit 202 performs distortion correction on the input partial image 0 and partial image 1 using the position detection conversion table 220 as a process preceding the connection position detection process. An image (hereinafter, simply referred to as a corrected image) 0 and a corrected image for position detection 1 are generated. The input partial images 0 and 1 are images captured by a two-dimensional solid-state image sensor whose light receiving region forms an area, and are image data expressed in a plane coordinate system (x, y). On the other hand, the corrected image subjected to distortion correction using the position detection conversion table 220 is image data in a coordinate system different from that of the input image. And a polar coordinate system having two declination angles θ and φ).

  FIG. 6 is a diagram illustrating a projection relationship in an imaging system using a fisheye lens. In the present embodiment, an image captured by one fish-eye lens is obtained by capturing a direction substantially equivalent to a hemisphere from a capturing point. As shown in FIG. 6, the fisheye lens generates an image at an image height h corresponding to the incident angle φ with respect to the optical axis. The relationship between the image height h and the incident angle φ is determined by a projection function according to a predetermined projection model. The projection function varies depending on the characteristics of the fisheye lens. However, in a fisheye lens of a projection model called an equidistant projection method, f is a focal length and is expressed by the following equation (1).

  Other examples of the projection model include a center projection method (h = f · tan φ), a stereoscopic projection method (h = 2f · tan (φ / 2)), and an equal solid angle projection method (h = 2f · sin (φ / 2)). )) And the orthogonal projection method (h = f · sin φ). In either method, the image height h of the image is determined according to the incident angle φ from the optical axis and the focal length f. In the present embodiment, a configuration of a so-called circular fisheye lens having a smaller image circle diameter than the image diagonal line is adopted, and the obtained partial image is substantially hemispherical in the shooting range as shown in FIG. The plane image includes the entire projected image circle.

  FIG. 7 is a diagram illustrating the data structure of image data in the spherical image format used in the present embodiment. As shown in FIG. 7, the image data in the spherical image format has pixel values with coordinates of a vertical angle φ corresponding to an angle with respect to a predetermined axis and a horizontal angle θ corresponding to a rotation angle around the axis. Is represented as an array of The horizontal angle θ ranges from 0 to 360 degrees (also expressed as −180 degrees to +180 degrees), and the vertical angle φ ranges from 0 to 180 degrees (also expressed as −90 degrees to +90 degrees). Becomes Each coordinate value (θ, φ) is associated with each point on the spherical surface that represents the omnidirectional centering on the imaging point, and the omnidirectional is mapped on the omnidirectional image. The relationship between the plane coordinates of the image captured by the fisheye lens and the coordinates on the spherical surface in the spherical image format can be associated by using the projection function described with reference to FIG.

  FIG. 8 is a diagram illustrating conversion data referred to by the position detection distortion correction unit 202 and the image synthesis distortion correction unit 210. The conversion tables 220 and 224 define the projection from a partial image expressed in a plane coordinate system to an image expressed in a spherical coordinate system. As shown in FIGS. 8A and 8B, the conversion tables 220 and 224 are mapped to the coordinate values (θ, φ) of the corrected image and the coordinate values (θ, φ) for each fisheye lens. The information for associating the coordinate values (x, y) of the partial image before correction with all coordinate values (θ, φ) (θ = 0,... 360 degrees, φ = 0,. Hold against. In the example of FIG. 8, the angle assigned to one pixel is 1/10 degrees in both the φ direction and the θ direction, and the conversion tables 220 and 224 have information indicating a 3600 × 1800 correspondence relationship for each fisheye lens. Will be.

  The conversion table 220 for position detection used in the splice position detection is based on the lens design data and the like, based on the projection relationship of the lens described in FIG. It is calculated after correcting distortion from a typical lens model, and is tabulated. In contrast, the image synthesis conversion table 224 is generated from the position detection conversion table 220 by a predetermined conversion process, as will be described in detail later. In the embodiment to be described, the conversion data is data in which the correspondence between coordinate values is tabulated. However, in other embodiments, the transformed data is one or more that defines the projection from a partial image (x, y) represented in a planar coordinate system to an image (θ, φ) represented in a spherical coordinate system. May be used as coefficient data of the function.

  Referring again to FIG. 4, the position detection distortion correction unit 202 converts the partial image 0 and the partial image 1 with reference to the position detection conversion table 220, and outputs the position detection corrected image 0 and the position detection corrected image. 1 is generated. More specifically, the position detection distortion correction unit 202 refers to the position detection conversion table 220 for all coordinate values (θ, φ) of the converted corrected image, and converts each coordinate value (θ, φ) into The coordinate value (x, y) of the partial image before conversion to be mapped is obtained, and the pixel value of the coordinate value (x, y) in the partial image is referred to. As a result, a corrected image is generated.

  FIG. 9 is a diagram illustrating mapping of two partial images captured by two fisheye lenses to a spherical coordinate system in the position detection process. As a result of the processing performed by the position detection distortion correction unit 202, the two partial images 0 and 1 captured by the fisheye lens are developed in a spherical image format as shown in FIG. The partial image 0 captured by the fisheye lens 0 is typically mapped to a substantially upper hemisphere of the whole celestial sphere, and the partial image 1 captured by the fisheye lens 1 is generally mapped to a lower hemisphere of the whole celestial sphere. Is mapped. Since the corrected image 0 and the corrected image 1 expressed in the omnidirectional format exceed the 180-degree angle of view of the fisheye lens, they respectively protrude from the hemisphere. As a result, when the corrected image 0 and the corrected image 1 are superimposed, the image becomes Overlapping areas where the photographing ranges overlap each other occur.

  As will be described in detail later, after the correction by the position detection distortion correction unit 202, the connection position between the images is detected by the connection position detection unit 204 in the overlapping area. In the conversion table 220 for position detection according to the present embodiment, as shown in FIG. 9, the optical axes of the two lens optical systems are projected onto the two poles (φ = 0 degrees, 180 degrees) of the spherical surface, and the image Are projected to the vicinity of the equator of the spherical surface (φ = 90 degrees ± ((full angle of view−180 degrees) / 2)). In the spherical coordinate system, the closer the vertical angle φ is to 0 ° or 180 ° to the pole, the greater the distortion and the lower the connection position detection accuracy. On the other hand, by performing the projection as described above, the overlapping region is positioned near the vertical angle of 90 degrees where the distortion amount when displaced in the θ direction is small, and the connection position detection is performed. Position detection accuracy can be improved. As a result, it is possible to detect the connection position with high accuracy even for an image captured by a lens optical system having large distortion.

  Here, referring to FIG. 4 again, the joint position detection unit 204 receives the input of the corrected images 0 and 1 converted by the position detection distortion correction unit 202, and performs the pattern matching process to perform the input of the corrected image 0 , 1 are detected, and detection result data 222 is generated.

  In pattern matching, typically, if the target region to be matched is a flat image or a region where the same pattern is repeated and has few features, the connection position can be detected with high accuracy. It becomes difficult. Therefore, the connection position detection unit 204 according to the present embodiment measures a feature amount that indicates the degree of the feature of the image of the target region to be matched, and uses the measured feature amount to determine the connection position of the target region. Adopt a configuration to determine. As a result, the stitching is performed preferentially in a region having a large feature, thereby improving the quality of the obtained omnidirectional image.

  FIG. 10 is a diagram showing functional blocks of the connection position detection unit according to the present embodiment shown in FIG. More specifically, the connection position detection unit 204 shown in FIG. 10 includes a template generation unit 232, a template feature amount calculation unit 234, a template ranking unit 236, a temporary position calculation unit 238, a search range setting unit 240, , A matching calculation unit 242, a score correction unit 244, a connection position determination unit 246, and a detection result generation unit 248. In the following, a description will be given of the connection position detection processing by template matching. For convenience of explanation, the template side is referred to as a position detection correction image 1 and the searched side is referred to as a position detection correction image 0.

  The template generation unit 232 uses the position detection correction image 1 as a template image in template matching, and from the position detection correction image 1, a plurality of images to be searched in the search image (hereinafter, template images). Is generated.) In the connection position detection processing by template matching, a connection position to the position detection correction image 0 is obtained for each template image that is a part of the position detection correction image 1. Here, since the purpose is to connect the overlapping areas between the corrected images 0 and 1, a plurality of template images are generated from the overlapping area in the entire position detection corrected image 1.

  The template feature amount calculation unit 234 calculates a feature amount for each of the plurality of template images generated by the template generation unit 232. Here, the feature amount is a value that is an index for quantifying the degree of the feature of the template image. In the present embodiment, a large feature amount means that a large feature is recognized in the image. Is small means that the image has few features and is individual. As the feature amount, at least one of an edge amount extracted from the template image, a variance calculated from the template image, and a standard deviation can be used, but is not particularly limited.

  FIG. 11 shows a block diagram of a template feature amount calculation unit according to a specific embodiment. FIGS. 11A to 11C show a configuration of the template feature amount calculation unit 234A according to the embodiment using the edge amount as an index. On the other hand, FIG. 11D illustrates a configuration of the template feature amount calculation unit 234B according to the embodiment using the standard deviation or the variance as an index.

  When the edge amount is used as the feature amount of the template image, an edge enhancement block for performing edge enhancement as shown in FIG. 11A can be used. The edge enhancement block shown in FIG. 11A includes an edge extraction filter unit 250, a gain multiplication unit 252, and an LPF (Low-Pass Filter) unit 254. In the edge emphasizing block, a signal in which the edge amount is adjusted by multiplying the edge amount extracted by the edge extraction filter unit 250 by the gain by the gain multiplication unit 252 is generated. Then, a signal obtained by adding the adjusted signal and a signal from which the input signal has been subjected to the LPF processing to the input signal in the LPF section 254 and removing the noise in the adding section 256 is output as an edge-emphasized signal.

  In this embodiment, only the edges need to be extracted. Therefore, for example, the LPF-processed signal is made zero by using the LPF coefficient as shown in FIG. 11C, and the edge extracted by the edge extraction filter unit 250 using the edge extraction filter coefficient shown in FIG. 11B. Only the quantity needs to be output. By inputting the template image to the edge enhancement block shown in FIG. 11A, the edge amount for each pixel is output. As the feature amount for the template image, the sum of the edge amounts of all the pixels of the template may be used, or the average of the edge amounts may be used. The larger the total edge amount of the template image, the more the image includes a portion where the brightness changes discontinuously. Therefore, a flat image typically has a small edge amount and a small feature amount.

When the standard deviation (or variance) of the image is used as the feature amount of the template image, a configuration as shown in FIG. 11D can be provided. The standard deviation (or variance) calculation unit 258 illustrated in FIG. 11D obtains the standard deviation σ (or variance σ ² ) of the template image using the following calculation formula (2). In the following equation (2), the total number of pixels of the template image is N (= W pixels × H pixels), and the luminance at the coordinates (i, j) of the template image is T (i, j). The larger the standard deviation (or variance) of the template image, the wider the luminance is distributed in the histogram. Therefore, a flat image typically has a narrow image luminance distribution and a small feature amount.

  Although the standard deviation and the variance are illustrated as the feature amounts of the template image, the kurtosis in the density histogram obtained from the template image (the degree to which the distribution of the histogram is concentrated around the average value or spreads toward the tail) ), And other statistical indices such as contrast based on difference statistics, such as an index based on a density histogram such as skewness (indicating the degree to which the shape of the histogram is distorted from a symmetric shape). The index may be used to characterize the template image. In the above description, the feature value is calculated using the luminance value. However, the feature value may be calculated for each of the RGB colors, and the feature value of the template image may be calculated as the sum of the feature values. The feature amounts such as the edge amount, the standard deviation, and the variance can be easily calculated from the target image, the calculation cost is low, and a good detection result is obtained. However, the present invention is not limited to this, and any index value that indicates the degree of the feature of the image can be adopted.

  Referring again to FIG. 10, the template ranking unit 236 ranks the order of processing of the plurality of template images based on the feature amounts calculated by the template feature amount calculation unit 234, and calculates the feature amount between the template images. Define relative relationships.

  The provisional position calculation unit 238 calculates a provisional position as a reference of a template matching process to be described later, with respect to the template image of interest out of the plurality of generated template images. Here, the provisional position is calculated based on the relative relationship between the feature amounts between the template image of interest and each of the peripheral template images around the position detection corrected image 1.

  The template matching is a process of searching for a template image from a search image to be searched. If a region corresponding to the template image is specified to some extent, the search range in the search image is determined. Can be limited. The search range setting unit 240 sets a search range for template matching based on the relative relationship between the feature amounts of the template image of interest and the peripheral template images. Typically, a predetermined search range in the position detection image 0 centering on the temporary position calculated by the temporary position calculation unit 238 is set. Further, a further narrowed search range may be set based on the determined connection position of the peripheral template image.

  The matching calculation unit 242 performs matching of the target template image with each part of the position detection corrected image based on the calculated temporary position, typically by a template matching method. calculate. In the matching calculation, the matching score, which is an evaluation value based on the similarity of the image at each position, is calculated while moving the template image within the search range centered on the temporary position set by the search range setting unit 240. Is calculated.

  The score correction unit 244 preferentially increases the score of the template image of interest with respect to the score based on the similarity of the image calculated by the matching calculation unit 242 described above, with the calculated temporary position as the center. Perform offset correction. As a result, a score is calculated in consideration of the provisional position calculated based on the relative relationship between the feature amounts between the template images.

  The connection position determination unit 246 determines a connection position of the target template image at a position where the corrected score is maximized on the position detection correction image 0. The connection position determined at this time is a position in which the connection position of the peripheral template image having a larger feature amount than the template image of interest is added.

  When the above-described processing by the provisional position calculation unit 238, the search range setting unit 240, the matching calculation unit 242, the score correction unit 244, and the connection position determination unit 246 is performed for each of the plurality of template images, the position corresponding to each of the template images is determined. The connection position on the correction image for detection 0 is obtained. The detection result generation unit 248 calculates the connection position for each pixel (θ, φ) in the spherical format based on the data set of the connection position corresponding to each template image obtained here, and Generate data 222.

  Referring again to FIG. 4, the table correction unit 206 corrects the previously prepared position detection conversion table 220 based on the detection result data 222 and passes the correction to the table generation unit 208. The table generation unit 208 generates an image synthesis conversion table 224 from the conversion data corrected by the table correction unit 206 based on rotational coordinate conversion.

  The image synthesis distortion correction unit 210 performs distortion correction on the original partial image 0 and the partial image 1 using the image synthesis conversion table 224 as a process prior to the image synthesis process. And a corrected image 1 for image synthesis. The generated corrected image for image synthesis is expressed in a spherical coordinate system similarly to the corrected image for position detection, but the definition of the coordinate axis is different from that of the corrected image for position detection due to the rotation coordinate conversion. Becomes The image synthesizing unit 212 synthesizes the obtained corrected image for synthesizing image 0 and the obtained corrected image for synthesizing image 1 to generate a synthesized image in the spherical image format. Note that the processing executed by the joint position detection unit 204, the table correction unit 206, the table generation unit 208, the image synthesis distortion correction unit 210, and the image synthesis unit 212 will be described later in detail together with the description of the processing flow.

  The functional block 200 illustrated in FIG. 4 can further include a display image generation unit 214. Since the generated composite image is expressed in a spherical image format, if it is displayed as it is on a flat display device such as a display, the image is distorted as the vertical angles approach 0 and 180 degrees. become. The display image generation unit 214 is a unit that executes image processing for projecting a spherical image on a flat display device. The display image generation unit 214 converts, for example, a composite image in the spherical image format from a spherical coordinate system to a planar coordinate system with a specific direction and a specific angle of view, and converts the image into a specific angle of view in a specific viewing direction specified by the user. A process of projecting an image can be performed.

[Flow of spherical image synthesis processing]
Hereinafter, the flow of the omnidirectional image synthesis processing according to the present embodiment will be described with reference to FIGS. FIG. 5 is a flowchart illustrating an overall flow of the omnidirectional image combining process performed by the imaging system 10 according to the present embodiment. The process illustrated in FIG. 5 is started from step S200 in response to, for example, pressing of the shutter button 18 instructing shooting with the two imaging optical systems and issuing a command from the CPU 130.

  In step S201, the imaging system 10 uses the position detection conversion table 220 to correct the partial image 0 and the partial image 1 acquired by the two solid-state imaging devices 22A and 22B by the position detection distortion correction unit 202. Make corrections. Thus, a corrected image for position detection 0 and a corrected image for position detection 1 are obtained. Thereby, a corrected image in the spherical image format as shown in FIG. 9 is obtained. In step S <b> 202, the imaging system 10 causes the connection position detection unit 204 to detect a connection position between images in an overlapping area of the position detection correction image 0 and the position detection correction image 1.

  FIG. 12 is a flowchart illustrating the connection position detection processing executed by the imaging system 10 according to the present embodiment. The process illustrated in FIG. 12 is called in step S202 illustrated in FIG. 5 and is started from step S300. In step S301, the imaging system 10 initializes a template image, a search image to be searched, a template block size, a template generation start coordinate, a template generation interval, and the total number of blocks.

  FIG. 13 is a diagram illustrating a method for generating a template image by the template generating unit according to the present embodiment. In the embodiment to be described, the template image 300 is an image of the overlapping area of the position detection corrected image 1, and the search image 310 is an image of the overlapping area of the position detection corrected image 0. The block size is a pixel size of a template image, and the generation interval is an interval at which adjacent template images are generated. The generation start coordinates are coordinates at which the first template image is cut out. Note that the block size and the generation interval may be determined in consideration of the desired accuracy and the processing amount of the connection.

  Assuming that the block size is W pixels × H pixels, the generation start coordinates are (sx, sy), and the generation interval is step pixels, a plurality of template images 302-1 to 302- # are formed as shown in FIG. Generated. The number of blocks # of the template to be generated is an integer value of a value obtained by dividing a horizontal size of the template generation image 300 (width size of the spherical format in the embodiment to be described = 3600 pixels) by a generation interval (step). Become.

  For the plurality of template images 302-1 to 302- # generated here, a corresponding portion 314 on the search image 310 is searched within a predetermined search range 312. Since both ends (0 degrees and 360 degrees) of the θ coordinate in the spherical image format are connected, when generating a template image or template matching, the right end is set as the left end, and the left end is set as the right end. Can handle.

  Referring to FIG. 12, in step S302, a template image is generated based on the current coordinates (sx, sy). In the first loop, a block-size area (W pixels × H pixels) is cut out from the generation start coordinates (sx, sy) initialized in step S301 to generate a template image. After the template image is generated, the coordinates are updated to (sx + step, sy), and the second and subsequent template generations are generated by specifying a block size area from the updated coordinates in the same manner. When the template image is generated, a template number (hereinafter, may be simply referred to as a number) as schematically shown in FIG. 13 is assigned to the generated template image. Although not particularly limited, in the embodiment to be described, it is assumed that the template images are generated in the overlapping region for one line and one turn in the θ direction.

  In step S303, the imaging system 10 calculates a feature amount from the template image generated in step S302. In step S304, the imaging system 10 determines whether or not processing of all blocks to be generated has been completed. If it is determined in step S304 that the process has not been completed (NO), the process loops to step S302, and proceeds to the generation of the next template image and the calculation of the feature amount. On the other hand, if it is determined in step S304 that the processing of all blocks has been completed (YES), the processing branches to step S305.

  In step S305, the imaging system 10 ranks the template images of all the blocks from the largest feature amount to the smallest feature amount. Thereby, the processing order after step S306 is defined. FIG. 14A is a diagram illustrating the ranking of template images according to the present embodiment. As shown in FIG. 14A, the template order of each number is assigned to the template image in the descending order of the feature amount. The processes of steps S306 to S311 shown in FIG. 12 are selected and processed in the order determined in step S305, starting from the template image having the highest rank.

  Referring to FIG. 12, in step S306, the imaging system 10 calculates a tentative position for a template image of interest to be processed. FIG. 14B is a diagram illustrating a first calculation method of the temporary position of the template image based on the ranking performed by the temporary position calculation unit according to the present embodiment. In the first calculation method, first, a connection position set for a peripheral template image located around the template image of interest is acquired.

  In the embodiment to be described, the template images are generated in a line in the θ direction in the overlapping area, and the template numbers are assigned in order from the left side, so that the adjacent relation is identified by the number. In the example shown in FIG. 14B, the connection positions of the template images (numbered 2 and 4 in the figure) before and after the target template image (numbered 3 in the figure) are acquired. The template order is assigned to each template image in step S305 in FIG. 12, and the process of detecting the connection position from step S306 to step S311 in FIG. 12 is performed in descending order. For this reason, for a template image having a feature amount larger than that of the template image of interest and having a higher rank, the connection position has already been determined.

  Therefore, in the example of FIG. 14B, since the order of the template image of interest (number 3) is third, the template images (number 4 and number 1) whose template order is first and second are connected. The position has already been determined. On the other hand, for the fourth to sixth template images (numbered 2, 5, and 6) highlighted in FIG. 14B, the connection position is not determined, and the initial connection position is determined. (0, 0) is set. The connection position of the template image of interest (No. 3) is also the initial connection position (0, 0) at the present time. The joint position indicates the amount of deviation between the coordinates of the template image in the overlap area of the corrected image 1 and the coordinates of the corresponding area in the overlap area of the corrected image 0. The joint position (0, 0) indicates that the template image is This means that the images are connected as they are at the coordinate position on the image 1.

  Therefore, in the example of FIG. 14B, as the connection position of the peripheral template images, the determined connection of the template image (the one with the number 4 in the figure) on the right side of the template image of interest (the one with the number 3 in the figure) is determined. The position and the initial connection position of the template image on the left (the number 2 in the figure) are acquired.

  Then, the averaging process is performed on the connection positions of the acquired peripheral template images, and the average value can be used as the provisional position of the template of interest. At this time, a simple arithmetic average may be calculated, but it is preferable to perform weighted averaging by changing the weight between the determined connection position and the undetermined connection position (initial connection position). Can be. Specifically, the weight can be increased so that the influence of the determined connection position becomes stronger.

Temporary position of the target template image (No. i) _(tx i, ty _i) can be calculated by the following equation (3). In the following formula (3), the splice point of the left side of the template image (No. _{i-1) (x i-} 1, y i-1), the weight w _- a, joint of the right template image (i + 1) The position is (x _{i + 1} , y _{i + 1} ), and the weight is w ₊ . w ₋ and w ₊ are weight values according to whether or not the connection position of the corresponding adjacent template image has been determined.

In the exemplary FIG. 14 (B), the example, the already determined weighting factor w _H "0.7", the weighting factor w _L undecided as "0.3", the joint position of the determined template number 4 (2, -2), assuming that the connection position of the undecided template number 2 is an initial value (0, 0), the provisional position of the template image of interest (number 3) is (1.4) by the above formula (3). , -1.4).

  FIG. 14C is a diagram illustrating a second method of calculating the temporary position of the template image based on the ranking performed by the temporary position calculation unit according to the present embodiment. In the second calculation method shown in FIG. 14C, first, as a peripheral template image of the template image of interest, the distance from the template image of interest and the set connection position are obtained for all template images. You. If the connection position has not been determined, the initial connection position (0, 0) is obtained in the same manner as described above. The distance between template images may simply be based on the number of blocks. In FIG. 14C, for example, the template of interest is set as template number 3, and distance 1 is set for template images of numbers 2 and 4, distance 2 is set for those of numbers 1 and 5, and A distance of 3 is given to the six.

  Then, the obtained connection positions of the peripheral template images are weighted by distance and averaged, and the average value can be used as the temporary position of the template image of interest. For the weighting based on the distance, a function that increases as the distance decreases can be used. For example, the reciprocal of the distance (1 / distance) can be used as the weight value.

Target template image (No. i) the temporary position _(tx i, ty _i) As a connecting position of each template image (ID _{j = 1~N) (x j,} y j), when the distance is _{D ij} , The temporary position (tx _i , ty _i ) can be calculated by the following formula (4). According to the second calculation method shown in FIG. 14 (C), the continuity with the surroundings can be more favorably maintained because the connection position of a distant template image other than the adjacent one is also considered. Note that, as the peripheral template images of the template image of interest, the distances and connection positions of all the template images are acquired, but a predetermined cutoff may be performed at the distances.

  In the above description, the weight value according to whether the connection position of the adjacent template images has been determined or the weight value according to the distance between the template images is used. However, the above calculation method is an example, and is not particularly limited. In another embodiment, weighting may be performed according to whether or not the connection position between adjacent template images has been determined and according to the distance between the template images.

  Referring to FIG. 12 again, in step S307, the imaging system 10 sets a search range of a search area for performing template matching. FIG. 15 is a diagram illustrating a search range setting method performed by the search range setting unit according to the present embodiment. In the template matching, a search area of a predetermined size is cut out from the search image for the template image of interest. The position detection image 0 that is the search image and the position detection image 1 that is the template image are generated by a predefined position detection conversion table, and are thus superimposed with a certain degree of accuracy. For this reason, since it is predicted that a portion corresponding to the template image will be found near the coordinates on the corresponding corrected image 0, the search area is cut out based on the coordinates. The search range determines the range in which the template matching is performed in the search area. The template image is matched with the image in the search area while shifting the position of the template image vertically and horizontally within the search range. Therefore, the processing time becomes longer as the search range becomes wider.

  FIG. 15A illustrates a method for setting the normal search range 360. In the normal setting method, the search area 330 centered on the initial position (○: coincident with the coordinates of the template image) is directly searched. The range is 360A. In this case, the search area 330 and the search range 360A are the same, and it is necessary to perform matching calculation in all the search areas.

  On the other hand, FIG. 15B illustrates a method for setting a search range based on the calculated temporary position. Normally, the determined tentative position (indicated by ● in the figure) 334 is moved from the initial position (indicated by ○ in the figure) 332 under the influence of the characteristic template. Is moved around the search area (350), and the area overlapping with the original search area 330 is set as the search range 360B, thereby narrowing the search range and shortening the processing time.

  FIG. 15C further describes a method of setting a search range based on a determined connection position for adjacent template images. As in the method shown in FIG. 15B, the search area is moved around the determined temporary position (●) (350), and an area overlapping the original search area 330 is obtained. Subsequently, referring to the determined connection positions of the adjacent template images, the search range 360C further narrowed down is determined. For example, if the joining positions of the adjacent template images on both sides have already been determined, the joining positions of the template of interest are considered to be limited substantially between them. Therefore, as shown in FIG. 15C, the processing time is further shortened by using the vertical connection positions 336U and 336L of the template images on both sides to restrict in the vertical direction and narrow the search range 360B. Can be.

  Referring again to FIG. 12, in step S308, the imaging system 10 calculates matching while shifting the template image vertically and horizontally in the search area within the search range set in step S307, and calculates the similarity. Find the score of

  Hereinafter, a zero-mean normalized cross-correlation method will be described as an example of template matching with reference to FIG. FIG. 16 is a diagram illustrating a process of searching for a template image in a search range by (A) a template image and (B) pattern matching. The total number of pixels of the template is N (= W pixels × H pixels), and the search position on the search range is represented by (kx, ky). When the search position (kx, ky) is the reference coordinate (0, 0), the luminance value of the template image at the coordinates (i, j) is T (i, j), and the luminance value on the search image is S ( kx + i, ky + j), the matching score M (kx, ky) by the ZNCC method can be obtained by the following equation (5). In FIG. 16, the search position (kx, ky) is set at the upper left of the template image. However, the way of obtaining the coordinates is not particularly limited, and may be set at the center of the template image.

  When the score M (kx, ky) is 1, perfect matching is obtained, and when the score is -1, negative-positive inversion is performed. A higher score M (kx, ky) indicates a higher similarity to the template image. As shown in FIG. 16B, template matching is performed while shifting the position of the template image vertically and horizontally within the search range, and a matching score M (kx, ky) at each search position is calculated.

  The above-mentioned ZNCC method is suitable in that it can absorb fluctuations in the gain of an image and can also absorb fluctuations in the average brightness of an image, but the score M (kx, ky) ) Is not limited to the ZNCC method. In other embodiments, SSD (Sum of Square Difference) method, SAD (Sum of Absolute Difference) method, ZSSD (Zero-mean Sum of Square Difference) method, ZSAD (Zero mean Sum of Absolute Difference) method, NCC (Normalized) Cross-Correlation) may be employed.

  Referring again to FIG. 12, in step S309, the imaging system 10 performs offset correction on the matching score M (kx, ky) based on the similarity at each position calculated in step S308, and the correction is performed. Calculate the final score. In step S310, the imaging system 10 determines a connection position for the template image of interest based on the corrected score at each position calculated in step S309.

  FIG. 17A shows a graph in which the offset function for the matching score is plotted against the search position (kx / ky). In the process of step S309, an offset value corresponding to the distance from the temporary position is added to the matching score M (kx, ky) by an offset function (kx, ky) as shown in FIG. The offset function shown in FIG. 17A is a function in which the offset value becomes maximum at the provisional position and monotonically decreases as the distance from the provisional position increases. The offset function may be determined based on the range of possible values of the matching score M. FIGS. 17B and 17C show matching positions based on scores before correction and matching positions based on scores after correction, in which a graph in which scores before correction and scores after correction are plotted at search positions (kx / ky) is shown. FIG. FIG. 17B illustrates a case of a template image having a large feature amount, and FIG. 17C illustrates a case of a template image having a small feature amount.

  As shown in FIG. 17B, the final matching score is raised around the temporary position by the offset correction. However, when the feature amount of the template image is large, the degree of similarity greatly changes due to a change in the position, so that the effect of the matching score based on the degree of similarity remains strong. For this reason, the position where the matching score based on the similarity is maximized is determined as the matching position regardless of whether or not the offset correction is performed. In other words, if the feature amount is large enough to clearly leave the peak of the matching score of the similarity, the similarity will predominantly determine the connection position.

  On the other hand, as shown in FIG. 17 (C), when the feature amount of the template image is small, a clear peak is not recognized in the normal matching score, and the matching score based on the similarity is higher than the matching score based on the similarity. The effect of the offset according to the distance is reflected in a deep color. For this reason, the vicinity of the tentative position where the offset function is substantially maximum is determined as the matching position. That is, when the feature amount is small, the temporary position predominantly determines the connection position.

When the matching position is determined, the connection position is determined as a shift amount (Δθ _i , Δφ _i ) from the position of the target template image (number i) superimposed on the coordinate value of the position detection correction image 0 as it is. Desired.

  By performing the above-described offset correction, there is a clear peak in the score based on the similarity, and when the similarity is large, the position becomes a connecting position. If a clear peak does not appear, such as when there is no feature, the temporary position is the connection position, maintaining continuity with the adjacent block, and reducing the rate at which the connection position is determined at a position far from the temporary position can do.

  Referring to FIG. 12 again, in step S311, the imaging system 10 determines whether the processing for all blocks of the template image has been completed. If it is determined in step S311 that the processing has not been completed (NO), the processing returns to step S306, and the processing is repeated until processing for all blocks is completed. On the other hand, if it is determined in step S311 that all the processes have been completed (YES), the process proceeds to step S312.

Through the processing from step S306 to step S311 described above, the connection positions (Δθ _i , Δφ _i ) corresponding to all of the plurality of generated template images (number i: i = 1 to #) are obtained. In step S312, the imaging system 10 determines the connection position (Δθ, Δφ) for each pixel (θ, φ) in the spherical format based on the obtained data set of the connection position corresponding to each template image. Calculate and generate detection result data 222. In step S313, the imaging system 10 terminates the main connection position detection processing, and returns to the processing illustrated in FIG.

FIG. 18 is a diagram illustrating a data structure of detection result data generated by the connection position detection unit 204 in the present embodiment. FIG. 19 is a diagram illustrating a process of generating detection result data by the position detection unit 204 according to the present embodiment. As a result of the processing in step S312, information in which a shift amount (Δθ, Δφ) is associated with each coordinate value (θ, φ) after conversion as shown in FIG. 18 is stored for all coordinate values. Detection result data 222 is obtained. At this time, the shift amount (Δθ, Δφ) corresponding to each coordinate value (θ, φ) is the shift amount (Δθ _i , Δφ _i ) for each template block (i) obtained by the connection position detection processing. , Can be set as the value of the center coordinate of each template block, and can be calculated by interpolation.

Specifically, first, as shown in FIG. 19A, the horizontal coordinate value θ is equal to the value of the center coordinate of the template block, and the upper end (φ = 0) and the lower end (having a height of 1800 In the case of a pixel, φ = 1799.), The shift amount (Δθ, Δφ) at each coordinate located is set as (0, 0). As for the other coordinates for which the shift amount is not set, as shown in FIG. 19B, a grid consisting of four nearby set points (indicated by A to D in the drawing) is considered, and two-dimensional linear interpolation is performed therein. The shift amount is calculated by calculation. Assuming that the point Q is a point that is internally divided into dθ: 1-dθ in the θ-axis direction and dφ: 1-dφ in the φ-axis direction in a four-point grid, the shift amount (Δθ _Q , Δφ _Q ) at the point Q Using the shift amounts ((Δθ _A, Δφ _A ),..., (Δθ _D, Δφ _D )) at the four points, it can be calculated by the following equation (6).

  In the embodiment to be described, the shift amount (0, 0) is set for the upper end (φ = 0) and the lower end (φ = 1799 in the above example) in the spherical format. However, the method of calculating the shift amount (Δθ, Δφ) corresponding to each coordinate value (θ, φ) is not particularly limited. Here, it is sufficient that the partial images 0 and 1 are connected without contradiction. In other embodiments, the two-dimensional linear interpolation operation is performed by setting coordinates having a shift amount (0, 0) inward. Can be. In this case, the shift amount (0, 0) may be set for all coordinates whose φ direction is outside the coordinates.

  Referring again to FIG. 5, in step S <b> 203, the imaging system 10 causes the table correction unit 206 to position the conversion table 220 for position detection on the spherical coordinates using the detection result data 222. Fix it. As shown in FIG. 18, the shift amount is obtained for each coordinate value in the omnidirectional image format by the connection position detection processing in step S202. Therefore, in step S203, the shift amount is used specifically for distortion correction of the partial image 0. In the detection distortion correction table 0, the input coordinate value (θ, φ) is corrected so as to correspond to (x, y) that was previously associated with (θ + Δθ, φ + Δφ). Note that it is not necessary to change the correspondence of the detection distortion correction table 1 used for distortion correction of the partial image 1.

  In step S204, the imaging system 10 generates the image synthesis conversion table 224 by performing the rotation coordinate conversion from the corrected position detection conversion table 220 by the table generation unit 208.

FIG. 20 is a flowchart illustrating a process of generating an image synthesis conversion table, which is executed by the imaging system 10 according to the present embodiment. FIG. 21 is a diagram illustrating mapping of two partial images captured by two fisheye lenses to a spherical coordinate system in the image synthesis processing. The process illustrated in FIG. 20 is called in step S204 illustrated in FIG. 5 and starts from step S400. In the loop of steps S401 to S406, the table generation unit 208 performs steps S402 to S405 for each coordinate value (θ _g , φ _g ) of the spherical coordinate system for image synthesis that is an input value of the distortion correction table for image synthesis. Execute the processing of The range of the set coordinate values is a range defined by the entire range of the horizontal angle (0 to 360 degrees) and the entire range of the vertical angle (0 to 180 degrees). Since the conversion process is performed for all coordinate values that are input values, each coordinate value is set here in order.

In step S402, the table generation unit 208 obtains the coordinate values (θ _d , φ _d ) of the spherical coordinate system for detecting the connection position corresponding to the coordinate values (θ _g, φ _g ) by the rotation coordinate conversion. By the rotation coordinate conversion, the definition of the coordinate axes of the horizontal angle θ _d and the vertical angle φ _{d around} the optical axis of one lens optical system as shown in FIG. Is converted into the definition of the horizontal angle θ _g and the vertical angle φ _g with respect to the basic axis. Coordinates (θ _d , φ _d ) corresponding to the above coordinates (θ _g, φ _g ) are set to a radius of 1 based on rotational coordinate conversion, and coordinates (θ _g, φ _g ) of a spherical coordinate system for image synthesis. three-dimensional orthogonal coordinate corresponding to _{(x g, y g, z} g) and the spherical coordinate system coordinates of the position detection (θ _d, φ _d) 3-dimensional orthogonal coordinate corresponding to _{(x d, y d, z} d ) Can be calculated by the following equation (7). In the following equation (7), the coefficient β is a rotation angle that defines a rotation coordinate transformation around the x-axis in the three-dimensional orthogonal coordinates, and is 90 degrees in the embodiment to be described.

  In the position detection conversion table 220, the optical axis is projected to the pole of the spherical surface, and the overlapping portion between the images is projected near the equator of the spherical surface, so that the vertical direction of the spherical image format and the zenith direction of the shot scene match. Not. On the other hand, in the image synthesizing conversion table 224, the optical axis is projected onto the spherical equator by the above-described rotational coordinate conversion, and the vertical direction of the omnidirectional image format matches the zenith direction of the captured scene. I will be.

In the loop from step S403 to step S405, the table generation unit 208 executes the processing of step S404 for each of the image 0 and the image 1. In step S404, the table generating unit 208 corrects the coordinate value (x, y) of the partial image (0, 1) associated with (θ _d , φ _d ) for each image (0, 1) after the correction. It is determined by referring to the conversion table for connection position detection. Although the conversion tables 220 and 224 hold the coordinate values (x, y) corresponding to one pixel at a time for both θ _d and φ _d , the coordinate values (θ _d , φ _d ) calculated by the conversion are stored. Is typically obtained as a value after the decimal point. For simplicity, the coordinate value (x, y) of the associated partial image (0, 1) is associated with the coordinate value existing in the table closest to the calculated coordinate value (θ _d , φ _d ). (X, y) can be adopted. In a preferred embodiment, the coordinates (θ _d ) calculated by referring to a plurality of coordinate values (x, y) associated with the nearest coordinate value and surrounding coordinate values among the table values are stored in the table. , Φ _d ), the coordinate value (x, y) of the associated partial image (0, 1) can be calculated by performing weighted interpolation according to the distance from the partial image (0, 1).

  In the loop of steps S403 to S405, the calculation is performed for all the images (0, 1). In the loop of steps S402 to S406, when the calculation for all the coordinate values to be input values is completed, in step S407, This processing ends. Thereby, all the data of the conversion table for image synthesis 224 is generated.

  Referring to FIG. 5 again, in step S205, the imaging system 10 performs distortion correction on the original partial image 0 and the partial image 1 by the image combining distortion correction unit 210 using the image combining conversion table 224. Then, the corrected image for image synthesis 0 and the corrected image for image synthesis 1 are obtained. As a result, as a result of the processing performed by the image-combining distortion correction unit 210, the two partial images 0 and 1 captured by the fisheye lens are developed in a spherical image format as shown in FIG. The partial image 0 captured by the fisheye lens 0 is typically mapped to the approximately left hemisphere of the whole celestial sphere, and the partial image 1 captured by the fisheye lens 1 is typically mapped to the right hemisphere of the celestial sphere. Is mapped.

  9 and FIG. 21, it is clear that the partial image 0 and the partial image 1 are mapped to different positions in the spherical format, and the zenith direction of the scene is the vertical direction of the image. Matches. The center of the partial images 0 and 1 is mapped on the equator with less distortion, and the overlapping area between the corrected image 0 and the corrected image 1 differs from that shown in FIG. , And in the vicinity of horizontal angles of 0 and 180 degrees.

  In step S206, the image combining unit 212 combines the image combining corrected image 0 and the image combining corrected image 1. In the synthesizing process, a blending process or the like is performed on an overlapping region that overlaps between images, and an existing pixel value is used for a region where only one of the pixel values exists. Through the synthesis processing, one celestial sphere image is generated from the two partial images captured by the fisheye lens.

  As described above, in the present embodiment, the feature amount indicating the degree of the feature of the template image to be matched is measured, and the joint position of the template image is determined using the measured feature amount. are doing. Then, the connection is preferentially performed in the region with the large feature, and the connection with the smaller feature is performed with reference to the connection position. This makes it possible to detect the connection position with high accuracy even in an area where the image is flat or an area in which the same pattern is repeated and has few features such as an area where the same pattern is repeated. The quality is improved. Even when the same pattern is repeated, such as when there are a plurality of connection position candidates only based on similarity, a unique connection position is determined by referring to a connection position of an area having adjacent features. can do.

  Further, since a process of calculating by excluding an inappropriate matching region having a small number of features is not employed, the algorithm can be simplified. Further, in the omnidirectional imaging system 10, since the omnidirectional image is the shooting range, many scenes are generated in which the overlapping region widely includes a flat region such as the sky, which has few features and is not suitable for joining. The joining processing of a plurality of images according to the present embodiment can be suitably applied to the joining of omnidirectional images having such properties.

[Other embodiments]
In the above-described embodiment, as an example of the image processing apparatus and the imaging system, the imaging system 10 that captures a still image of the celestial sphere by an imaging optical system provided therein and combines the captured images with an internal distortion correction / image synthesis block is used. explained. However, the configurations of the image processing device and the imaging system are not particularly limited. In another embodiment, the present invention may be configured as a celestial sphere moving image capturing system that captures moving images of the celestial sphere, and receives a plurality of partial images (still images or moving images) captured by a plurality of imaging optical systems. A camera processor that generates a spherical image (still image or moving image), and receives a plurality of partial images (still image or moving image) captured by a spherical imaging device that is in charge of shooting, and a spherical image An information processing device such as a personal computer, a workstation, or a virtual machine on a physical computer system that synthesizes (still image or moving image), a portable information terminal such as a smartphone or a tablet, or the like can also be configured as the image processing device. . Further, the image processing apparatus may be configured as an imaging system including an image processing apparatus such as a camera processor, an information processing apparatus, and a portable information terminal as described above, and an imaging optical system separated from the image processing apparatus.

  Hereinafter, referring to FIG. 22 and FIG. 23, an omnidirectional imaging device and an external device that receives a plurality of partial images captured by the omnidirectional imaging device and generates a combined omnidirectional image will be described. A celestial sphere imaging system according to another embodiment including the above computer device will be described. FIG. 22 schematically shows a spherical imaging system 400 according to another embodiment.

  The omnidirectional imaging system 400 according to the embodiment shown in FIG. 22 includes an omnidirectional imaging device 410 that is exclusively responsible for imaging, and a computer device 430 connected to the omnidirectional imaging device 410 and exclusively responsible for image processing. It is comprised including. Note that FIG. 22 shows only a main configuration, and a detailed configuration is omitted. In addition, components having the same functions as those in the embodiment described with reference to FIGS. 1 to 21 are denoted by the same reference numerals. Also, the omnidirectional imaging system 400 according to the embodiment shown in FIGS. 22 and 23 will be described with reference to FIGS. 1 to 21 except that the image processing for synthesizing the omnidirectional image is performed exclusively by the computer device 430. Since the configuration similar to that of the embodiment described above is provided, the following description will focus on the differences.

  In the embodiment shown in FIG. 22, the omnidirectional imaging device 410 includes a digital still camera processor 100, a lens barrel unit 102, and a three-axis acceleration sensor 120 connected to the processor 100. The lens barrel unit 102 has the same configuration as that shown in FIG. 2, and the processor 100 also has the same configuration as that shown in FIG.

  The processor 100 includes an ISP 108, a USB block 146, and a serial block 158, and controls USB communication with a computer device 430 connected via a USB connector 148. The wireless NIC 160 is connected to the serial block 158, and controls wireless communication with a computer device 430 connected via a network.

  The computer device 430 illustrated in FIG. 22 can be configured as a desktop personal computer, a general-purpose computer such as a workstation, or the like. The computing device 430 includes hardware components such as a processor, memory, ROM, storage, and the like. In the embodiment shown in FIG. 22, the computer device 430 includes a USB interface 432 and a wireless NIC 434, and is connected to the omnidirectional imaging device 410 via a USB bus or a network.

  The computer 430 further includes, as processing blocks for image synthesis, a position detection distortion correction unit 202, a connection position detection unit 204, a table correction unit 206, a table generation unit 208, and an image synthesis distortion correction unit 210. And an image synthesizing unit 212. In the present embodiment, the two partial images captured by the plurality of imaging optical systems of the lens barrel unit 102 and the conversion table for position detection of the omnidirectional imaging device 410 are connected to an external computer via a USB bus or a network. Transferred to device 430.

  In the computer device 430, the position detection distortion correction unit 202 performs distortion correction on the partial images 0 and 1 transferred from the omnidirectional imaging device 410 using the position detection conversion table transferred together. The correction images 0 and 1 for position detection are generated. The connection position detection unit 204 detects a connection position between the converted corrected images 0 and 1 and generates detection result data. The table correction unit 206 corrects the transferred position detection conversion table based on the detection result data. The table generation unit 208 performs rotation coordinate conversion from the corrected conversion data, and generates an image synthesis conversion table.

  The image-combination distortion correction unit 210 performs distortion correction on the original partial image 0 and the partial image 1 using the image-synthesis conversion table as a process preceding the image synthesis process, and 1 is generated. The image synthesizing unit 212 synthesizes the obtained image synthesizing corrected images 0 and 1 to generate a synthetic image in a spherical image format.

  The functional block illustrated in FIG. 22 can further include a display image generation unit 214. The display image generation unit 214 performs image processing for projecting the spherical image on the flat display device. The computer device 430 according to the present embodiment reads out a program from a ROM, an HDD, or the like, and develops the program in a work space provided by the RAM, thereby implementing the above-described functional units and the processes described below under the control of the CPU. .

  FIG. 23 is a flowchart illustrating an overall flow of a celestial sphere image combining process in the celestial sphere imaging system 400 according to another embodiment. FIG. 23 illustrates a flow from when a captured image is input by the omnidirectional imaging device 410 to when the image is stored in the computer device 430.

  The process illustrated in FIG. 23 is started from step S500, for example, in response to an instruction to shoot with the two imaging optical systems by pressing the shutter button on the omnidirectional imaging device 410. First, processing in the omnidirectional imaging device 410 is performed.

  In step S501, the celestial sphere imaging device 410 captures a partial image 0 and a partial image 1 using the two solid-state imaging devices 22A and 22B. In step S502, the omnidirectional imaging device 410 transfers the partial image 0 and the partial image 0 to the computer device 430 via a USB bus or a network. At the same time, the conversion table for position detection is transferred to the computer device 430 via a USB bus or a network. At this time, when the tilt correction is also performed on the computer device 430 side, the tilt information acquired by the three-axis acceleration sensor 120 is transferred to the computer device 430.

  The above-described conversion table for position detection of the omnidirectional imaging device 410 may be transferred when the omnidirectional imaging device 410 and the computer device 430 recognize each other. That is, it is sufficient that the conversion table for position detection has been transferred to the computer device 430 once, and it is not necessary to transfer the conversion table every time. The conversion table for position detection is stored in, for example, an SDRAM (not shown) or the like, and is read therefrom and transferred. The processing up to this point is the processing in the omnidirectional imaging device 410, and the processing from the next step S503 is executed in the computer 430 of the transfer destination.

  In step S503, the computer device 430 performs distortion correction on the transferred partial image 0 and partial image 1 using the transferred position detection conversion table by the position detection distortion correction unit 202, and The corrected images 0 and 1 are obtained. At this time, when the inclination correction is performed on the transfer destination computer device 430 side, the correction corresponding to the vertical direction may be performed in advance on the position detection conversion table based on the transferred inclination information. In step S504, the computer device 430 performs the connection position detection between the images in the overlap region of the position detection corrected images 0 and 1 by the connection position detection unit 204 to obtain detection result data. In step S505, the computer 430 corrects the transferred position detection conversion table using the detection result data by the table correction unit 206 so that the image is aligned on spherical coordinates. In step S506, the computer device 430 generates an image synthesis conversion table by performing rotation coordinate conversion from the corrected position detection conversion table by the table generation unit 208.

  In step S507, the computer device 430 performs distortion correction on the original partial images 0 and 1 by using the image synthesis conversion table by the image synthesis distortion correction unit 210, and converts the image synthesis corrected images 0 and 1 to each other. obtain. In step S508, the computer device 430 combines the corrected images for image combination 0 and 1 by the image combining unit 212. Through the synthesis processing, one celestial sphere image is generated from the two partial images captured by the fisheye lens. Then, in step S509, the computer device 430 saves the spherical image generated in step S508 in the external storage, and ends in step S510.

  The operation according to the flowchart in the other embodiment shown in FIG. 23 can also be executed by a program on a computer. That is, the CPU that controls the operation of the celestial sphere imaging device 410 and the CPU that controls the operation of the computer device 430 read programs stored in storage media such as ROM and RAM, respectively, and load the programs on the memory. Thus, the processing of each part in charge of the omnidirectional image synthesis processing described above is realized. FIGS. 22 and 23 illustrate the omnidirectional image capturing system configured separately, and are not limited to the specific embodiments illustrated in FIGS. 22 and 23. Each functional unit for realizing the omnidirectional imaging system can be distributedly mounted in one or more imaging devices and one or more computer systems in various modes.

  According to the embodiment described above, when detecting a connection position between a plurality of input images, an image processing apparatus capable of detecting an appropriate connection position even for a partial region having a small feature of the input image, An image processing method, a program, and an imaging system can be provided.

  Note that, in the above-described embodiment, the plurality of partial images are each taken at substantially the same time using different lens optical systems. However, in other embodiments, the same lens optical system is used, and at different times. In addition, the present invention may be applied to the superposition of a plurality of partial images photographed and taken in different photographing directions (azimuths) from a predetermined photographing point. Furthermore, in the above-described embodiment, two partial images captured by a lens optical system having an angle of view greater than 180 degrees are superimposed and synthesized, but in other embodiments, one or more lens optical systems are used. The present invention may be applied to superposition synthesis of three or more partial images captured by the system. In the above-described embodiment, an imaging system using a fish-eye lens has been described as an example. However, the invention may be applied to a spherical imaging system using an ultra-wide-angle lens. Furthermore, in the preferred embodiment described above, superposition of omnidirectional images has been described. However, the present invention is not particularly limited to this, and it is needless to say that the present invention can be applied to any image processing for detecting a connection position of a plurality of images. No.

  The functional unit can be realized by a computer-executable program written in a legacy programming language such as assembler, C, C ++, C #, Java (registered trademark), or an object-oriented programming language. ROM, EEPROM, EPROM , Flash memory, flexible disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray disk, SD card, MO, etc., stored in a readable recording medium, or through an electric communication line Can be distributed. Further, a part or all of the functional units can be mounted on a programmable device (PD) such as a field programmable gate array (FPGA), or is mounted as an ASIC (integrated for a specific application). Circuit configuration data (bit stream data) to be downloaded to the PD in order to realize the functional unit on the PD, HDL (Hardware Description Language) for generating the circuit configuration data, and VHDL (Very High Speed Integrated Circuits). Hardware description language), Verilog-HDL, and the like can be distributed on a recording medium as data.

  Although the embodiments of the present invention have been described above, the embodiments of the present invention are not limited to the above-described embodiments, and other embodiments, additions, modifications, deletions, and the like may be made by those skilled in the art. Modifications can be made within the range that can be made, and any aspect is included in the scope of the present invention as long as the functions and effects of the present invention are exhibited.

(Note)
(Appendix 1)
An image processing device for detecting a connection position between a plurality of input images,
Target image generating means for generating, from the first input image, a plurality of target images to be searched in the second input image,
For each of the plurality of target images, a feature amount calculating unit that calculates a feature amount,
For the target image of interest of the plurality of target images, connecting the target image to the second input image based on a connection position of another target image having a larger feature amount than the target image. An image processing apparatus, comprising: a connecting position detecting unit that detects a position.
(Appendix 2)
Tentative position calculating means for calculating a tentative position with respect to the target image based on a relative relationship between feature amounts between the target image and each of one or more peripheral target images;
A matching calculating unit that calculates matching between the target image and a portion of the second input image based on the temporary position calculated by the temporary position calculating unit with respect to the target image. 2. The image processing device according to 1.
(Appendix 3)
A process ranking unit that ranks the processes for the plurality of target images based on the feature amount and defines a relative relationship between the feature amounts, wherein the provisional position calculation unit is more than the attention target image. Using the determined joint position of the peripheral target image having a large feature amount and the initial joint position of the peripheral target image having a smaller feature amount than the target image, the temporary position with respect to the target image is calculated. 3. The image processing apparatus according to claim 2, wherein
(Appendix 4)
The tentative position calculating means may perform weighting according to a relative relationship between the feature amounts of each of the one or more peripheral target images with respect to the target image and a distance between each of the one or more peripheral target images from the target image. 4. The image processing apparatus according to claim 2, wherein the provisional position with respect to the target image is calculated by performing both or one of the weights according to the weight.
(Appendix 5)
The image processing according to any one of Supplementary notes 2 to 4, further comprising an evaluation value correction unit that corrects the evaluation value based on the similarity calculated by the matching calculation unit so that the evaluation value is prioritized around the temporary position. apparatus.
(Appendix 6)
A search range setting unit configured to set a search range in the matching in the second input image based on a relative relationship between feature amounts between the target image and one or more peripheral target images. The image processing apparatus according to any one of supplementary notes 2 to 5, including:
(Appendix 7)
An image processing apparatus that combines a plurality of input images,
A feature value obtaining unit configured to obtain a feature value of a plurality of target images in the first input image;
In the target image of interest among the plurality of target images, the first position is detected at a connection position detected using a connection position of another target image having a larger feature amount than the target image with respect to the second input image. An image processing apparatus comprising: a synthesizing unit that synthesizes an input image and the second input image.
(Appendix 8)
An image processing method for detecting a connection position between a plurality of input images, the computer comprising:
Generating, from the first input image, a plurality of target images to be searched in the second input image, respectively;
Calculating a feature amount for each of the plurality of target images;
Acquiring, for a target image of interest of the plurality of target images, a connection position of another target image around which a feature amount is larger than the target image;
Detecting a connection position of the target image to the second input image based on a connection position of the another target image with respect to the target image.
(Appendix 9)
A program for realizing an image processing device for detecting a connection position between a plurality of input images, the computer comprising:
Target image generating means for generating a plurality of target images to be searched in the second input image from the first input image,
A feature value calculating unit that calculates a feature value for each of the plurality of target images; and a target image of interest of the plurality of target images, the other target image having a larger feature value than the target image. A program for functioning as a connecting position detecting means for detecting a connecting position to the second input image based on the connecting position.
(Appendix 10)
An imaging system for detecting a connection position between a plurality of captured images,
Imaging means;
Means for acquiring a plurality of input images from a plurality of captured images, each of which is imaged in a plurality of directions, by the imaging means;
Target image generation means for generating a plurality of target images to be searched in the second input image from the first input image,
For each of the plurality of target images, a feature amount calculating unit that calculates a feature amount,
For the target image of interest of the plurality of target images, a connection of the target image to the second input image based on a connection position of another peripheral target image having a larger feature amount than the target image. An image capturing system, comprising: a connecting position detecting unit that detects a position.

10, 300 spherical imaging system, 12 imaging body, 14 housing, 18 shutter button, 20 imaging optical system, 22 solid-state imaging device, 100 processor, 102 lens barrel unit, 108 ... ISP, 110, 122 ... DMAC, 112 ... Arbiter (ARBMEMC), 114 ... MEMC, 116, 138 ... SDRAM, 118 ... distortion correction and image synthesis block, 120 ... three-axis acceleration sensor, 124 ... image processing block, 126 ... Image data transfer unit, 128 SDRAM, 130 CPU, 132 resize block, 134 JPEG block, 136 H. 264 blocks, 140 memory card control block, 142 memory card slot, 144 flash ROM, 146 USB block, 148 USB connector, 150 peripheral block, 152 audio unit, 154 speaker, 156 microphone 158: Serial block, 160: Wireless NIC, 162: LCD driver, 164: LCD monitor, 166 ... Power switch, 168 ... Bridge, 200 ... Function block, 202 ... Distortion correction unit for position detection, 204 ... Connection position detection unit, 206: Table correction unit, 208: Table generation unit, 210: Image synthesis distortion correction unit, 212: Image synthesis unit, 214: Display image generation unit, 220: Position detection conversion table, 220: Detection result data, 224: Conversion table for image synthesis, 23 ... Template generation unit, 234 ... Template feature amount calculation unit, 236 ... Template ranking unit, 238 ... Temporary position calculation unit, 240 ... Search range setting unit, 242 ... Matching calculation unit, 244 ... Score correction unit, 246 ... Connection position Determination unit, 248 detection result generation unit, 250 edge extraction filter unit, 252 gain multiplication unit, 254 LPF unit, 256 addition unit, 300 image for template, 302 image for template, 310 image for search, Reference numeral 312: Search range, 314: Corresponding part, 330: Search area, 332: Initial position, 334: Temporary position, 336: Adjoining connection position, 360: Search range, 410: Spherical sphere imaging device, 430: Computer device, 432: USB I / F, 434: Wireless NIC

JP 2001-148779 A

Claims (6)

An image processing apparatus that combines a plurality of input images,
A feature value obtaining unit configured to obtain a feature value of a plurality of target images in the first input image;
Inclination information acquisition means for acquiring inclination information;
Using the connection position between the first input image and the second input image for another target image having a larger feature amount than the target image of interest among the plurality of target images, the target image and the target image The first input image and the second input image based on a joint position detected by performing a search between a portion of the second input image and the inclination information acquired by the inclination information acquiring means ; An image processing apparatus, comprising: synthesizing means for synthesizing an input image. Temporary position calculating means for calculating a temporary position based on a relative relationship between feature amounts between the target image and each of one or more peripheral target images;
Matching calculation means for calculating an evaluation value of matching with the corresponding part of the second input image based on the temporary position calculated by the temporary position calculation means for the target image; A connection position based on combining the input image and the second input image is determined based on the first input image and the second input image of the target image determined based on the evaluation value. The image processing device according to claim 1, wherein the image processing device is a connection position.   A process ranking unit that ranks the processes for the plurality of target images based on the feature amounts and defines a relative relationship between the feature amounts, wherein the temporary position calculation unit determines the relative positions of the plurality of target images based on the target image. Also, using the determined connection position of the peripheral target image having a large feature amount and the initial connection position of the peripheral target image having a smaller feature amount than the target image, the temporary connection position of the target image is calculated. The image processing apparatus according to claim 2, wherein the position is calculated.   The tentative position calculating means may perform weighting according to a relative relationship between the feature amounts of each of the one or more peripheral target images with respect to the target image and a distance between each of the one or more peripheral target images from the target image. The image processing apparatus according to claim 2, wherein the provisional position of the target image is calculated by performing both or one of the weights according to the weight.   The evaluation value correction unit according to any one of claims 2 to 4, further comprising an evaluation value correction unit that corrects the evaluation value based on the similarity calculated by the matching calculation unit so that the evaluation value is prioritized around the temporary position. Image processing device.   The search range setting unit according to claim 2, further comprising a search range setting unit configured to set a search range in the matching based on a relative relationship between feature amounts between the target image and each of one or more peripheral target images. The image processing device according to claim 1.