JP2016006674A - Image processor, program, image processing method and imaging system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor for compositing a plurality of input images, and to provide a program, an image processing method and an imaging system.SOLUTION: This image processor 10 includes feature amount acquisition means 234 for acquiring a feature amount of a prescribed object image in a first input image, connection position detection means 246 for detecting a connection position between the first input image and a second input image with a system corresponding to the feature amount, and composition means 212 for compositing the first input image and the second input image at the connection position.

Description

  The present invention relates to an image processing device, a program, an image processing method, and an imaging system, and more particularly to an image processing device, a program, an image processing method, and an imaging system for joining a plurality of input images.

  2. Description of the Related Art An omnidirectional imaging system is known that uses a plurality of wide-angle lenses such as fish-eye lenses and super-wide-angle lenses to image omnidirectional (hereinafter referred to as omnidirectional sphere) at once. In the omnidirectional imaging system, an image from each lens is projected onto the sensor surface, and the obtained images are combined by image processing to generate an omnidirectional image. For example, an omnidirectional image can be generated using two wide-angle lenses having an angle of view exceeding 180 degrees.

  In the image processing, distortion correction and projective transformation are performed on the partial images captured by each lens optical system based on a predetermined projection model and taking into account distortion from an ideal model. Then, the partial images are connected using the overlapping portions included in the partial images, and processing to form one omnidirectional image is performed. In the process of joining the images, a joining position where the subject overlaps is detected using pattern matching or the like in the overlapping part between the partial images.

  However, with the joint position detection technique based on the conventional pattern matching, if the target area to be matched is a flat area or an area in which the same pattern is repeated, and there are few features, the appropriate joint area is detected. It was difficult to detect the position. For this reason, the partial images cannot be stitched well, and the quality of the obtained omnidirectional image may be deteriorated.

  Various techniques for connecting a plurality of partial images captured using a plurality of cameras are known. For example, Japanese Patent Laid-Open No. 2001-148777 (Patent Document 1) avoids pattern matching errors caused by using inappropriate matching regions when performing pattern matching, and synthesizes images with high accuracy. An image processing apparatus intended to perform the above is disclosed. The prior art of Patent Document 1 is configured to determine whether or not the matching area is appropriate for the matching area extracted by the matching area extraction unit, and perform extraction again for an inappropriate matching area. . Further, the matching result by the pattern matching unit is also determined as to whether or not the matching result is appropriate, and if the matching result is not appropriate, extraction of the matching region is performed again.

  The prior art disclosed in Patent Document 1 removes in advance areas that are not suitable as matching areas, such as areas of white pixels and black pixels, and areas where straight lines that continue in the vertical, horizontal, and diagonal directions exist. By performing pattern matching, the accuracy of image synthesis is improved.

  However, the prior art disclosed in Patent Document 1 is a technique of re-extracting a matching area when the matching area is inappropriate. For this reason, it is still difficult to determine an appropriate connection position with respect to the matching area determined to be inappropriate.

  The present invention has been made in view of the insufficiency of the prior art described above, and the present invention is suitable for combining a plurality of input images according to the feature amount of a predetermined target image of the input images. An object of the present invention is to provide an image processing device, a program, an image processing method, and an imaging system capable of detecting a connection position and combining them at the detected connection position.

  In order to solve the above-mentioned problems, the present invention provides an image processing apparatus for synthesizing a plurality of input images having the following characteristics. The image processing apparatus includes a feature amount acquisition unit that acquires a feature amount of a predetermined target image in the first input image, and a connection between the first input image and the second input image by a method according to the feature amount. A connecting position detecting means for detecting a position; and a combining means for combining the first input image and the second input image at the connecting position.

  According to the above configuration, when combining a plurality of input images, it is possible to detect an appropriate connection position according to the feature amount of a predetermined target image of the input image and combine the detected input positions.

A sectional view showing an omnidirectional imaging system by this embodiment. The hardware block diagram of the omnidirectional imaging system by this embodiment. The figure which shows the flow of the whole image processing in the omnidirectional imaging system by this embodiment. The main functional block diagram of the omnidirectional image synthetic | combination process implement | achieved on the omnidirectional imaging system by this embodiment. The flowchart which shows the whole flow of the omnidirectional image synthesis process which the omnidirectional imaging system by this embodiment performs. The figure explaining the projection relationship in the omnidirectional imaging system using a fisheye lens. The figure explaining the data structure of the image data of the omnidirectional image format used by this embodiment. The figure explaining the conversion data which the distortion correction part for position detection and the distortion correction part for image composition refer to. The figure explaining the mapping to the spherical coordinate system of the two partial images imaged with the two fisheye lenses in the case of a position detection process. The functional block diagram of the connection position detection part by this embodiment. The block diagram of the template feature-value calculation part by specific embodiment. The flowchart which shows the joint position detection process which the omnidirectional imaging system by this embodiment performs. The figure explaining the production | generation method of the template image by the template production | generation part by this embodiment. The figure explaining the calculation method of the temporary position of the template image based on ranking of the template image by this embodiment and (B, C) the ranking performed by the temporary position calculation part by this embodiment. The figure explaining the setting method of the search range which the search range setting part by this embodiment performs. The flowchart which shows the (A) matching calculation process and the (B) score correction process which the omnidirectional imaging system by this embodiment performs. The figure explaining the process which searches a template image in a search range by (A) template image and (B) pattern matching. (A) A graph of an offset function, and (B, C) a graph in which the score before correction and the score after correction are plotted at the search position are shown together with the matching position based on the score before correction and the matching position based on the score after correction. Figure. The figure which shows the data structure of the detection result data produced | generated by the connection position detection part in this embodiment. The figure explaining the process which produces | generates the detection result data by the connection position detection part of this embodiment. 5 is a flowchart showing processing for generating an image composition conversion table executed by the omnidirectional imaging system according to the present embodiment. The figure explaining the mapping to the spherical coordinate system of the two partial images imaged with the two fisheye lenses in the case of an image synthesis process. Schematic of the omnidirectional imaging system in other embodiments. The flowchart figure which shows the whole flow of the omnidirectional image synthetic | combination process in the omnidirectional imaging system in other embodiment.

  Hereinafter, although embodiment of this invention is described, embodiment of this invention is not limited to embodiment described below. In the following embodiments, as an example of an image processing apparatus and an imaging system, an imaging body including two fisheye lenses is included in an optical system, and distortion correction and projection are performed on two partial images captured by the two fisheye lenses themselves. A description will be given using an omnidirectional imaging system 10 having an image processing function for performing conversion, image joining, and generating an omnidirectional image.

[overall structure]
Hereinafter, the overall configuration of the omnidirectional imaging system according to the present embodiment will be described with reference to FIGS. FIG. 1 is a sectional view showing an omnidirectional imaging system (hereinafter simply referred to as an imaging system) 10 according to the present embodiment. An imaging system 10 illustrated in FIG. 1 includes an imaging body 12, a casing 14 that holds the imaging body 12, and components such as a controller and a battery, and a shutter button 18 provided on the casing 14. The imaging body 12 shown in FIG. 1 includes two imaging optical systems 20A and 20B and two solid-state imaging elements 22A and 22B such as a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. A combination of the imaging optical system 20 and the solid-state imaging element 22 one by one is referred to as an imaging optical system. Each of the imaging optical systems 20 can be configured as, for example, six groups of seven fisheye lenses. In the embodiment shown in FIG. 1, the fisheye lens has a total angle of view greater than 180 degrees (= 360 degrees / n; n = 2), and preferably has an angle of view of 185 degrees or more. Has an angle of view of 190 degrees or more.

  The optical elements (lens, prism, filter, and aperture stop) of the two imaging optical systems 20A and 20B have a positional relationship with respect to the solid-state imaging elements 22A and 22B. Positioning is performed so that the optical axes of the optical elements of the imaging optical systems 20A and 20B are positioned perpendicular to the center of the light receiving area of the corresponding solid-state imaging element 22, and the light receiving area is connected to the corresponding fisheye lens. It is performed so that it becomes an image plane. Each of the solid-state imaging elements 22 is a two-dimensional solid-state imaging element in which a light receiving region forms an area area, and converts the light collected by the combined imaging optical system 20 into an image signal.

  In the embodiment shown in FIG. 1, the imaging optical systems 20A and 20B have the same specifications, and are combined in opposite directions so that their optical axes match. The solid-state imaging elements 22A and 22B convert the received light distribution into image signals and output them to image processing means on a controller (not shown). Although the details will be described later in the image processing means, the partial images respectively input from the solid-state image pickup devices 22A and 22B are connected and combined to form an image with a solid angle of 4π radians (hereinafter referred to as “global celestial image”). Is generated. The omnidirectional image is an image of all directions that can be seen from the shooting point. Here, in the embodiment shown in FIG. 1, the omnidirectional image is generated, but a so-called panoramic image obtained by photographing 360 degrees only on the horizontal plane may be used.

  Here, the captured images can be easily joined by matching the scanning directions of the solid-state imaging devices 22A and 22B to each other. That is, by matching the scanning direction and order of the solid-state imaging elements 22 at the portions where they are connected to each other, it is possible to obtain an effect for connecting objects located at the boundaries of the cameras, particularly moving objects. For example, when the upper left part of the captured image captured by the solid-state image sensor 22A and the lower left part of the captured image captured by the solid-state image sensor 22B coincide with each other as a part where the images are joined, the solid-state image sensor 22A The scanning is performed from right to left from the top to the bottom of the solid-state imaging device. On the other hand, the solid-state image sensor 22B scans from the right to the left from the bottom to the top of the solid-state image sensor. As described above, by controlling so that the scanning directions of the respective solid-state imaging devices 22 coincide with each other based on the portion where the images are to be joined, an effect that the joining is easy can be obtained.

  As described above, since the fisheye lens has a total angle of view exceeding 180 degrees, when composing an omnidirectional image, reference data in which overlapping image portions represent the same image in the captured images by the respective imaging optical systems. As a reference for stitching images together. The generated omnidirectional image is, for example, an external storage such as a display device, a printing device, an SD (registered trademark) card, or a compact flash (registered trademark) provided in or connected to the imaging body 12. Output to media.

  FIG. 2 shows a hardware configuration of the imaging system 10 according to the present embodiment. The imaging system 10 includes a digital still camera processor (hereinafter simply referred to as a processor) 100, a lens barrel unit 102, and various components connected to the processor 100. The lens barrel unit 102 includes the two sets of lens optical systems 20A and 20B and the solid-state imaging elements 22A and 22B. The solid-state imaging device 22 is controlled by a control command from a CPU 130 described later in the processor 100.

  The processor 100 includes an ISP (Image Signal Processor) 108, a DMAC (Direct Memory Access Controller) 110, an arbiter (ARBMEMC) 112 for arbitrating memory access, a MEMC (Memory Controller) 114 for controlling memory access, And a distortion correction / image synthesis block 118. The ISPs 108A and 108B perform white balance setting and gamma setting on the image data input through the signal processing of the solid-state imaging devices 22A and 22B, respectively. The SDRAM 116 is connected to the MEMC 114. Data is temporarily stored in the SDRAM 116 when processing is performed in the ISPs 108A and 180B and the distortion correction / image synthesis block 118. The distortion correction / image synthesis block 118 performs distortion correction and top-and-bottom correction on the two partial images obtained from the two imaging optical systems using the information from the three-axis acceleration sensor 120 to synthesize an image.

  The processor 100 further includes a DMAC 122, an image processing block 124, a CPU 130, an image data transfer unit 126, an SDRAM C 128, a memory card control block 140, a USB block 146, a peripheral block 150, and an audio unit 152. Serial block 158, LCD (Liquid Crystal Display) driver 162, and bridge 168.

  The CPU 130 controls the operation of each unit of the imaging system 10. The image processing block 124 includes a resize block 132, a JPEG block 134, an H.264 image, and an H.264 image. Various image processing is performed on the image data using a H.264 block 136 or the like. The resize block 132 is a block for enlarging or reducing the size of the image data by interpolation processing. The JPEG block 134 is a codec block that performs JPEG compression and expansion. H. H.264 block 136 is an H.264 block. It is a codec block that performs video compression and decompression such as H.264. The image data transfer unit 126 transfers the image processed by the image processing block 124. The SDRAM C 128 controls the SDRAM 138 connected to the processor 100, and the SDRAM 138 temporarily stores image data when various processes are performed on the image data in the processor 100.

  The memory card control block 140 controls reading and writing with respect to the memory card inserted into the memory card slot 142 and the flash ROM 144. The memory card slot 142 is a slot for detachably attaching a memory card to the imaging system 10. The USB block 146 controls USB communication with an external device such as a personal computer connected via the USB connector 148. A power switch 166 is connected to the peripheral block 150. The audio unit 152 is connected to a microphone 156 from which a user inputs an audio signal and a speaker 154 from which a recorded audio signal is output, and controls audio input / output. The serial block 158 controls serial communication with an external device such as a personal computer, and is connected with a wireless NIC (Network Interface Card) 160. The LCD driver 162 is a drive circuit that drives the LCD monitor 164 and converts it into a signal for displaying various states on the LCD monitor 164.

  The flash ROM 144 stores a control program and various parameters described by codes that can be read by the CPU 130. When the power is turned on by operating the power switch 166, the control program is loaded into the main memory. The CPU 130 controls the operation of each part of the apparatus according to a program read into the main memory, and temporarily stores data necessary for control in the SDRAM 138 and a local SRAM (not shown).

  FIG. 3 is a diagram illustrating the flow of the entire image processing in the imaging system 10 according to the present embodiment. First, in steps S101A and 101B, an image is captured by each of the solid-state imaging devices 22A and 22B. In steps S102A and 102B, the optical black correction process, the defective pixel correction process, the linear correction process, the shading process, and the like are performed on the Bayer RAW image output from each of the solid-state imaging devices 22A and 22B by the ISP 108 illustrated in FIG. Region division averaging processing is performed. In steps S103A and 103B, the image is stored in the memory. In steps S104A and 104B, white balance processing, gamma correction processing, Bayer interpolation processing, YUV conversion processing, edge enhancement processing, and color correction processing are further performed by the ISP 108 shown in FIG. 2, and in steps S105A and 105B, Saved in memory.

  When the processing described above is completed for each of the two solid-state imaging devices 22A and 22B, in step S106, distortion correction and composition processing are performed on each partial image subjected to the above processing. In step S107, the image is appropriately tagged and the omnidirectional image is saved as a file in the built-in memory or external storage. In addition, in the process of the distortion correction and the synthesis process, the inclination top / bottom correction may be performed by obtaining information from the triaxial acceleration sensor 120 as appropriate. Further, the stored image file may be appropriately compressed.

[Spherical image composition function]
Hereinafter, the omnidirectional image synthesis function included in the imaging system 10 according to the present embodiment will be described in detail with reference to FIGS. 4 to 22. FIG. 4 shows main functional blocks 200 of the omnidirectional image synthesis processing realized on the imaging system 10 according to the present embodiment. As shown in FIG. 4, the distortion correction / image synthesis block 118 includes a position detection distortion correction unit 202, a joint position detection unit 204, a table correction unit 206, a table generation unit 208, and an image synthesis distortion correction unit. 210 and an image composition unit 212.

  In addition, two partial images are input to the distortion correction / image synthesis block 118 through image signal processing by the ISPs 108A and 108B from the two solid-state imaging devices 22A and 22B. Here, the solid-state image pickup devices 22A and 22B are referred to with the numbers “0” and “1”, and an image using the solid-state image pickup device 22A as a source is referred to as “partial image 0”. An image having the element 22B as a source is referred to as “partial image 1”. Further, the distortion correction / image synthesis block 118 is provided with a position detection conversion table 220 created in advance by a manufacturer or the like according to a predetermined projection model based on design data of each lens optical system.

  The position detection distortion correction unit 202 performs distortion correction on the input partial image 0 and partial image 1 using the position detection conversion table 220 as the preceding process of the joint position detection process, thereby correcting the position detection. An image (hereinafter, simply referred to as a corrected image) 0 and a position detection corrected image 1 are generated. The input partial images 0 and 1 are images captured by a two-dimensional solid-state imaging device in which a light receiving region forms an area area, and are image data expressed in a plane coordinate system (x, y). On the other hand, the corrected image subjected to distortion correction using the position detection conversion table 220 is image data of a coordinate system different from the input image, and more specifically, a spherical coordinate system (radial radius is 1). And a polar coordinate system having two declination angles θ and φ).

  FIG. 6 is a diagram for explaining a projection relationship in an imaging system using a fisheye lens. In the present embodiment, an image photographed with one fisheye lens is a photograph of the direction of approximately a hemisphere from the photographing point. In addition, as shown in FIG. 6, the fisheye lens generates an image with an image height h corresponding to the incident angle φ with respect to the optical axis. The relationship between the image height h and the incident angle φ is determined by a projection function corresponding to a predetermined projection model. Although the projection function differs depending on the nature of the fisheye lens, the projection model fisheye lens called the equidistant projection method is expressed by the following formula (1), where f is the focal length.

  Other projection models include a central projection method (h = f · tan φ), a solid projection method (h = 2f · tan (φ / 2)), and an equal solid angle projection method (h = 2f · sin (φ / 2). )) And an orthogonal projection method (h = f · sinφ). In either method, the image height h of the image is determined corresponding to the incident angle φ from the optical axis and the focal length f. In the present embodiment, a configuration of a so-called circumferential fisheye lens in which the image circle diameter is smaller than the image diagonal line is employed, and the obtained partial image is approximately a hemisphere of the shooting range as shown in FIG. The plane image includes the entire image circle onto which the minutes have been projected.

  FIG. 7 is a diagram for explaining the data structure of image data in the omnidirectional image format used in the present embodiment. As shown in FIG. 7, the image data in the omnidirectional image format includes pixel values whose coordinates are a vertical angle φ corresponding to an angle formed with respect to a predetermined axis and a horizontal angle θ corresponding to a rotation angle around the axis. Expressed as an array of The horizontal angle θ is in the range of 0 to 360 degrees (can be expressed as −180 degrees to +180 degrees), and the vertical angle φ is in the range of 0 to 180 degrees (also can be expressed as −90 degrees to +90 degrees). It becomes. Each coordinate value (θ, φ) is associated with each point on the spherical surface representing the omnidirectional centered on the photographing point, and the omnidirectional is mapped on the omnidirectional image. The relationship between the plane coordinates of the image taken with the fisheye lens and the coordinates on the spherical surface of the omnidirectional image format can be associated by using the projection function as described with reference to FIG.

  FIG. 8 is a diagram for explaining conversion data referred to by the position detection distortion correction unit 202 and the image composition distortion correction unit 210. The conversion tables 220 and 224 define the projection from the partial image expressed in the plane coordinate system to the image expressed in the spherical coordinate system. As shown in FIGS. 8A and 8B, the conversion tables 220 and 224 are mapped to the coordinate value (θ, φ) of the corrected image and the coordinate value (θ, φ) for each fisheye lens. The information for associating the coordinate values (x, y) of the partial image before correction are all coordinate values (θ, φ) (θ = 0,... 360 degrees, φ = 0,..., 180 degrees). Hold against. In the example of FIG. 8, the angle handled by one pixel is 1/10 degrees in both the φ direction and the θ direction, and the conversion tables 220 and 224 have information indicating a correspondence relationship of 3600 × 1800 for each fisheye lens. It will be.

  The position detection conversion table 220 used in the connection position detection is an ideal result caused by radial distortion, eccentric distortion, etc. based on the projection relationship of the lens explained in FIG. It is calculated and corrected into a table after correcting distortion from a typical lens model. In contrast, the image composition conversion table 224 is generated from the position detection conversion table 220 by a predetermined conversion process, as will be described in detail later. In the embodiment to be described, the conversion data is data in which the correspondence relationship of coordinate values is tabulated. However, in other embodiments, the transformation data is one or more that defines a projection from a partial image (x, y) expressed in a planar coordinate system to an image (θ, φ) expressed in a spherical coordinate system. It is good also as coefficient data of the function.

  Referring to FIG. 4 again, the position detection distortion correction unit 202 converts the partial image 0 and the partial image 1 with reference to the position detection conversion table 220, and corrects the position detection correction image 0 and the position detection correction image. 1 is generated. More specifically, the position detection distortion correction unit 202 refers to the position detection conversion table 220 for all coordinate values (θ, φ) of the converted corrected image, and sets each coordinate value (θ, φ). The coordinate value (x, y) of the partial image before conversion to be mapped is obtained, and the pixel value in the partial image of the coordinate value (x, y) is referred to. As a result, a corrected image is generated.

  FIG. 9 is a diagram for describing mapping of two partial images captured by two fisheye lenses to a spherical coordinate system in the position detection process. As a result of the processing by the position detection distortion correction unit 202, the two partial images 0 and 1 captured by the fisheye lens are developed on an omnidirectional image format as shown in FIG. The partial image 0 captured by the fisheye lens 0 is typically mapped to approximately the upper hemisphere of the entire celestial sphere, and the partial image 1 captured by the fisheye lens 1 is approximately aligned to the approximately lower hemisphere of the entire celestial sphere. To be mapped. The corrected image 0 and the corrected image 1 expressed in the omnidirectional format protrude from the hemisphere because the total angle of view of the fisheye lens exceeds 180 degrees. As a result, when the corrected image 0 and the corrected image 1 are superimposed, Overlapping areas with overlapping shooting ranges occur.

  As will be described in detail later, after correction by the position detection distortion correction unit 202, the connection position detection unit 204 detects a connection position between images in the overlap region. In the position detection conversion table 220 according to the present embodiment, as shown in FIG. 9, the optical axes of the two lens optical systems are projected onto two spherical poles (φ = 0 degrees, 180 degrees), and between the images. Are created so that they are projected in the vicinity of the equator of the spherical surface (φ = 90 degrees ± ((total angle of view−180 degrees) / 2)). In the spherical coordinate system, the closer to the pole where the vertical angle φ is 0 degree or 180 degrees, the greater the distortion, and the joint position detection accuracy deteriorates. On the other hand, by using the projection as described above, the overlapping position is detected in the vicinity of a vertical angle of 90 degrees where the amount of distortion when shifted in the θ direction is small, and the connecting position is detected. The position detection accuracy can be improved. As a result, it is possible to detect a connection position with high accuracy even for an image captured by a lens optical system having a large distortion.

  Here, referring to FIG. 4 again, the joint position detection unit 204 receives the correction images 0 and 1 converted by the position detection distortion correction unit 202 and receives the input correction image 0 by pattern matching processing. , 1 is detected, and detection result data 222 is generated.

  In pattern matching, typically, if the target area to be matched is a flat image or an area in which the same pattern is repeated and there are few features, it is possible to detect the connection position with high accuracy. It becomes difficult. Therefore, the connection position detection unit 204 according to the present embodiment measures a feature amount that indicates the degree of the feature of the image of the target region to be matched, and uses this measured feature amount to determine the connection position of the target region. Adopt a configuration to determine. As a result, stitching is performed by an appropriate evaluation method according to the feature of the region, thereby improving the quality of the obtained omnidirectional image.

  FIG. 10 is a diagram showing functional blocks of the connection position detection unit according to the present embodiment shown in FIG. More specifically, the connection position detection unit 204 shown in FIG. 10 includes a template generation unit 232, a template feature amount calculation unit 234, a template ranking unit 236, a temporary position calculation unit 238, and a search range setting unit 240. , A matching calculation unit 242, a score correction unit 244, a connection position determination unit 246, and a detection result generation unit 248. Hereinafter, the connection position detection processing by template matching will be described. For convenience of explanation, the template side is referred to as a position detection correction image 1 and the searched side is referred to as a position detection correction image 0.

  The template generation unit 232 uses the position detection correction image 1 as a template image in template matching, and uses a plurality of images (hereinafter referred to as template images) to be searched for in the search image from the position detection correction image 1. Is generated). In the joint position detection process by template matching, the joint position to the position detection correction image 0 is obtained for each template image that is a part of the position detection correction image 1. Here, since the connection position in the overlapping area between the two correction images 0 and 1 is intended, a plurality of template images are generated from the overlapping area portion of the entire position detection correction image 1.

  The template feature amount calculation unit 234 calculates a feature amount for each of the plurality of template images generated by the template generation unit 232. Here, the feature amount is a value for indexing the degree of the feature of the template image. In the present embodiment, the large feature amount means that a large feature is recognized in the image, and the feature amount. “Small” means that the image has few features and no individuality. As the feature amount, at least one of the edge amount extracted from the template image, the variance calculated from the template image, and the standard deviation can be used, but it is not particularly limited.

  FIG. 11 shows a block diagram of a template feature quantity calculation unit according to a specific embodiment. FIGS. 11A to 11C show the configuration of the template feature quantity calculation unit 234A according to the embodiment using the edge quantity as an index. On the other hand, FIG. 11D shows a configuration of the template feature amount calculation unit 234B according to the embodiment using standard deviation or variance as an index.

  When the edge amount is used as the feature amount of the template image, an edge enhancement block for performing edge enhancement as shown in FIG. 11A can be used. The edge enhancement block illustrated in FIG. 11A includes an edge extraction filter unit 250, a gain multiplication unit 252, and an LPF (Low-Pass Filter) unit 254. In the edge enhancement block, a signal is generated by adjusting the edge amount by multiplying the edge amount extracted by the edge extraction filter unit 250 by the gain multiplication unit 252. Then, a signal obtained by adding the adjusted signal and the signal obtained by removing the noise by performing the LPF process on the input signal by the LPF unit 254 by the adding unit 256 is output as an edge-emphasized signal.

  In the present embodiment, only the edges need be extracted. For this reason, for example, the LPF coefficient as shown in FIG. 11C is used to make the signal subjected to the LPF processing zero, and the edge extracted by the edge extraction filter unit 250 using the edge extraction filter coefficient shown in FIG. Only the amount needs to be output. By inputting the template image to the edge enhancement block shown in FIG. 11A, the edge amount for each pixel is output. As the feature amount for the template image, the sum of the edge amounts of all the pixels of the template may be used, or the average of the edge amounts may be used. It means that as the total amount of edges of the template image is larger, the image includes more portions where the brightness changes discontinuously. Therefore, a flat image typically has a small edge amount and a small feature amount.

When the standard deviation (or variance) of the image is used as the feature amount of the template image, a configuration as shown in FIG. 11D can be provided. The standard deviation (or variance) calculation unit 258 shown in FIG. 11D calculates the standard deviation σ (or variance σ ² ) of the template image using the following calculation formula (2). In the following formula (2), the total number of pixels of the template image is N (= W pixels × H pixels), and the luminance at the coordinates (i, j) of the template image is T (i, j). The larger the standard deviation (or variance) of the template image, the wider the luminance is distributed in the histogram. Therefore, a flat image typically has a narrow brightness distribution of the image and a small feature amount.

  Although the standard deviation and the variance are exemplified as the feature amount of the template image, the kurtosis in the density histogram obtained from the template image (the degree of whether the histogram distribution is concentrated around the average value or spreads toward the tail) And other statistical indicators such as contrast based on density histograms, contrast based on differential statistics, etc., and the degree of distortion (representing the degree to which the histogram is distorted from a symmetric shape) An index may be used to characterize the template image. In the above description, the feature value is calculated using the luminance value. However, the feature value may be calculated for each color of RGB, and the feature value of the template image may be calculated as the sum of the feature values. Feature quantities such as edge quantity, standard deviation, and variance can be easily calculated from the target image, which is preferable because the calculation cost is low and good detection results can be obtained. However, the present invention is not limited to this, and any index value that indicates the degree of characteristics of an image can be used.

  Referring to FIG. 10 again, the template ranking unit 236 ranks the processing order of the plurality of template images based on the feature amounts calculated by the template feature amount calculation unit 234, and calculates the feature amounts between the template images. Define relative relationships.

  The temporary position calculation unit 238 calculates a temporary position that is used as a reference for a template matching process to be described later for a template image of interest among the generated template images. Here, the temporary position is calculated on the basis of the relative relationship of the feature amounts between the template image of interest and each of the surrounding template images on the position detection correction image 1.

  Template matching is a process of searching for a template image from search images to be searched. If a region corresponding to the template image is specified to some extent, a search range in the search image is determined. It can be limited. The search range setting unit 240 sets a search range for template matching based on the relative relationship of feature amounts between the template image of interest and each of the peripheral template images for the template of interest. Typically, a predetermined search range in the position detection image 0 around the temporary position calculated by the temporary position calculation unit 238 is set. Further, a narrowed search range may be set based on the connection positions determined for the peripheral template image.

  The matching calculation unit 242 performs matching between the target template image and each portion of the position detection correction image based on the calculated temporary position, typically by the template matching method, for the target template image. calculate. In the matching calculation, a matching score, which is an evaluation value based on the similarity of images at each position, is moved while the template image is moved in the search range centered on the temporary position set by the search range setting unit 240. Calculated. Further, in the present embodiment, a plurality of evaluation methods are prepared according to the classification of the feature amount of the template image. Then, the matching calculation unit 242 according to the present embodiment calculates a matching score by switching the matching evaluation method used at that time according to the feature amount calculated by the template feature amount calculation unit 234.

  The score correction unit 244 preferentially increases the score centered on the calculated temporary position with respect to the score based on the similarity of the image calculated by the matching calculation unit 242 with respect to the template image of interest. Perform offset correction. In addition, the range which a score can take changes with the used evaluation methods. For this reason, the score correction unit 244 switches the offset correction method according to the evaluation method used by the matching calculation unit 242. Thus, it is possible to correct the score appropriate for the matching evaluation method, taking into account the temporary position calculated based on the relative relationship of the feature amounts between the template images.

  The connection position determination unit 246 determines the connection position of the target template image at a position where the corrected score is maximized on the position detection correction image 0. The connection position determined at this time is obtained by adding the connection position of the peripheral template image having a larger feature amount than the template image of interest.

  When the processing by the temporary position calculation unit 238, the search range setting unit 240, the matching calculation unit 242, the score correction unit 244, and the connection position determination unit 246 described above is performed for each of the plurality of template images, the position corresponding to each template image A connection position on the detection correction image 0 is obtained. The detection result generation unit 248 calculates the connection position for each pixel (θ, φ) in the omnidirectional format based on the connection position data set corresponding to each template image obtained here, and the detection result. Data 222 is generated.

  Referring back to FIG. 4, the table correction unit 206 corrects the position detection conversion table 220 prepared in advance based on the detection result data 222 and passes the correction to the table generation unit 208. The table generation unit 208 generates an image composition conversion table 224 based on the rotation coordinate conversion from the conversion data corrected by the table correction unit 206.

  The image composition distortion correction unit 210 performs distortion correction on the original partial image 0 and the partial image 1 using the image composition conversion table 224 as the preceding process of the image composition processing, and corrects the image composition corrected image 0. And the corrected image 1 for image synthesis is generated. Like the position detection correction image, the generated image composition correction image is expressed in a spherical coordinate system, but the definition of coordinate axes differs from that of the position detection correction image due to the rotational coordinate conversion. It becomes. The image composition unit 212 synthesizes the obtained image composition correction image 0 and the image composition correction image 1 to generate a composite image in the omnidirectional image format. Note that the processing executed by the connection position detection unit 204, the table correction unit 206, the table generation unit 208, the image composition distortion correction unit 210, and the image composition unit 212 will be described in detail later along with the description of the processing flow.

  The functional block 200 illustrated in FIG. 4 can further include a display image generation unit 214. Since the generated composite image is expressed in an omnidirectional image format, when displayed as it is on a flat display device such as a display, the image is displayed with distortion as the vertical angles approach 0 degrees and 180 degrees. become. The display image generation unit 214 is a unit that executes image processing for projecting the omnidirectional image onto a flat display device. The display image generation unit 214 converts, for example, a composite image in the omnidirectional image format from a spherical coordinate system to a plane coordinate system with a specific direction and a specific angle of view, and has a fixed angle of view in a specific viewing direction specified by the user. A process of projecting onto an image can be performed.

[Flow of spherical image composition processing]
Hereinafter, the flow of the omnidirectional image synthesis process according to the present embodiment will be described with reference to FIGS. 5, 12, and 21. FIG. 5 is a flowchart showing an overall flow of the omnidirectional image synthesis process executed by the imaging system 10 according to the present embodiment. The processing shown in FIG. 5 is started from step S200 in response to, for example, that the photographing is instructed by the two imaging optical systems by pressing the shutter button 18 and the instruction is issued from the CPU 130.

  In step S201, the imaging system 10 uses the position detection conversion table 220 to distort the partial image 0 and the partial image 1 acquired by the two solid-state imaging devices 22A and 22B by the position detection distortion correction unit 202. Make corrections. As a result, a corrected image for position detection 0 and a corrected image for position detection 1 are obtained. As a result, a corrected image in the omnidirectional image format as shown in FIG. 9 is obtained. In step S <b> 202, the imaging system 10 detects a connection position between images in the overlapping region of the position detection correction image 0 and the position detection correction image 1 by the connection position detection unit 204.

  FIG. 12 is a flowchart illustrating a connection position detection process executed by the imaging system 10 according to the present embodiment. The process shown in FIG. 12 is called in step S202 shown in FIG. 5, and is started from step S300. In step S301, the imaging system 10 performs initial setting of a template image, a search image to be searched, a template block size, a start coordinate for generating a template, a template generation interval, and the total number of blocks.

  FIG. 13 is a diagram illustrating a template image generation method by the template generation unit according to the present embodiment. In the embodiment to be described, the template image 300 is an image of the overlapping area portion of the position detection correction image 1, and the search image 310 is an image of the overlapping area portion of the position detection correction image 0. The block size is a pixel size constituting a template image, and the generation interval is an interval at which adjacent template images are generated. The generation start coordinates are coordinates for cutting out the first template image. Note that the block size and generation interval may be determined in consideration of the desired accuracy and processing amount of stitching.

  Assuming that the block size is W pixels × H pixels, the generation start coordinates are (sx, sy), and the generation interval is a step pixel, a plurality of template images 302-1 to 302- # are formed in a manner as shown in FIG. Generated. The number of blocks of the generated template # is an integer value obtained by dividing the horizontal size of the template generation image 300 (width size of the omnidirectional format = 3600 pixels in the embodiment to be described) by the generation interval (step). Become.

  Corresponding portions 314 on the search image 310 are searched within the predetermined search range 312 for the plurality of template images 302-1 to 302- # generated here. Since both ends (0 degrees and 360 degrees) of the θ coordinate of the omnidirectional image format are connected, when generating a template image or template matching, the right end is the left end and the left end is the right end. It can be handled.

  Referring to FIG. 12, in step S302, a template image is generated based on the current coordinates (sx, sy). In the first loop, a block size area (W pixels × H pixels) is cut out from the generation start coordinates (sx, sy) that are initially set in step S301 to generate a template image. After the template image is generated, the coordinates are updated to (sx + step, sy), and the second and subsequent template generations are generated by designating a block size area from the updated coordinates in the same manner. When the template image is generated, a template number (simply referred to as a number hereinafter) as schematically shown in FIG. 13 is assigned to the generated template image. Although not particularly limited, in the embodiment to be described, it is assumed that the template image is generated for one line and one turn in the θ direction in the overlapping region.

  In step S303, the imaging system 10 calculates a feature amount from the template image generated in step S302. In step S304, the imaging system 10 determines whether or not processing of all blocks to be generated has been completed. If it is determined in step S304 that the process has not been completed (NO), the process loops to step S302 to proceed to generation of the next template image and calculation of the feature amount. On the other hand, if it is determined in step S304 that all blocks have been processed (YES), the process branches to step S305.

  In step S <b> 305, the imaging system 10 ranks the template images of all blocks from the largest feature amount to the smallest one. Thereby, the processing order after step S306 is defined. FIG. 14A is a diagram for explaining ranking of template images according to the present embodiment. As shown in FIG. 14A, the template order is assigned to the template images of each number in descending order of the feature amount. The processes in steps S306 to S311 shown in FIG. 12 are selected and processed in the order determined in step S305 in order from the template image with the highest rank.

  Referring to FIG. 12, in step S <b> 306, the imaging system 10 calculates a temporary position for the template image of interest that is focused as a processing target. FIG. 14B is a diagram illustrating a first method for calculating the temporary position of the template image based on the ranking performed by the temporary position calculation unit according to the present embodiment. In the first calculation method, first, a connection position set for a peripheral template image positioned around the template image of interest is acquired.

  In the embodiment to be described, the template images are generated in a line in the θ direction in the overlapping region, and the template numbers are assigned in order from the left side, so that the adjacent relationship is identified by the number. In the example shown in FIG. 14B, the connection positions of the template images (numbers 2 and 4 in the figure) before and after the target template image (number 3 in the figure) are acquired. Each template image is given a template order in step S305 in FIG. 12, and the process of detecting the connection position from step S306 to step S311 in FIG. 12 is performed in descending order. For this reason, the connection position has already been determined for a template image having a larger feature amount than the template image of interest and a higher rank.

  Accordingly, in the example of FIG. 14B, the template image of interest (number 3) is ranked third, so that the template images (number 4 and number 1) that are the first and second templates are connected. The position has already been determined. On the other hand, for the 4th to 6th template images (numbers 2, 5, and 6) that are highlighted in FIG. 14B, the connection position is not yet determined, and the initial connection position ( 0,0) is set. Note that the connection position of the template image of interest (number 3) is also the initial connection position (0, 0) at the present time. The joint position represents the amount of deviation between the coordinates of the template image in the overlap area of the corrected image 1 and the coordinates of the corresponding area in the overlap area of the corrected image 0. The joint position (0, 0) is corrected by the template image. It means that the coordinate positions on the image 1 are connected as they are.

  Therefore, in the example of FIG. 14B, as the connection position of the peripheral template images, the determined connection of the template image (number 4 in the drawing) on the right side of the template image of interest (number 3 in the drawing) is determined. The position and the initial connection position of the left template image (number 2 in the figure) are acquired.

  Then, the acquired connection positions of the peripheral template images are averaged, and the average value can be used as the temporary position of the template of interest. At this time, a simple arithmetic average may be calculated, but preferably, the weighted average is performed by changing the weight between the determined connection position and the undecided connection position (initial connection position). Can do. Specifically, the weight can be increased so that the influence of the determined connection position becomes stronger.

The temporary position (tx _i , ty _i ) of the target template image (number i) can be calculated by the following calculation formula (3). In the following equation (3), the connection position of the left template image (number i-1) is (x _i−1 , y _i−1 ), the weight is w ₋ , and the right template image (i + 1) is connected. The position is (x _{i + 1} , y _{i + 1} ) and its weight is w ₊ . w ₋ and w ₊ are weight values according to whether or not the connection positions of the corresponding adjacent template images have been determined.

In the example of FIG. 14B, for example, the determined weight coefficient w _H is “0.7”, the undecided weight coefficient w _L is “0.3”, and the connection position of the determined template number 4 is (2, -2) When the connection position of the undecided template number 2 is an initial value (0, 0), the temporary position of the template image of interest (number 3) is (1.4, -1.4).

  FIG. 14C is a diagram illustrating a second method for calculating the temporary position of the template image based on the ranking performed by the temporary position calculation unit according to the present embodiment. In the second calculation method shown in FIG. 14C, first, as the peripheral template image of the template image of interest, the distance from the template image of interest and the set connection position are obtained for all template images. The For those for which the connection position has not been determined, the initial connection position (0, 0) is acquired in the same manner as described above. The distance between the template images may simply be the number of blocks. In FIG. 14C, for example, the template of interest is template number 3, distance 1 is assigned to the template images of number 2 and number 4, and distance 2 is assigned to those of number 1 and number 5. A distance of 3 is given for 6 things.

  Then, the connected positions of the acquired peripheral template images are weighted by the distance and averaged, and the average value can be used as the temporary position of the template image of interest. For the weighting based on the distance, a function that increases as the distance is closer can be used. For example, the reciprocal of distance (1 / distance) can be used as the weight value.

If the temporary position of the template image of interest (number i) is (tx _i , ty _i ), the connection position of each template image (number j = 1 to N) is (x _j , y _j ), and the distance is D _ij. The temporary position (tx _i , ty _i ) can be calculated by the following calculation formula (4). According to the second calculation method shown in FIG. 14C, since the connection positions of distant template images other than those adjacent are considered, continuity with the surroundings can be maintained better. Note that the distances and connection positions of all template images are acquired as the peripheral template images of the template image of interest, but a predetermined censoring may be performed at the distance.

  In the above description, the weight value according to whether or not the connection position of adjacent template images has been determined or the weight value according to the distance between the template images has been described. However, the calculation method described above is an example and is not particularly limited. In another embodiment, weighting may be performed in accordance with whether or not the connection positions of adjacent template images have been determined, and in accordance with the distance between the template images.

  Referring to FIG. 12 again, in step S307, the imaging system 10 sets a search range of a search area for performing template matching. FIG. 15 is a diagram illustrating a search range setting method performed by the search range setting unit according to the present embodiment. In template matching, a search area of a predetermined size is extracted from the search image for the template image of interest. Since the position detection image 0 that is a search image and the position detection image 1 that is a template image are generated by a predefined position detection conversion table, they are superimposed with a certain degree of accuracy. For this reason, since it is predicted that the part corresponding to the template image is found in the vicinity of the coordinates on the corresponding corrected image 0, the search area is cut out based on the coordinates. The search range determines a range for performing template matching in the search region, and matching with the image in the search region is performed while shifting the position of the template image vertically and horizontally within the search range. For this reason, the processing time becomes longer as the search range becomes wider.

  FIG. 15A illustrates a method for setting the normal search range 360. In the normal setting method, the search region 330 centered on the initial position (◯: coincides with the coordinates of the template image) is searched as it is. The range is 360A. In this case, the search area 330 and the search range 360A are the same, and it is necessary to perform matching calculation in the entire search area.

  In contrast, FIG. 15B illustrates a method of setting a search range based on the calculated temporary position. Usually, the determined temporary position (indicated by ● in the figure) 334 is moved from the initial position (indicated by ○ in the figure) 332 under the influence of a characteristic template, so that the temporary position (●) The search area is moved around 350 (350), and the area overlapping with the original search area 330 is set as the search area 360B, so that the search area can be narrowed and the processing time can be shortened.

  FIG. 15C further illustrates a method of setting a search range based on a connection position that has been determined for adjacent template images. Similarly to the method shown in FIG. 15B, the search area is moved around the determined temporary position (●) (350), and an area overlapping the original search area 330 is obtained. Subsequently, the determined search range 360C is determined by referring to the determined connection positions for the adjacent template images. For example, when the connection positions of the adjacent template images on both sides have been determined, the connection positions of the template of interest are considered to be generally limited between them. Therefore, as shown in FIG. 15C, the processing time can be further shortened by narrowing the search range 360B by limiting the vertical direction by using the vertical connection positions 336U and 336L of the template images on both sides. Can do.

  Referring to FIG. 12 again, in step S308, the imaging system 10 calculates matching while shifting the template image in the vertical direction and the horizontal direction on the search region within the search range set in step S307, and the similarity Find the score.

  FIG. 16A is a flowchart showing details of matching calculation processing executed by the imaging system 10 according to the present embodiment. The process shown in FIG. 16A is called in step S308 shown in FIG. 12, and is started from step S400. In step S401, the imaging system 10 determines whether the feature amount of the template image of interest is greater than or equal to a predetermined threshold value. In the embodiment to be described, a predetermined threshold value is set, and the feature amount is classified according to the magnitude of the threshold value. If it is determined in step S401 that the feature amount of the template image of interest is greater than or equal to a predetermined threshold (YES), the process branches to step S402.

  In step S402, the imaging system 10 calculates a matching score of an evaluation method suitable for an image having a large feature amount. In step S404, the imaging system 10 ends the process and returns to the process illustrated in FIG. on the other hand. If it is determined in step S401 that the feature amount of the template image of interest is not equal to or greater than the predetermined threshold (NO), the process branches to step S403. In step S403, the imaging system 10 calculates a matching score based on an evaluation method suitable for an image having a small feature amount. In step S404, the imaging system 10 ends the process and returns to the process illustrated in FIG.

  Hereinafter, as an example of template matching, a zero-mean normalized cross-correlation (Zero) method and a zero-mean sum of squared difference (ZSSD) method will be described with reference to FIG. FIG. 17 is a diagram for explaining processing for searching for a template image in a search range by (A) a template image and (B) pattern matching. The total number of pixels of the template is N (= W pixels × H pixels), and the search position on the search range is represented by (kx, ky). The brightness value of the template image at the coordinates (i, j) when the search position (kx, ky) is the reference coordinates (0,0) is T (i, j), and the brightness value on the search image is S ( kx + i, ky + j), the matching score ZNCC (kx, ky) by the ZNCC method can be obtained by the following equation (5). In FIG. 17, the search position (kx, ky) is set at the upper left of the template image, but the method of taking the coordinates is not particularly limited, and may be set at the center of the template image.

  The matching score ZNCC (kx, ky) represents the degree of similarity. When it is 1, it is a perfect match, and when it is -1, it is a negative / positive inversion. That is, a higher matching score ZNCC (kx, ky) indicates a higher degree of similarity with the template image. As shown in FIG. 17B, template matching is performed while shifting the position of the template image vertically and horizontally within the search range, and a matching score ZNCC (kx, ky) at each search position is calculated.

  On the other hand, the dissimilarity ZSSD (kx, ky) by the ZSSD method can be obtained by the following equation (6). In the ZSSD method, since the sum of squares of the difference in luminance value of pixels at the same position is obtained, the ZSSD value is a so-called dissimilarity, which means that a larger value is not similar. Therefore, in order to obtain a score based on the degree of similarity, a negative sign is added and −ZSSD (kx, ky) is set as a matching score.

  The matching score -ZSSD (kx, ky) has a range of possible values from − (255 × 255) to 0 when each pixel value is an 8-bit value. A higher matching score-ZSSD (kx, ky) indicates a higher degree of similarity with the template image.

  The ZNCC method described above is suitable in that it can absorb fluctuations in the gain of the image, can absorb fluctuations in the average brightness of the image, and has high robustness in matching similarity. Is. The ZNCC method can be suitably applied particularly when the luminance of the image is sufficiently spread and distributed and the feature amount is large. On the other hand, the ZSSD method is superior to SSD in that it can absorb fluctuations in the average brightness of an image, and the calculation cost is lower than that of SSD (Sum of Squared Differences). However, the calculation is simpler than ZNCC. The ZSSD method can be suitably applied particularly to an image having a small feature amount and blurred. Instead of ZSSD, ZSAD (Zero-mean Sum of Absolute Differences) may be used.

  Referring to FIG. 12 again, in step S309, the imaging system 10 offsets the matching score ZNCC (kx, ky) or -ZSSD (kx, ky) based on the similarity at each position calculated in step S308. Make corrections and calculate the corrected final score.

  FIG. 16B is a flowchart showing details of the score correction processing executed by the imaging system 10 according to the present embodiment. The process shown in FIG. 16B is called in step S309 shown in FIG. 12, and starts from step S500. In step S501, the imaging system 10 determines whether the feature amount of the template image of interest is greater than or equal to a predetermined threshold value. If it is determined in step S501 that the feature amount of the template image of interest is greater than or equal to a predetermined threshold (YES), the process branches to step S502. In this case, since the ZNCC method is used to calculate the score, in step S502, the imaging system 10 calculates an offset using the offset function for the ZNCC method, and in step S504, the process is terminated. Returning to the processing shown in FIG.

  On the other hand, if it is determined in step S501 that the feature amount of the template image of interest is not equal to or greater than the predetermined threshold (NO), the process branches to step S503. In this case, since the ZSSD method is used to calculate the score, in step S503, the imaging system 10 calculates an offset using an offset function for the ZSSD method, and in step S504, the process is terminated. Returning to the processing shown in FIG.

  In step S310, the imaging system 10 determines a connection position for the template image of interest based on the corrected score at each position calculated in step S309.

  FIG. 18A shows a graph in which the offset function for the matching score is plotted with respect to the search position (kx / ky). In the process of step S309, the matching score ZNCC (kx, ky) or -ZSSD (kx, ky) is determined according to the distance from the temporary position by the offset function (kx, ky) as shown in FIG. Add the offset value. The offset function shown in FIG. 18A is a function in which the offset value becomes maximum at the temporary position and monotonously decreases as the distance from the temporary position increases. As described above, the range of possible values of the offset function varies depending on the matching score evaluation method, and thus the offset function is determined to have a size corresponding to the range. For example, in ZNCC, the value ranges from −1 to 1, and in −ZSSD, from − (255 × 255) to 0, the size is set accordingly. FIGS. 18B and 18C show a graph obtained by plotting the score before correction and the score after correction at the search position (kx / ky), the matching position based on the score before correction, and the matching position based on the score after correction. It is a figure shown with. FIG. 18B illustrates the case of a template image having a large feature amount, and FIG. 18C illustrates the case of a template image having a small feature amount.

  As shown in FIG. 18B, the final matching score is raised around the temporary position by the offset correction. However, when the feature amount of the template image is large, the similarity due to the change in position greatly fluctuates, so that the influence of the matching score based on the similarity remains dark. For this reason, regardless of whether offset correction is performed or not, the position where the matching score based on the similarity is maximized is determined as the matching position. In other words, if the feature quantity is large enough that the peak of the similarity matching score remains clearly, the similarity will predominately determine the connection position.

  On the other hand, as shown in FIG. 18 (C), when the feature amount of the template image is small, a clear peak is not recognized in the normal matching score, and from the temporary position rather than the matching score based on the similarity. The effect of offset according to distance is reflected deeply. For this reason, the vicinity of the temporary position where the offset function is approximately maximum is determined as the matching position. That is, when the feature amount is small, the temporary position predominates and determines the connection position.

When the matching position is determined, a connection position is obtained as a shift amount (Δθ _i , Δφ _i ) from the position superimposed on the coordinate value of the position detection correction image 0 as it is with respect to the template image of interest (number i). Desired.

  By performing the offset correction described above, there is a clear peak in the score based on the similarity, and when the similarity is large, the position becomes the connection position. When there is no clear peak, such as when there is no feature, the tentative position is the connection position, continuity with adjacent blocks is maintained, and the ratio at which the connection position is determined at a position far from the temporary position is reduced. can do.

  Referring to FIG. 12 again, in step S311, the imaging system 10 determines whether the processing for all the template image blocks has been completed. If it is determined in step S311 that the process has not been completed (NO), the process returns to step S306 and is repeated until the processes for all the blocks are completed. On the other hand, if it is determined in step S311 that all the processes have been completed (YES), the process proceeds to step S312.

Through the processing from step S306 to step S311, the connection positions (Δθ _i , Δφ _i ) corresponding to all of the generated template images (number i: i = 1 to #) are obtained. In step S312, the imaging system 10 determines the connection position (Δθ, Δφ) for each pixel (θ, φ) in the omnidirectional format based on the connection position data set corresponding to each obtained template image. The detection result data 222 is generated by calculation. In step S313, the imaging system 10 ends the present joint position detection process and returns to the process illustrated in FIG.

FIG. 19 is a diagram illustrating a data structure of detection result data generated by the connection position detection unit 204 in the present embodiment. FIG. 20 is a diagram illustrating processing for generating detection result data by the position detection unit 204 of the present embodiment. As a result of the processing in step S312, the information in which the shift amounts (Δθ, Δφ) are associated with the coordinate values (θ, φ) after conversion as shown in FIG. 19 is held for all coordinate values. Detection result data 222 is obtained. At this time, the shift amounts (Δθ, Δφ) corresponding to the respective coordinate values (θ, φ) are the shift amounts (Δθ _i , Δφ _i ) for each template block (i) obtained by the connection position detection process. It can be set as the value of the center coordinate of each template block and calculated by interpolation.

Specifically, first, as shown in FIG. 20A, the horizontal coordinate value θ is equal to the value of the center coordinate of the template block, and the upper end (φ = 0) and the lower end (height is 1800). In the case of a pixel, φ = 1799.) The shift amount (Δθ, Δφ) at each coordinate located at (0) is set as (0, 0). As for other coordinates for which the shift amount is not set, as shown in FIG. 20B, a lattice composed of four neighboring points (indicated by A to D in the figure) is considered, and two-dimensional linear interpolation is included therein. The shift amount is calculated by calculation. If the point Q is a point that internally divides dθ: 1-dθ in the θ-axis direction and dφ: 1-dφ in the φ-axis direction in the four-point lattice, the shift amount (Δθ _Q , Δφ _Q ) at the point Q is near Using the shift amounts ((Δθ _A, Δφ _A ),..., (Δθ _D, Δφ _D )) at the four points, it can be calculated by the following equation (7).

  In the embodiment to be described, the shift amount (0, 0) is set with respect to the upper end (φ = 0) and the lower end (φ = 1799 in the above example) in the omnidirectional format. However, the calculation method of the shift amount (Δθ, Δφ) corresponding to each coordinate value (θ, φ) is not particularly limited. Here, the partial images 0 and 1 only need to be connected without contradiction, and in another embodiment, the above-described two-dimensional linear interpolation calculation can be performed by setting the coordinates to be shifted inward (0, 0). it can. In this case, the shift amount (0, 0) may be set for all the coordinates whose φ direction is outside the coordinates.

  Referring to FIG. 5 again, in step S203, the imaging system 10 causes the table correction unit 206 to use the detection result data 222 so that the position detection conversion table 220 is aligned on the spherical coordinates. Correct it. As shown in FIG. 19, the shift amount is obtained for each coordinate value of the omnidirectional image format by the joint position detection process in step S202. Specifically, in step S203, the distortion amount of partial image 0 is used. In the detected distortion correction table 0, the input coordinate value (θ, φ) is corrected so as to correspond to (x, y) that was associated with (θ + Δθ, φ + Δφ) before the correction. Note that it is not necessary to change the association of the detection distortion correction table 1 used for distortion correction of the partial image 1.

  In step S <b> 204, the imaging system 10 generates the image composition conversion table 224 by performing rotational coordinate conversion from the corrected position detection conversion table 220 using the table generation unit 208.

FIG. 21 is a flowchart illustrating the image synthesis conversion table generation processing executed by the imaging system 10 according to the present embodiment. FIG. 22 is a diagram for describing mapping of two partial images captured by two fisheye lenses to a spherical coordinate system in the image composition process. The process shown in FIG. 21 is called in step S204 shown in FIG. 5 and starts from step S600. In the loop of step S601 to step S606, the table generation unit 208 performs steps S602 to S605 for each coordinate value (θ _g , φ _g ) of the spherical coordinate system for image synthesis that is an input value of the distortion correction table for image synthesis. Execute the process. The range of coordinate values to be set is a range defined by the entire range of horizontal angles (0 to 360 degrees) and the entire range of vertical angles (0 to 180 degrees). In order to perform conversion processing for all coordinate values that become input values, the coordinate values are set in order here.

In step S602, the table generation unit 208 obtains coordinate values (θ _d , φ _d ) of the joint coordinate detection spherical coordinate system corresponding to the coordinate values (θ _g, φ _g ) by rotational coordinate conversion. By rotating coordinate transformation, the definition of the coordinate axes of the horizontal angle θ _d and the vertical angle φ _d with the optical axis of one lens optical system as an axis as shown in FIG. 9 makes it perpendicular to the optical axis as shown in FIG. Are converted into definitions of a horizontal angle θ _g and a vertical angle φ _g with respect to a simple axis. Coordinates (θ _d , φ _d ) corresponding to the above coordinates (θ _g, φ _g ) are based on rotational coordinate transformation and have a radius of 1 and coordinates (θ _g, φ _g ) in a spherical coordinate system for image synthesis. The three-dimensional orthogonal coordinates (x _d , y _d , z _d ) corresponding to the coordinates (θ _d, φ _d ) of the position detection spherical coordinate system and the three-dimensional orthogonal coordinates (x _g , y _g , z _g ) corresponding to ) Can be calculated by the following equation (8). In the following formula (8), the coefficient β is a rotation angle that defines the rotation coordinate transformation around the x axis in the three-dimensional orthogonal coordinates, and is 90 degrees in the embodiment to be described.

  In the position detection conversion table 220, the optical axis is projected onto the spherical pole and the overlapping portion between the images is projected near the equator of the spherical surface, and the vertical direction of the omnidirectional image format matches the zenith direction of the captured scene. Not. In contrast, in the image composition conversion table 224, the optical axis is projected onto the spherical equator, and the vertical direction of the omnidirectional image format matches the zenith direction of the captured scene. Will come to do.

In the loop from step S603 to step S605, the table generating unit 208 executes the process of step S604 for each of image 0 and image 1. In step S604, the table generation unit 208 converts the coordinate values (x, y) of the partial image (0, 1) associated with (θ _d , φ _d ) for each image (0, 1) after correction. It is obtained by referring to the connection position detection conversion table. The conversion tables 220 and 224 hold the corresponding coordinate values (x, y) in increments of 1 pixel for both θ _d and φ _d , but the coordinate values (θ _d , φ _d ) calculated by the conversion are stored. Is typically obtained with a value after the decimal point. Conveniently, the coordinate values (x, y) of the associated partial image (0, 1) are associated with the coordinate values existing in the nearest table of the calculated coordinate values (θ _d , φ _d ). The coordinate value (x, y) obtained can be adopted. In a preferred embodiment, the coordinates (θ _d ) calculated with reference to a plurality of coordinate values (x, y) associated with the nearest coordinate value and the surrounding coordinate values among the coordinate values existing in the table. , Φ _d ), the coordinate value (x, y) of the associated partial image (0, 1) can also be calculated by performing weighted interpolation according to the distance from it.

  In the loop of step S603 to step S605, calculation is performed for all the images (0, 1), and in the loop of step S602 to step S606, when calculation for all coordinate values that are input values is completed, in step S607, This process is terminated. As a result, all data of the image composition conversion table 224 is generated.

  Referring to FIG. 5 again, in step S205, the imaging system 10 performs distortion correction on the original partial image 0 and partial image 1 using the image synthesis conversion table 224 by the image synthesis distortion correction unit 210. Then, a corrected image for image synthesis 0 and a corrected image for image synthesis 1 are obtained. Thereby, as a result of the processing by the image composition distortion correction unit 210, the two partial images 0 and 1 captured by the fisheye lens are developed on the omnidirectional image format as shown in FIG. The partial image 0 captured by the fisheye lens 0 is typically mapped to approximately the left hemisphere of the entire celestial sphere, and the partial image 1 captured by the fisheye lens 1 is approximately aligned to the approximately right hemisphere of the entire celestial sphere. To be mapped.

  9 and 22, it is clear that partial image 0 and partial image 1 are mapped at different positions with respect to the omnidirectional format, and the zenith direction of the scene is the vertical direction of the image. It matches. The central part of the partial images 0 and 1 is mapped on the equator with less distortion, and the overlapping area between the corrected image 0 and the corrected image 1 is different from that shown in FIG. , As well as in the vicinity of horizontal angles of 0 degrees and 180 degrees.

  In step S <b> 206, the image composition unit 212 synthesizes the image composition correction image 0 and the image composition correction image 1. In the combining process, blending processing or the like is performed for overlapping areas that overlap between images, and the existing pixel values are used for areas where only one of the pixel values exists. Through the above synthesis process, one omnidirectional image is generated from the two partial images captured by the fisheye lens.

  As described above, in the present embodiment, a feature amount that indicates the degree of the feature of the template image to be matched is measured, and the connection position of the template image is determined using the measured feature amount. doing. More specifically, the evaluation method for obtaining the matching score is switched according to the feature of the region. As a result, since the connection position can be determined based on an appropriate evaluation method for a region having a strong feature and a region having a weak feature, it is possible to reduce errors in detecting the connection position and improve the alignment accuracy. As a result, it is possible to detect the connecting position with high accuracy even in an area where the image is flat or an area where the same pattern is repeated and has few features, and the quality of the obtained omnidirectional image is high. Be improved. Furthermore, the algorithm is simplified because no processing is performed that excludes an inappropriate matching region with few features.

  In general, the ZNCC method and the ZSSD method are more accurate than the ZNCC method. However, in the above-described embodiment, the ZNCC method is basically used, and control is performed so that the ZSSD method is used for an image in a weak region, particularly a flat image. The advantage of using the ZSSD method instead of the ZNCC method for images with weak features is explained as follows.

  For example, consider performing matching evaluation on a flat image such as a blue sky or a pure white wall. Such a flat image typically includes noise during shooting. Compared with the ZNCC method that gives a kind of so-called correlation value when performing matching evaluation between flat images with noise, the ZSSD method that gives an error value as a sum total of errors for each pixel Since the error does not become so large, a good value is given as a matching evaluation value. In the above-described embodiment, since it is desired to take a good value as a matching evaluation value for a flat image, a ZSSD method that gives an error value is used for a template image with a small feature amount, instead of the ZNCC method that gives a correlation value. I am going to use it.

  In the omnidirectional imaging system 10, since the omnidirectional range is the imaging range, many scenes are generated in which the overlapping region includes both a flat and weak feature such as the sky and a strong feature such as the subject. . The joining process of a plurality of images according to the present embodiment can be suitably applied to the joining of omnidirectional images having such properties.

[Other Embodiments]
In the above-described embodiment, as an example of the image processing apparatus and the imaging system, the imaging system 10 that captures a still image of a celestial sphere with its imaging optical system and synthesizes it with an internal distortion correction / image synthesis block is used. explained. However, the configurations of the image processing apparatus and the imaging system are not particularly limited. In another embodiment, it may be configured as an omnidirectional moving image capturing system that captures a moving image of the omnidirectional sphere, and receives a plurality of partial images (still images or moving images) captured by a plurality of imaging optical systems. A camera processor that generates omnidirectional images (still images or moving images), and omnidirectional images that receive multiple partial images (still images or moving images) captured by an omnidirectional imaging device that is exclusively responsible for shooting An information processing device such as a personal computer, a workstation, or a virtual machine on a physical computer system, a portable information terminal such as a smartphone or a tablet, etc. that synthesizes (still images or moving images) can also be configured as the image processing device. . Moreover, you may comprise as an imaging system containing image processing apparatuses, such as a camera processor, information processing apparatus, and a portable information end as mentioned above, and the imaging optical system isolate | separated from this image processing apparatus.

  Hereinafter, with reference to FIG. 23 and FIG. 24, an omnidirectional imaging device and an external that generates a synthesized omnidirectional image by receiving input of a plurality of partial images photographed by the omnidirectional imaging device. An omnidirectional imaging system according to another embodiment including the computer apparatus will be described. FIG. 23 shows an outline of an omnidirectional imaging system 400 according to another embodiment.

  The omnidirectional imaging system 400 according to the embodiment shown in FIG. 23 includes an omnidirectional imaging apparatus 410 that is exclusively responsible for imaging, and a computer apparatus 430 that is connected to the omnidirectional imaging apparatus 410 and is exclusively responsible for image processing. Consists of. Note that FIG. 23 shows only the main configuration, and omits the detailed configuration. In addition, the same reference numerals are given to components that function in the same manner as the embodiment described with reference to FIGS. In addition, the omnidirectional imaging system 400 according to the embodiment shown in FIGS. 23 and 24 refers to FIGS. 1 to 22 except that image processing for synthesizing the omnidirectional image is performed exclusively by the computer device 430. Therefore, the following description will focus on the differences.

  In the embodiment shown in FIG. 23, the omnidirectional imaging device 410 includes a digital still camera processor 100, a lens barrel unit 102, and a three-axis acceleration sensor 120 connected to the processor 100. The lens barrel unit 102 has the same configuration as that shown in FIG. 2, and the processor 100 also has the same configuration as that shown in FIG.

  The processor 100 includes an ISP 108, a USB block 146, and a serial block 158, and controls USB communication with a computer device 430 connected via a USB connector 148. A wireless NIC 160 is connected to the serial block 158 to control wireless communication with the computer device 430 connected via the network.

  A computer apparatus 430 shown in FIG. 23 can be configured as a general-purpose computer such as a desktop personal computer or a workstation. The computer device 430 includes hardware components such as a processor, a memory, a ROM, and a storage. In the embodiment shown in FIG. 23, the computer device 430 includes a USB interface 432 and a wireless NIC 434, and is connected to the omnidirectional imaging device 410 via a USB bus or a network.

  The computer apparatus 430 further includes a position detection distortion correction unit 202, a connection position detection unit 204, a table correction unit 206, a table generation unit 208, and an image synthesis distortion correction unit 210 as processing blocks for image synthesis. And an image composition unit 212. In this embodiment, two partial images captured by a plurality of imaging optical systems of the lens barrel unit 102 and the position detection conversion table of the omnidirectional imaging device 410 are externally connected via a USB bus or a network. Transferred to device 430.

  In the computer apparatus 430, the position detection distortion correction unit 202 performs distortion correction on the partial images 0 and 1 transferred from the omnidirectional imaging apparatus 410 using the position detection conversion table transferred together, The position detection correction images 0 and 1 are generated. The joint position detection unit 204 detects the joint position between the converted corrected images 0 and 1 and generates detection result data. The table correction unit 206 corrects the transferred position detection conversion table based on the detection result data. The table generation unit 208 performs rotational coordinate conversion from the corrected conversion data, and generates an image composition conversion table.

  The image composition distortion correction unit 210 performs distortion correction on the original partial image 0 and the partial image 1 using the image composition conversion table as the preceding process of the image composition processing, and corrects the image composition corrected images 0 and 0. 1 is generated. The image composition unit 212 synthesizes the obtained image composition correction images 0 and 1 to generate a composite image in the omnidirectional image format.

  The functional block shown in FIG. 23 can further include a display image generation unit 214. The display image generation unit 214 executes image processing for projecting the omnidirectional image onto the flat display device. The computer apparatus 430 according to the present embodiment reads out a program from a ROM, an HDD, or the like, and develops it in a work space provided by the RAM, thereby realizing each of the above-described functional units and processes described below under the control of the CPU. .

  FIG. 24 is a flowchart showing an overall flow of the omnidirectional image synthesis process in the omnidirectional imaging system 400 according to another embodiment. FIG. 24 illustrates a flow from when a captured image is input by the omnidirectional imaging device 410 to when the image is stored by the computer device 430.

  The processing shown in FIG. 24 is started from step S700 in response to, for example, instructing photographing with the two imaging optical systems by pressing the shutter button in the omnidirectional imaging device 410. First, processing in the omnidirectional imaging device 410 is executed.

  In step S701, the omnidirectional imaging device 410 captures the partial image 0 and the partial image 1 with the two solid-state imaging devices 22A and 22B. In step S702, the omnidirectional imaging device 410 transfers the partial image 0 and the partial image 0 to the computer device 430 via the USB bus or the network. In addition, the position detection conversion table is transferred to the computer device 430 via the USB bus or the network. At this time, when tilt correction is also performed on the computer device 430 side, the tilt information acquired by the triaxial acceleration sensor 120 is transferred to the computer device 430.

  Note that the above-described position detection conversion table of the omnidirectional imaging device 410 may be transferred when the omnidirectional imaging device 410 and the computer device 430 recognize each other. That is, it is sufficient that the position detection conversion table is once transferred to the computer apparatus 430, and it is not necessary to transfer the conversion table every time. The position detection conversion table is stored, for example, in an SDRAM (not shown) or the like, and is read and transferred from here. The processing up to this point is processing in the omnidirectional imaging device 410, and processing from the next step S703 is processing executed in the transfer destination computer device 430.

  In step S703, the computer apparatus 430 uses the position detection distortion correction unit 202 to perform distortion correction on the transferred partial image 0 and partial image 1 using the transferred position detection conversion table, and performs position detection. Corrected images 0 and 1 are obtained. At this time, when the tilt correction is performed on the transfer destination computer device 430 side, the position detection conversion table may be corrected in advance in accordance with the vertical direction based on the transferred tilt information. In step S <b> 704, the computer apparatus 430 performs joint position detection between images in the overlapping region of the position detection corrected images 0 and 1 by the joint position detection unit 204, and obtains detection result data. In step S <b> 705, the computer apparatus 430 uses the table correction unit 206 to correct the transferred position detection conversion table using the detection result data so that the image is aligned on the spherical coordinates. In step S <b> 706, the computer apparatus 430 generates an image composition conversion table by performing rotational coordinate conversion from the corrected position detection conversion table using the table generation unit 208.

  In step S <b> 707, the computer apparatus 430 performs distortion correction on the original partial images 0 and 1 using the image synthesis conversion table 210 by the image synthesis distortion correction unit 210, and obtains the image synthesis corrected images 0 and 1. obtain. In step S <b> 708, the computer apparatus 430 synthesizes the corrected images for image synthesis 0 and 1 by the image synthesis unit 212. Through the above synthesis process, one omnidirectional image is generated from the two partial images captured by the fisheye lens. In step S709, the computer apparatus 430 stores the omnidirectional image generated in step S708 in the external storage, and the process ends in step S710.

  In addition, the operation | movement concerning the flowchart in other embodiment shown in FIG. 24 can also be made to perform the program on a computer. That is, the CPU that controls the operation of the omnidirectional imaging device 410 and the CPU that controls the operation of the computer device 430 read out the programs stored in the storage medium such as ROM and RAM, respectively, and expand them on the memory. The processing of each responsible portion of the above-described omnidirectional image synthesis processing is realized. FIG. 23 and FIG. 24 exemplify the omnidirectional imaging system configured separately, and are not limited to the specific embodiments shown in FIG. 23 and FIG. Each functional unit for realizing the omnidirectional image capturing system can be distributedly mounted in various modes on one or more imaging devices and one or more computer systems.

  According to the embodiment described above, when a connection position is detected between a plurality of input images, an appropriate connection position corresponding to the feature amount of the partial area is detected for each partial area of the input image. It is possible to provide an image processing apparatus, an image processing method, a program, and an imaging system capable of performing the above.

  In the above-described embodiment, each of the plurality of partial images is captured substantially simultaneously using different lens optical systems. However, in the other embodiments, the same lens optical system is used and the times are different. Thus, the present invention may be applied to superposition of a plurality of partial images that are photographed and photographed from different photographing directions (orientations) from a predetermined photographing point. Furthermore, in the above-described embodiment, two partial images captured by a lens optical system having an angle of view greater than 180 degrees are superimposed and synthesized, but in other embodiments, one or more lens optics are combined. You may apply to the superimposition synthesis | combination of the 3 or more partial image imaged by the system. In the above-described embodiment, the imaging system using the fisheye lens has been described as an example. However, the imaging system may be applied to an omnidirectional imaging system using an ultra-wide angle lens. Further, in the above-described preferred embodiment, the superposition of the omnidirectional images has been described. Yes.

  The functional unit can be realized by a computer-executable program written in a legacy programming language such as an assembler, C, C ++, C #, Java (registered trademark), an object-oriented programming language, or the like. ROM, EEPROM, EPROM , Stored in a device-readable recording medium such as a flash memory, a flexible disk, a CD-ROM, a CD-RW, a DVD-ROM, a DVD-RAM, a DVD-RW, a Blu-ray disc, an SD card, an MO, or through an electric communication line Can be distributed. In addition, a part or all of the functional unit can be mounted on a programmable device (PD) such as a field programmable gate array (FPGA) or mounted as an ASIC (application-specific integration). Circuit configuration data (bit stream data) downloaded to the PD in order to realize the above functional unit on the PD, HDL (Hardware Description Language) for generating the circuit configuration data, VHDL (Very High Speed Integrated Circuits) Hardware Description Language), Verilog-HDL, and the like can be distributed on a recording medium as data.

  Although the embodiments of the present invention have been described so far, the embodiments of the present invention are not limited to the above-described embodiments, and those skilled in the art may conceive other embodiments, additions, modifications, deletions, and the like. It can be changed within the range that can be done, and any embodiment is included in the scope of the present invention as long as the effects of the present invention are exhibited.

(Appendix)
(Appendix 1)
An image processing apparatus for detecting a connection position between a plurality of input images,
Target image generation means for generating a plurality of target images to be searched for in the second input image, respectively, from the first input image;
Feature quantity calculating means for calculating a feature quantity for each of a plurality of target images generated by the target image generating means;
Using the evaluation method according to the feature amount calculated by the feature amount calculation unit for the target image of interest of the plurality of target images whose feature amounts have been calculated by the feature amount calculation unit, An image processing apparatus comprising: matching calculation means for calculating matching with the portion of the second input image.
(Appendix 2)
The image according to claim 1, further comprising: an evaluation value correction unit that corrects the evaluation value at each position calculated by the matching calculation unit using a correction method according to the evaluation method used by the matching calculation unit. Processing equipment.
(Appendix 3)
The evaluation value further includes temporary position calculation means for calculating a temporary position based on a relative relationship of feature amounts between the target image and each of the one or more peripheral target images with respect to the target image. The image processing apparatus according to appendix 2, wherein the correction unit corrects the evaluation value based on the similarity calculated by the matching calculation unit so that priority is given to the temporary position.
(Appendix 4)
The image processing apparatus according to any one of appendices 1 to 3, wherein the feature amount is at least one of an edge amount, a variance, and a standard deviation of the target image.
(Appendix 5)
As the evaluation method according to the feature amount, an evaluation method that gives a correlation value is used when the feature amount is larger than the reference, and an evaluation method that gives an error value when the feature amount is smaller than the reference. The image processing apparatus according to any one of appendices 1 to 4.
(Appendix 6)
A connection position determination for determining a connection position to the second input image according to a connection position of a peripheral target image having a larger feature amount than the target image based on the matching evaluation value for the target image. The image processing apparatus according to any one of appendices 1 to 5, including: means.
(Appendix 7)
Converting a plurality of captured images into a coordinate system different from the captured image using conversion data such as projecting an overlapping region where the imaging ranges between the captured images overlap based on the projection model to the vicinity of the spherical equator, Conversion means for generating a plurality of input images;
Correction means for correcting the conversion data based on a connection position detection result by the connection position determination means;
The data generation means for generating the image composition conversion data for defining the conversion for the plurality of captured images at the time of image composition by performing rotational coordinate conversion on the conversion data modified by the modification means, Image processing device.
(Appendix 8)
An image processing method for detecting a connection position between a plurality of input images, the computer comprising:
Generating, from the first input image, a plurality of target images to be searched for in the second input image, respectively;
Calculating a feature amount for each of a plurality of target images;
Calculating a matching between the target image of interest and a portion of the second input image using an evaluation method corresponding to the calculated feature amount for the target image of interest of the plurality of target images; An image processing method is executed.
(Appendix 9)
A program for realizing an image processing apparatus for detecting a connection position between a plurality of input images, the computer comprising:
Target image generation means for generating a plurality of target images to be searched for in the second input image from the first input image,
For each of the plurality of target images generated by the target image generation unit, a feature amount calculation unit that calculates a feature amount, and a target image of interest of the plurality of target images for which the feature amount has been calculated by the feature amount calculation unit On the other hand, in order to function as a matching calculation unit that calculates matching between the target image and the portion of the second input image using an evaluation method according to the feature amount calculated by the feature amount calculation unit. Program.
(Appendix 10)
An imaging system for detecting a connection position between a plurality of captured images,
Imaging means;
Means for acquiring a plurality of input images from a plurality of captured images respectively captured in a plurality of directions by the imaging means;
Target image generation means for generating a plurality of target images to be searched for in the second input image, respectively, from the first input image;
Feature quantity calculating means for calculating a feature quantity for each of a plurality of target images generated by the target image generating means;
Using the evaluation method according to the feature amount calculated by the feature amount calculation unit for the target image of interest of the plurality of target images whose feature amounts have been calculated by the feature amount calculation unit, An imaging system comprising: matching calculation means for calculating matching with the portion of the second input image.

DESCRIPTION OF SYMBOLS 10,300 ... Omnispherical imaging system, 12 ... Imaging body, 14 ... Housing, 18 ... Shutter button, 20 ... Imaging optical system, 22 ... Solid-state image sensor, 100 ... Processor, 102 ... Lens barrel unit, 108 ... ISP, 110, 122 ... DMAC, 112 ... Arbiter (ARBMEMC), 114 ... MEMC, 116, 138 ... SDRAM, 118 ... Distortion correction / image synthesis block, 120 ... Triaxial acceleration sensor, 124 ... Image processing block, 126 ... Image data transfer unit, 128... SDRAMC, 130... CPU, 132... Resize block, 134. 264 block, 140 ... memory card control block, 142 ... memory card slot, 144 ... flash ROM, 146 ... USB block, 148 ... USB connector, 150 ... peripheral block, 152 ... audio unit, 154 ... speaker, 156 ... microphone, 158 ... Serial block, 160 ... Wireless NIC, 162 ... LCD driver, 164 ... LCD monitor, 166 ... Power switch, 168 ... Bridge, 200 ... Functional block, 202 ... Position detection distortion correction unit, 204 ... Link position detection unit, 206 ... Table correction unit, 208 ... Table generation unit, 210 ... Image synthesis distortion correction unit, 212 ... Image synthesis unit, 214 ... Display image generation unit, 220 ... Conversion table for position detection, 220 ... Detection result data, 224 ... Image composition conversion table, 23 ... Template generation unit, 234 ... Template feature amount calculation unit, 236 ... Template ranking unit, 238 ... Temporary position calculation unit, 240 ... Search range setting unit, 242 ... Matching calculation unit, 244 ... Score correction unit, 246 ... Connection position Determination unit, 248 ... detection result generation unit, 250 ... edge extraction filter unit, 252 ... gain multiplication unit, 254 ... LPF unit, 256 ... addition unit, 300 ... template image, 302 ... template image, 310 ... search image, 312 ... Search range, 314 ... Corresponding part, 330 ... Search area, 332 ... Initial position, 334 ... Temporary position, 336 ... Adjacent connection position, 360 ... Search range, 410 ... Omni-sphere imaging device, 430 ... Computer device, 432 ... USB I / F, 434 ... Wireless NIC

JP 2001-148777 A

Claims (7)

An image processing apparatus for combining a plurality of input images,
Feature amount acquisition means for acquiring a feature amount of a predetermined target image in the first input image;
A connection position detecting means for detecting a connection position between the first input image and the second input image in a manner according to the feature amount;
An image processing apparatus comprising: a combining unit that combines the first input image and the second input image at the connection position.   The image processing apparatus according to claim 1, wherein matching is performed by a method according to the feature amount.   The image processing apparatus according to claim 1, wherein the feature amount is at least one of an edge amount, a variance, and a standard deviation of the target image.   The image processing apparatus according to claim 1, wherein the method according to the feature amount is a method in which a correlation value is given when the feature amount is larger than a reference, and an error value is given when the feature amount is smaller than the reference. A program for realizing an image processing apparatus for synthesizing a plurality of input images, comprising:
Feature amount acquisition means for acquiring a feature amount of a predetermined target image in the first input image;
A connection position detecting means for detecting a connection position between the first input image and the second input image by a method according to the feature amount, and the first input image and the second input image at the connection position. A program for functioning as a synthesis means for synthesis. An image processing method for combining a plurality of input images, wherein a computer
Obtaining a feature amount of a predetermined target image in the first input image;
Detecting a connection position between the first input image and the second input image in a method according to the feature amount;
An image processing method that executes the step of synthesizing the first input image and the second input image at the connection position. An imaging system for combining a plurality of input images,
Imaging means;
Means for acquiring a plurality of input images from a plurality of captured images respectively captured in a plurality of directions by the imaging means;
Feature amount acquisition means for acquiring a feature amount of a predetermined target image in the first input image;
A connection position detecting means for detecting a connection position between the first input image and the second input image in a manner according to the feature amount;
An imaging system comprising: a combining unit that combines the first input image and the second input image at the connection position.