JP6238677B2 - Image processing apparatus, imaging apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, an image processing method, and an image processing program, and more particularly, to an image processing device, an imaging device, an image processing method, and an image processing program that connect a plurality of images.

  Conventionally, an image processing apparatus and an imaging apparatus that connect a plurality of images to obtain a panoramic image are known (for example, Patent Document 1 and Patent Document 2). A plurality of images for obtaining a panoramic image are taken by a single camera or a plurality of cameras. The plurality of captured images are connected, for example, with the ends of adjacent images overlapped to generate a single panoramic image.

JP 2008-61237 A Special table 2001-501342 gazette

  However, there are cases where a plurality of images to be connected are shot under conditions in which the brightness of the subject to be shot and the light source state are different. Further, when a plurality of images are taken by a plurality of cameras, differences such as white balance and exposure due to individual differences of the cameras may occur. Therefore, the color and brightness of the image may vary from image to image. When images with different hues and brightness are connected, the joints connecting the images become conspicuous. Therefore, in order to connect images with different hues and brightness, a process of adding blur to the connection area (stitching portion) is performed, or another image is generated for connection, and the image is connected via this separate image. It can be considered to be connected. However, such a countermeasure slows down the arithmetic processing speed. In particular, in a moving image, a decrease in the frame rate, a sense of incongruity caused by the fact that the blur of the connected portion differs from frame to frame, and the like occur.

  In view of the above, an object of the present invention is to increase the processing speed of connecting a plurality of images in both a still image and a moving image and to realize image processing in which a connecting portion of the plurality of images is not conspicuous.

  In order to achieve the above object, an image processing apparatus according to the present invention includes an image input unit that inputs a plurality of images and a plurality of images input by the image input unit that overlap each other as a connected region, An image connecting unit for connecting the images, and the image connecting unit divides the connected region into a plurality of sub-regions that are overlapped with each other so that the change in the color difference in the connected region becomes gradual. A blending processing unit that superimposes the sub-regions on each other by changing the mixing ratio is provided.

  “Color” refers to a color including an achromatic color and a chromatic color, and can be expressed by an RGB value, for example. The “color difference” includes a hue difference, a saturation difference, or a brightness difference.

  The image processing apparatus according to the present invention includes the blending processing unit that superimposes the sub-regions on each other by changing the mixing ratio for each sub-region so that the change in the color difference in the connected region becomes gradual. Therefore, it is possible to realize image processing in which a connection portion of a plurality of images is not conspicuous. In addition, the blending processing unit moderates the change in the color difference in the connected region by changing the mixing ratio of the image for each sub-region. Therefore, for example, it is not necessary to separately calculate a coefficient (alpha value) indicating transparency for blurring. Further, it is not necessary to generate another image for connection. Therefore, the processing speed in the image connection process can be increased.

  As described above, according to the image processing apparatus of the present invention, in both a still image and a moving image, the processing speed for connecting a plurality of images is increased, and image processing in which a connecting portion of the plurality of images is not conspicuous is realized. be able to.

  For example, the blending processing unit sets a plurality of sub-regions by dividing a connection region in a connection direction for connecting a plurality of images. Thereby, the change of the color difference in a connection area | region can be made loose in a connection direction.

  As an example, the blending processing unit decreases the mixing ratio of the plurality of images as the ratio of the one image in the plurality of sub-regions approaches the end of the connected region, and the ratio of the other image in the plurality of sub-regions. , It is determined so as to decrease as it approaches the tip of the connection region. Thereby, the color difference in a connection area | region can be changed gradually in a connection direction.

  The blending processing unit may calculate the RGB values of the sub areas based on the mixing ratio. Thereby, blending of images in the sub-region can be easily executed.

  For example, the image connection unit further includes an image correction unit that corrects a plurality of images, and the image correction unit projects a plurality of images onto the spherical surface of the virtual sphere to obtain a plurality of sphere projection images, and Each of the spherical projection images is projected onto a virtual cylindrical peripheral surface to obtain a plurality of cylindrical projection images, thereby correcting the plurality of images. The blending processing unit corrects the plurality of images corrected by the image correction unit. May be connected. Thereby, perspective abnormality (a phenomenon in which perspective is not accurately expressed) and distortion of a plurality of images can be corrected, and the plurality of images can be smoothly connected.

  An imaging apparatus according to the present invention includes an imaging unit that captures a plurality of images and the above-described image processing apparatus. Thereby, according to the imaging device according to the present invention, as in the above-described image processing device, the processing speed for connecting a plurality of images is increased in both a still image and a moving image, and a connecting portion of the plurality of images is provided. Inconspicuous image processing can be realized.

  An image processing method according to the present invention includes: an image input step for inputting a plurality of images; and an image connection step for connecting a plurality of images by overlapping a part of the plurality of images input in the image input step as a connection region. In the image connection step, the connection region is divided into a plurality of sub-regions that are overlapped with each other, and the mixing ratio for each sub-region is changed so that the change in the color difference in the connection region becomes gradual. , Including a blending process step of superimposing the sub-regions on each other. As a result, according to the image processing method of the present invention, as in the above-described image processing device, the processing speed for connecting a plurality of images is increased in both a still image and a moving image, and a connecting portion of the plurality of images is also provided. It is possible to realize image processing in which is not conspicuous.

  An image processing program according to the present invention connects a plurality of images to a computer by superimposing an image input step for inputting a plurality of images and a part of the plurality of images input in the image input step as a connection region. An image processing program for executing an image connection step, wherein the image connection step divides the connection region into a plurality of sub-regions that are overlapped with each other so that a change in color difference in the connection region is moderated. In addition, a blending process step of superimposing the sub-regions on each other by changing the mixing ratio for each sub-region is included. As a result, according to the image processing program of the present invention, as in the above-described image processing apparatus, the processing speed for connecting a plurality of images is increased in both a still image and a moving image, and a connecting portion for a plurality of images is also provided. It is possible to realize image processing in which is not conspicuous.

  The image processing apparatus according to the present invention includes the blending processing unit that superimposes the sub-regions on each other by changing the mixing ratio for each sub-region so that the change in the color difference in the connected region becomes gradual. Therefore, it is possible to realize image processing in which a connection portion of a plurality of images is not conspicuous. In addition, the blending processing unit moderates the change in the color difference in the connected region by changing the mixing ratio of the image for each sub-region. Therefore, for example, it is not necessary to separately calculate a coefficient (alpha value) indicating transparency for blurring. Further, it is not necessary to generate another image for connection. Therefore, the processing speed in the image connection process can be increased.

  As described above, according to the image processing apparatus of the present invention, in both a still image and a moving image, the processing speed for connecting a plurality of images is increased, and image processing in which a connecting portion of the plurality of images is not conspicuous is realized. be able to.

It is a block diagram of the imaging device which concerns on this embodiment. It is an image figure showing the positional relationship of the imaging part which concerns on this embodiment. It is a flowchart which shows the flow of a process of the pre-processing part which concerns on this embodiment. It is explanatory drawing of the temporary image which the pre-processing part which concerns on this embodiment processes. It is an image figure of the spherical cylinder projection which concerns on this embodiment. It is an image figure of the coordinate transformation by the spherical cylinder projection which concerns on this embodiment. It is a flowchart which shows the flow of the process at the time of imaging | photography concerning this embodiment. It is explanatory drawing of the image which the image correction part and image connection part which concern on this embodiment process. It is an image photograph which shows the connection area | region which connected the image with the imaging device which concerns on this embodiment. Fig. 10 is a partially enlarged photograph of Fig. 9. FIG. 11 is a part of a panoramic photograph including the connection area shown in FIGS. 9 and 10. FIG. It is an image photograph which shows the connection area | region which connected the image, without using the imaging device which concerns on this embodiment. FIG. 13 is a part of a panoramic photograph including the connected area shown in FIG. 12.

Embodiments of the present invention will be described below with reference to the accompanying drawings.
<Configuration>
FIG. 1 shows a configuration diagram of an imaging apparatus 1 according to the present embodiment. As an example, the imaging apparatus 1 is a multi-lens camera that generates a panoramic image by connecting a plurality of captured images. The imaging device 1 includes an imaging unit 2, an image input unit 3, a control unit 4, a preprocessing unit 5, an image correction unit 22, an image connection unit 6, a storage unit 7, an output unit 10, and an input. Part 11.

  The imaging unit 2 captures a plurality of images constituting a panoramic image. The imaging unit 2 includes an imaging optical system 14 that includes a lens, a diaphragm, and the like, and an imaging element 15. The image sensor 15 is an image sensor such as a CCD or a CMOS. The imaging element 15 includes a color filter, an amplifier, an A / D converter, and the like, photoelectrically converts an image captured by the imaging optical system 14, and outputs a digital image signal to the image input unit 3. In the present embodiment, in the imaging unit 2, three sets of imaging mechanisms including the imaging optical system 14 and the imaging element 15 (first imaging unit 2A, second imaging unit 2B, and third imaging unit 2C) are provided. Yes.

  FIG. 2 is an image diagram showing the positional relationship of the imaging unit 2. The first imaging unit 2A, the second imaging unit 2B, and the third imaging unit 2C rotate in the left-right rotation direction R with respect to the central axis 2X, and are shifted in the up-down direction H. The first imaging unit 2A rotates counterclockwise by an angle a1 with respect to the second imaging unit 2B and is installed at a low position by a height h1. The third imaging unit 2C rotates clockwise by an angle a2 with respect to the second imaging unit 2B, and is installed at a high position by a height h2 with respect to the second imaging unit 2B.

  Therefore, the first imaging unit 2A captures the position rotated by the angle a1 from the imaging region of the second imaging unit 2B, and the third imaging unit 2C detects the angle a2 from the imaging region of the second imaging unit 2B. Take a picture of the rotated position. The vertical displacement (height h1, h2) is actually slightly smaller than the lateral displacement (angles a1, a2). By connecting the images captured by the first imaging unit 2A, the second imaging unit 2B, and the third imaging unit 2C in the left-right rotation direction R, the third imaging is performed from the imaging region of the first imaging unit 2A. A panoramic image in which the area up to the photographing area of the part 2C is continuous is generated.

  The imaging optical system 14 of the first imaging unit 2A, the imaging optical system 14 of the second imaging unit 2B, and the imaging optical system 14 of the third imaging unit 2C are aligned at the position of the entrance pupil. Yes. That is, the entrance pupil position of the imaging optical system 14 of the first imaging unit 2A, the imaging optical system 14 of the second imaging unit 2B, and the imaging optical system 14 of the third imaging unit 2C, which are set up and down. Are consistent.

  Since the positional relationship among the first imaging unit 2A, the second imaging unit 2B, and the third imaging unit 2C is relatively fixed, the vertical deviation (height h1, h2) and the horizontal deviation ( The angles a1 and a2) do not vary during shooting.

  The image input unit 3 inputs a plurality of images to the control unit 4, and outputs a digital image signal input from the imaging unit 2 to the control unit 4. The control unit 4 controls each process of the imaging device 1. The preprocessing unit 5 performs setting of an effective image area and a connected area before photographing. The image correction unit 22 corrects the image captured by the imaging unit 2, and the image connection unit 6 connects the corrected images. The image input unit 3, the control unit 4, the preprocessing unit 5, the image correction unit 22, and the image connection unit 6 are realized by a calculation unit such as a CPU, and a storage unit such as a main memory, a flash memory, and a hard disk. Work together.

  The preprocessing unit 5 includes a temporary image reading unit 16, a temporary image correction unit 17, an effective image region setting unit 18, and a connected region setting unit 19. Since the image processed by the preprocessing unit 5 is a setting image, it is called a temporary image. The temporary image reading unit 16 reads a plurality of temporary images captured by the imaging unit 2 via the image input unit 3 and the control unit 4. The temporary image correction unit 17 corrects the perspective abnormality and distortion of the temporary image read by the temporary image reading unit 16. In the present embodiment, the temporary image correction unit 17 performs correction by spherical cylindrical projection.

  Correction by spherical cylindrical projection is a correction method that combines spherical projection and cylindrical projection, and projects multiple images on the spherical surface of a virtual sphere in order to eliminate perspective abnormality and distortion of the captured image. This is a correction method in which an image projected onto a spherical surface is further projected onto a virtual cylindrical circumferential surface in order to develop it on a plane.

  The temporary image correction unit 17 obtains a plurality of spherical projection images by projecting a plurality of temporary images onto the spherical surface of the virtual sphere, and projects the plurality of spherical projection images onto a virtual cylindrical circumferential surface to thereby provide a plurality of projections. A plurality of captured temporary images are corrected by acquiring a cylindrical projection image. The effective image area setting unit 18 sets an effective image area to be used as a panoramic image in a plurality of captured temporary images. The connection area setting unit 19 sets a connection area for connecting a plurality of captured images.

  The image correcting unit 22 and the image connecting unit 6 function at the time of capturing a panoramic image after the setting of the effective image region and the connecting region by the preprocessing unit 5 is completed. The image correction unit 22 corrects perspective abnormality and distortion of a plurality of images input via the image input unit 3 and the control unit 4. In the present embodiment, the image correction unit 22 performs correction by spherical cylindrical projection. The image correction unit 22 projects a plurality of captured images onto a spherical surface of a virtual sphere to obtain a plurality of spherical projection images, and projects the plurality of spherical projection images onto a virtual cylindrical circumferential surface to project a plurality of spherical projection images. By acquiring a cylindrical projection image, a plurality of captured images are corrected.

  The image connecting unit 6 includes a blending processing unit 24, a connected region / non-connected region combining unit 28, and an image shaping unit 29. The blending processing unit 24 includes a connected area determination unit 25 and a blending execution unit 26. The connection area determination unit 25 divides the connection area into a plurality of sub areas that are overlapped with each other. In this embodiment, a connection area | region is divided | segmented into the connection direction which connects a some image, and a some sub area | region is set. The blending execution unit 26 superimposes the sub-regions on each other by changing the mixing ratio for each sub-region so that the change in the color difference in the connected region becomes gradual.

  In the present embodiment, the blending processing unit 24 reduces the mixing ratio of the plurality of images as the ratio of the plurality of sub-regions of one image approaches the tip of the connected region, and the plurality of sub-regions of the other image. The ratio at is determined to decrease as it approaches the tip of the connected region. As an example, the blending processing unit 24 calculates the RGB value of the sub-region based on the mixing ratio of the images.

  The connected region / non-connected region combining unit 28 combines the connected region divided and overlapped into a plurality of sub-regions by the blending processing unit 24 and the non-connected region that is an image portion other than the connected region, thereby combining a single image. Generate. The image shaping unit 29 shapes the generated single image into an effective image area set by the preprocessing unit 5.

  The storage unit 7 includes storage means such as a main memory, a flash memory, and a hard disk. The storage unit 7 includes an image connection setting data storage unit 31, a captured image storage unit 32, an image correction data storage unit 33, a corrected image storage unit 34, a blended image storage unit 35, and a combining process. A completed image storage unit 36 and a shaped image storage unit 37 are provided.

  The image connection setting data storage unit 31 stores the contents set by the preprocessing unit 5 as image connection setting data. The image connection setting data includes data indicating the size and position of the effective image area, data indicating the size and position of the connection area, and setting data for a plurality of sub areas.

  The captured image storage unit 32 stores a plurality of images captured by the imaging unit 2 and input via the image input unit 3 and the control unit 4. The image correction data storage unit 33 stores data used when the temporary image correction unit 17 and the image correction unit 22 correct an image, holds a coordinate conversion table, and stores various parameters and various setting values. To do. The corrected image storage unit 34 stores the corrected image corrected by the image correction unit 22. The blended image storage unit 35 stores the image processed by the blending processing unit 24. The combined image storage unit 36 stores a single image generated by the connected region and non-connected region combining unit 28. The shaped image storage unit 37 stores the image shaped by the image shaping unit 29.

  The output unit 10 outputs a signal from the imaging device 1 to the outside. The output unit 10 includes a display device such as a display, a printing device such as a printer, a transmission device to another computer, and the like. In the present embodiment, the output unit 10 includes a display device such as a display. The input unit 11 inputs an external signal to the imaging device 1. The input unit 11 includes input means such as a touch panel, a shutter button, various input buttons, and a keyboard. In the present embodiment, the input unit 11 includes a shooting instruction input button that doubles as a moving image shooting start instruction and a moving image shooting stop instruction button and a still image shooting shutter button.

<Process flow>
[Processing Sequence of Pre-Processing Unit 5]
Next, the flow of image processing executed by the imaging device 1 will be described. First, before starting normal shooting, the size and position of an effective image area used as a panoramic image and the size and position of a connected area are set by the preprocessing unit 5.

  When the shooting instruction input button of the input unit 11 is pressed, the first imaging unit 2A, the second imaging unit 2B, and the third imaging unit 2C start capturing a still image at the same time. The imaging here is called provisional imaging because it is an imaging for setting an effective image area and a connected area. The temporary image obtained by the temporary imaging is input to the preprocessing unit 5 via the image input unit 3 and the control unit 4. The preprocessing unit 5 uses this temporary image to set an effective image area and a connected area. A processing sequence of the preprocessing unit 5 is shown in FIG. 3, and FIG. 4 is an explanatory diagram of a temporary image processed by the preprocessing unit 5.

  In step S1 of FIG. 3, the temporary image reading unit 16 reads the input temporary image. As illustrated in step S1 of FIG. 4, the first temporary image 41 captured by the first imaging unit 2A, the second temporary image 42 captured by the second imaging unit 2B, and the third imaging unit. The third temporary image 43 captured by 2C is read.

  In step S <b> 2 of FIG. 3, the temporary image correction unit 17 corrects the plurality of read temporary images 41, 42, and 43. In the present embodiment, as described above, correction by spherical cylindrical projection is performed. Correction by spherical cylinder projection will be described below. Currently used lenses have the property of projecting a two-dimensional lens in two dimensions. Therefore, in the planar projection used for normal photography, in the case of a three-dimensional object, abnormalities and distortion of the edge of the captured image occur, and in the case of a planar object, the perspective of the edge of the captured image. An abnormality occurs. For this reason, in addition to the problem of perspective abnormality and distortion of individual images, there may be a problem that the images cannot be matched and connected. Therefore, by projecting the image into a sphere, the perspective abnormality and distortion of the individual images are corrected, and the images can be matched and smoothly connected. Further, the spherical image projected in a spherical shape is developed into a planar image by cylindrical projection.

  An image of spherical cylindrical projection is shown in FIG. The radius of the virtual sphere 70 that performs the sphere projection is set to be equal to the distance L on the optical axis from the exit pupil of the imaging optical system 14 to the image sensor 15 (hereinafter referred to as “exit pupil distance”). In the present embodiment, the exit pupil distances L are the same in each of the first imaging unit 2A to the third imaging unit 2C. The first plane 71, the second plane 72, and the third plane 73 expressed in contact with the first spherical surface 75, the second spherical surface 76, and the third spherical surface 77 outside the virtual sphere 70 are imaged. The image plane on which the projected image is projected is shown. The first temporary image 41 before correction is a flat image projected onto the first plane 71, and the second temporary image 42 before correction is a flat image projected onto the second plane 72, The third temporary image 43 before correction is a planar image projected onto the third plane 73.

  When the temporary images 41, 42, 43 before correction are projected in a sphere, the first temporary image 41 is projected onto the first spherical surface 75, the second temporary image 42 is projected onto the second spherical surface 76, The third temporary image 43 is projected onto the third spherical surface 77. That is, the pixels 78 constituting the first temporary image 41 before correction are converted into corresponding pixels 79 on the spherical surface 75. Since the four corners of the image cannot be connected well only by correction by spherical projection, a spherical projection image obtained by projecting onto the spherical surfaces 75, 76, 77 is developed in a plane by cylindrical projection. Thereby, the coordinate of the pixel 79 is further converted into the pixel 80 by the cylindrical projection. The pixel 80 is located on the first plane 71.

  The coordinate conversion by spherical cylinder projection will be further described below. FIG. 6 shows an image diagram of coordinate conversion by spherical cylindrical projection. (A) The front view of the virtual sphere 70 is shown in the figure, and the right side view of the virtual sphere 70 is shown in the (b) figure. The plane projection plane (image plane) 81 corresponds to the first plane 71 to the third plane 73 in FIG.

  The “sphere projection calculation step” will be described below. A straight line is drawn between the point Pn on the image plane 81 and the center point C of the virtual sphere 70 to determine the intersection Sn of the virtual sphere 70 and the straight line. This intersection Sn, that is, the points S1, S2, S3 in FIG. 6, corresponds to the pixel 79 (shown in FIG. 5) mapped on the spherical surface of the virtual sphere 70 from the image plane 81, and P1, P2, P3 in FIG. Corresponds to the pixel 78 before correction (shown in FIG. 5).

The equation of the virtual sphere 70 around the point C (0, 0, L) is
The parameter L in the above equation (1) indicates the radius of the virtual sphere 70. In this embodiment, the radius of the virtual sphere 70 is set to be equal to the exit pupil distance described above.

On the other hand, the equation of the straight line connecting the point S (x ₂ , y ₂ , z ₂ ) (intersection of the virtual sphere 70 and a straight line) and the center point C on the spherical surface of an arbitrary virtual sphere 70 is
From
It becomes.
Let t be the straight line equation,

On the image plane 81, z = 0, and t is determined as follows from Equation (6) under the conditions.
Using the formula (7), the formula (4), the formula (5), and the formula (6), the point P (x, y, z (= 0)) on the image plane 81 and the point P to the point C are moved. linear relationship virtual sphere 70 and intersecting point coordinates _{S (x 2, y 2,} z 2) extending Te is calculated.

  Next, the “cylindrical projection calculation step” will be described. The relationship between the intersection point Sn and the point Pn ′ on the image plane 81 when the spherical projection image represented by the intersection point Sn is projected onto the peripheral surface of the virtual cylinder and developed in a plane is obtained. The point Pn ′, that is, the points P1 ′, P2 ′, and P3 ′ in FIG. 6 correspond to the corrected pixel 80 (shown in FIG. 5).

The equation of a straight line connecting the point Q (0, y _c , L) on the center line of the virtual cylinder and the point P (x _p , y _p , 0) on the image plane 81 is
It becomes.
These are the equations of the virtual sphere 70

Assigned to

From
It becomes.
From the equations (9), (10), and (11), the intersection Sn (the intersection between the virtual sphere 70 and the straight line connecting the coordinate Pn ′ on the image plane 81 and the cylindrical projection coordinate Qn on the y axis) is obtained. It is done.

  By adopting a reverse tracing or reverse tracking method and executing the spherical projection calculation step after executing the cylindrical projection calculation step, the coordinate Pn before correction is specified from the corrected coordinate Pn ′. Can be associated.

  First, a cylindrical projection calculation step is executed to identify the coordinates of the points Sn on the virtual sphere 70 corresponding to the coordinates of the points Pn ′ on the image plane 81 when the cylindrical projection is performed, and associate them with each other. Next, a spherical projection calculation step is executed, and the coordinates of the point Pn on the image plane 81 before correction are specified and associated with the coordinates of the point Sn obtained by the cylindrical projection calculation step. Thereby, the coordinates of the pixel after correction (the coordinates of the point Pn ′) and the coordinates of the pixel before the correction (the coordinates of the point Pn) can be associated with each other.

  The coordinate conversion table held in the image correction data storage unit 33 stores the correspondence between the corrected pixel coordinates and the uncorrected pixel coordinates. By converting coordinates using the coordinate conversion table, correction by spherical cylindrical projection can be executed at high speed. Note that the image correction data storage unit 33 stores a value indicating the exit pupil distance to be substituted for the parameter L, the number of pixels of the image sensor 15, and the size of one pixel. The number of pixels of the image sensor 15 and the size of one pixel are used to determine the values of x and y.

  For example, when the number of pixels of the image sensor 15 is 640 × 480 pixels, the coordinate conversion table is expressed as a table having integer values x and y as members in a two-dimensional array such as TABLE [640] [480]. Also good. In this case, the correspondence relationship between the coordinates (X, Y) of the image after coordinate conversion (after correction) and the coordinates (a, b) of the image before coordinate conversion (before correction) is TABLE [X] [Y]. x = a (coordinate value), TABLE [X] [Y]. It can be expressed as y = b (coordinate value). Thus, by using the coordinate conversion table, the coordinates (X, Y) of the corrected image can be determined at high speed. The temporary image correction unit 17 reads the coordinate conversion table from the image correction data storage unit 33 and executes image correction by spherical cylindrical projection.

  When the image correction by the temporary image correction unit 17 is completed, an effective image region is set by the effective image region setting unit 18 in step S3 of FIG. As described above, since the first imaging unit 2A to the third imaging unit 2C are shifted from each other in the vertical direction, the captured images are also shifted in the vertical direction. Therefore, in order to obtain a panoramic image shaped into a rectangle or the like, it is necessary to set an effective image area that can be used in the captured image.

  A frame 45 in step S3-1 in FIG. 4 indicates the size and position of the effective image area in the vertical direction. Since the images at the upper and lower ends protruding from the frame line 45 are not used, they are cut as unused upper and lower ends 41c, 42c, and 43c. The size and position of the effective image area in the vertical direction are set based on the value of the vertical shift (the height h1 and the height h2 shown in FIG. 2) from the first imaging unit 2A to the third imaging unit 2C. May be determined.

  Further, in order to connect images more naturally, it is necessary to set how much the images are overlapped and connected at which position. At this time, it is not necessary to synthesize all the overlapped areas. The smaller the area to be combined, the smoother the connected part and the higher the processing speed. Therefore, the end portions of the image that are not used are cut as non-use connection end portions 41b, 42b, and 43b. As shown in step S3-2 of FIG. 3, among the areas surrounded by the frame line 45, the areas excluding the unused connection end portions 41b, 42b, and 43b are effective image areas 41a, 42a, and 43a. The size and position of the connection direction W of the effective image areas 41a, 42a, and 43a are set by the amount of deviation in the left-right rotation direction R from the first imaging unit 2A to the third imaging unit 2C (angle a1, angle shown in FIG. You may determine based on the value of a2). The effective image areas 41a, 42a, and 43a may be set using predetermined setting values (for example, a data array).

  Further, the corrected temporary images 41, 42, 43 are displayed on the display device of the output unit 10, and the temporary images 41, 42, 43 are overlaid while adjusting the overlapping position of the temporary images 41, 42, 43. The effective image areas 41a, 42a, and 43a may be set so that the provisional images 41, 42, and 43 can be connected to each other at a position where the combined state can be seen more visually.

  Next, in step S4 of FIG. 3, the connected area setting unit 19 sets a connected area divided into a plurality of sub-areas in the effective image areas 41a, 42a, and 43a. As shown in step S4 in FIG. 4, in the effective image areas 41 a, 42 a, and 43 a, a part where adjacent images are overlapped is a connected area 46, and a part other than the connected area 46 is a non-connected area 47. The connection region 46 is divided into a plurality of sub-regions along the connection direction W. In the present embodiment, for the sake of explanation, it is divided into eight sub-regions as an example.

  The sub area is denoted as SUBDST, and the required number of rect variables corresponding to SUBDST is defined. In the effective image area 41a of the first image, SUBDST [0] to SUBDST [7] are set in the connection area 46. In the effective image area 42a of the second image, SUBDST [8] to SUBDST [15] are set in the connection area 46 connected to the effective image area 41a, and the connection area connected to the effective image area 43a of the third image. At 46, SUBDST [16] to SUBDST [23] are set. In the effective image area 43a, SUBDST [24] to SUBDST [31] are set in the connection area 46. As an example, the size of one sub-region is 1 pixel to 2 pixels.

  In step S5 of FIG. 3, the image connection setting data storage unit 31 stores the contents set by the preprocessing unit 5 as image connection setting data. The image connection setting data includes data indicating the size and position of the effective image area, data indicating the size and position of the connection area, and setting data of a plurality of sub areas (SUBDST [0] to SUBDST [31]). included.

  When the setting by the preprocessing unit 5 is completed, panoramic image shooting by the imaging device 1 can be executed. As described above, since the positional relationship among the first imaging unit 2A, the second imaging unit 2B, and the third imaging unit 2C is relatively fixed, the vertical displacement (FIG. 2, height h1, h2) and the left and right deviations (angles a1 and a2 shown in FIG. 2) do not vary. Therefore, when the setting by the preprocessing unit 5 is completed, the setting by the preprocessing unit 5 needs to be performed again unless the arrangement of the first imaging unit 2A, the second imaging unit 2B, and the third imaging unit 2C is changed. There is no.

[Processing sequence during shooting]
FIG. 7 shows a processing sequence at the time of shooting, and FIG. 8 is an explanatory diagram of an image to be processed. When the shooting instruction input button of the input unit 11 is pressed, the imaging unit 2 starts imaging. In step S <b> 10 of FIG. 7, a plurality of images captured by the imaging unit 2 are read via the image input unit 3 and the control unit 4. As illustrated in step S10 of FIG. 8, the first image 51 captured by the first image capturing unit 2A, the second image 52 captured by the second image capturing unit 2B, and the third image capturing unit. A third image 53 captured by 2C is read.

  In step S <b> 11 of FIG. 7, the captured image storage unit 32 stores images 51, 52, and 53. In step S <b> 12, the image correction unit 22 reads and corrects the images 51, 52, and 53 from the captured image storage unit 32. Here, the image correction unit 22 reads the above-described coordinate conversion table from the image correction data storage unit 33, executes correction by spherical cylindrical projection, and corrects the perspective abnormality and distortion of the images 51, 52, and 53. . The correction by the spherical cylindrical projection is the same as the above-described correction executed by the temporary image correction unit 17 in the preprocessing unit 5.

  In step S13 of FIG. 7, a plurality of images 51, 52, and 53 corrected by the corrected image storage unit 34 are stored. In step S <b> 14, the connection area determination unit 25 reads the corrected images 51, 52, and 53 from the corrected image storage unit 34 and reads the image connection setting data from the image connection setting data storage unit 31. In step S15, the effective image area and the connection area in the images 51, 52, and 53 are determined based on the image connection setting data.

  As shown in step S15-1 and step S15-2 in FIG. 8, the unused connection end portions 51b, 52b, and 53b are cut based on the data indicating the size and position of the effective image area. In this embodiment, since the upper and lower end portions that are not used are cut after the image connection processing, the image portions 51a, 52a, 52a, 52b, 53b other than the unused connection end portions 51b, 52b, 53b are configured. 53a includes an unused upper and lower end portion 67 (step S22 shown in FIG. 8) to be cut later, in addition to the effective image area.

  In step S16 of FIG. 7, the connected region determination unit 25 divides the connected region into a plurality of sub regions based on the data indicating the size and position of the connected region and the setting data of the plurality of sub regions. As shown in step S16 of FIG. 8, a connected area 56 and a non-connected area 57 are determined for each of the image portions 51a, 52a, and 53a. The connection area 56 is divided into a plurality of sub-areas SUBDST [0] to SUBDST [31].

  In step S17 of FIG. 7, the blending execution unit 26 changes the image mixing ratio for each sub-region, and superimposes the sub-regions SUBDST [0] to SUBDST [31] on each other. As shown in step S <b> 17 of FIG. 8, the connection area 46 of the first image 51 and the connection area 56 of the second image 52 are overlapped for each sub-area, and the connection area 56 of the second image 52 and the third area The connection area 56 of the image 53 is overlaid for each sub-area. The sub-region after being overlapped is defined as DST.

  In the connection area 56 of the first image 51 and the connection area 56 of the second image 52, the sub area SUBDST [0] and the sub area SUBDST [8] overlap to form the sub area DST [0], and the sub area SUBDST [1]. ] And sub-region SUBDST [9] overlap to form sub-region DST [1], and sub-region SUBDST [2] and sub-region SUBDST [10] overlap to form sub-region DST [2]. Similarly, SUBDST [3] to SUBDST [7] overlap with SUBDST [11] to SUBDST [15] to form sub-regions DST [3] to DST [7]. At this time, in the sub areas DST [0] to DST [7], the first image 51 and the second image 52 are weighted and blended.

  In the present embodiment, the proportion of the first image 51 in the sub-region DST [0] is 8/9, and the proportion of the second image 52 is 1/9. The proportion of the first image 51 in the sub-region DST [1] is 7/9, and the proportion of the second image 52 is 2/9. The proportion of the first image 51 in the sub-region DST [2] is 6/9, and the proportion of the second image 52 is 3/9. The proportion of the first image 51 in the sub-region DST [3] is 5/9, and the proportion of the second image 52 is 4/9. The proportion of the first image 51 in the sub-region DST [4] is 4/9, and the proportion of the second image 52 is 5/9. The proportion of the first image 51 in the sub-region DST [5] is 3/9, and the proportion of the second image 52 is 6/9. The proportion of the first image 51 in the sub-region DST [6] is 2/9, and the proportion of the second image 52 is 7/9. The proportion of the first image 51 in the sub-region DST [7] is 1/9, and the proportion of the second image 52 is 8/9.

  As described above, the blending execution unit 26 sets the mixing ratio of the first image 51 and the second image 52, and the ratio of the first image 51 in the sub-area is the tip of the connected area 56 (SUBDST [7] side). Decide to decrease as you approach. In addition, the blending execution unit 26 sets the mixing ratio of the first image 51 and the second image 52, and the ratio of the second image 52 in the plurality of sub-regions to the tip of the connected region 56 (SUBDST [8] side). It is determined to decrease as it approaches. In the present embodiment, the blending execution unit 26 makes the ratio of the first image 51 and the ratio of the second image 52 constant by 1/9 (that is, 1 / (number of subregions + 1)) for each subregion. The rate is decreased.

  Here, since the first image 51 and the second image 52 are blended at the determined mixing ratio, it is not necessary to use an alpha value indicating transparency. For example, when the first image 51 is a black image and the second image 52 is a white image, a black RGB value (0, 0, 0) and a white RGB value (255, 255, 255) are mixed. It may be increased or decreased corresponding to. Since the proportion of the first image 51 in the sub-region DST [0] is 8/9 and the proportion of the second image 52 is 1/9, the RGB values are multiplied by the respective proportions and added. Then, the RGB value of the sub-region DST [0] becomes (227, 227, 227) (rounded to the first decimal place), and a light gray image is obtained. By similarly calculating the RGB values of the other sub-regions DST, the first image 51 and the second image 52 are blended at different ratios for each sub-region.

  Similar to the processing in the connection area 56 of the first image 51 and the second image 52 described above, in the connection area 56 of the second image 52 and the connection area 56 of the third image 53, the sub area SUBDST [16]. The sub-region SUBDST [23] and the sub-region SUBDST [24] to the sub-region SUBDST [31] are overlapped and blended. Similar to the connection of the first image 51 and the second image 52, the mixing ratio of the second image 52 and the third image 53 is set so that the ratio in the sub-area of the second image 52 is the tip of the connection area 56. It is determined so that it decreases as it approaches (SUBDST [23] side), and the ratio in the plurality of sub-regions of the third image 53 decreases as it approaches the tip of the connected region 56 (SUBDST [24] side). Yes. As described above, the blending processing unit 24 performs multi-stage blending in which the connected region 56 is divided into a plurality of blends.

  In step S18 of FIG. 7, the blended image storage unit 35 stores the blended images of the connected regions 60 and 61 and the image of the non-connected region 57. As shown in step S18 of FIG. 8, the unconnected region 57 (non-connected region [0]) of the first image 51, the combined connected region 60 (DST [0] to DST [7]), the second The non-connected region 57 (non-connected region [1]) of the image 52 and the blended connected region 61 (DST [8] to DST [15]) are stored.

  In step S19 in FIG. 7, the connected region and unconnected region combining unit 28 reads the blended connected region 60 and 61 images and the unconnected region 57 image from the blended image storage unit 35 to obtain a single image. Generate. As shown in step S19 of FIG. 8, the unconnected region 57 (non-connected region [0]) of the first image 51, the connected region 56 of the first image, and the connected region 56 of the second image 52 are blended. Connected region 60 (DST [0] to DST [7]), non-connected region 57 (non-connected region [1]) of the second image 52, connected region 56 of the second image 52 and the third A connected area 61 (DST [8] to DST [15]) obtained by blending the connected areas 56 of the image 53 is generated into a single image 62 composed of continuous areas [0] to [18].

  In step S20 of FIG. 7, the single image 62 generated by the combined image storage unit 36 is stored. In step S <b> 21, the image shaping unit 29 reads the single image 62 from the combined image storage unit 36 and reads the image connection setting data from the image connection setting data storage unit 31. In step S22, the image shaping unit 29 shapes a single image based on data indicating the size and position of the effective image area. As shown in step S <b> 22 of FIG. 8, the area surrounded by the frame 66 is an effective image area, and the unused upper and lower end portions 67 protruding from the frame 66 are cut.

  In step S <b> 23 of FIG. 7, the shaped image storage unit 37 stores the shaped single image 63. In step S24, in the case of a still image, the image linking process by the image linking unit 6 ends, and the single image 63 shaped through the output unit 10 can be output. In the case of a moving image, the processing from step S10 to step S23 is continuously repeated until moving image shooting is completed. The shaped image storage unit 37 stores the shaped single image 63 in, for example, the AVI format, and can output the image via the output unit 10 during or after moving image shooting.

  According to the imaging apparatus 1 described above, the blending processing unit 24 that superimposes the sub-regions on each other by changing the mixing ratio for each sub-region so that the change in the color difference between the connected regions 60 and 61 becomes gradual. Therefore, it is possible to realize image processing in which the connected portions of the plurality of images 51, 52, 53 are not conspicuous.

  FIG. 9 and FIG. 10 show an example of a connection region 60 in which the first image 51 and the second image 52 are connected by the imaging device 1 according to this embodiment, and FIG. 11 shows the connection shown in FIG. 9 and FIG. A part of the panoramic image including the region 60 is shown. FIG. 10 is a partially enlarged photograph of FIG. In the image connection shown in FIGS. 9 to 11, the blending processing unit 24 divides the connection region 60 into 16 sub-regions instead of the above-described eight regions and blends them. The processing is the same as described above.

  FIG. 12 shows a connection region 84 in which the first image 51 and the second image 52 are connected without using the imaging device 1 according to the present embodiment, and FIG. 13 shows the connection shown in FIG. A part of the panoramic image including the region 84 is shown. In FIG. 12, the connection area 84 is not divided into sub-areas, and the first image 51 and the second image 52 are blended at a ratio of 1/2. Therefore, as shown in FIG. 13, in the image connection portion indicated by the dotted line, the difference in color brightness between the first image 51 and the second image 52 appears strongly, and the connection portion becomes conspicuous.

  On the other hand, in FIG. 9 and FIG. 10, since the change of the color difference in the connection area 60 is gradual, the connection part is not conspicuous in the image connection part indicated by the dotted line as shown in FIG. Therefore, according to the imaging device 1, it is possible to obtain a single image 63 in which the connected portion looks more natural.

  In addition, the blending processing unit 24 moderates the change in the color difference in the connected regions 60 and 61 by changing the mixing ratio of the image for each sub-region. Therefore, for example, it is not necessary to separately calculate a coefficient (alpha value) indicating transparency for blurring. Further, it is not necessary to generate another image for connection. Therefore, the processing speed in the image connection process can be increased.

  As described above, according to the imaging device 1 according to the present embodiment, in both the still image and the moving image, the processing speed for connecting the plurality of images 51, 52, and 53 is increased and the plurality of images 51, 52, and 53 are combined. It is possible to realize image processing in which the connected portions are not conspicuous.

  The blending processing unit 24 divides the connection area 56 in the connection direction W that connects the plurality of images 51, 52, 53 and sets a plurality of sub areas. Thereby, the change of the color difference in the connected connection areas 60 and 61 can be moderated in the connection direction W.

  The blending processing unit 24 decreases the mixing ratio of the plurality of images 51, 52, and 53 as the ratio in the plurality of sub-regions of one image approaches the tip of the connection region 56 and the plurality of sub-regions of the other image. The ratio in the region is determined so as to decrease as the tip of the connection region 56 is approached. Thereby, the color difference in the connection areas 60 and 61 can be gradually changed in the connection direction.

  FIGS. 9 to 11 show the connected region 60 divided into 16 sub-regions, but the ratio of the first image 51 and the second region are similar to the blend of the eight sub-regions described above. As the ratio of the images 52 approaches the end of each connected area 56 (shown in FIG. 8), 1 / (number of sub areas + 1) is decreased at a constant rate for each sub area. That is, the proportion of the first image 51 in the sub-region [1] shown in FIG. 10 is 16/17, the proportion of the second image 52 is 1/17, and the proportion in the sub-region [2] The ratio occupied by the first image 51 is 15/17, the ratio occupied by the second image 52 is 2/17, and the ratio occupied by the first image 51 in the sub-region [3] is 14/17. 17, the proportion of the second image 52 is 3/17, and the proportion of the first image 51 in (substantially) the sub-region [16] is 1/17. The ratio occupied by is 16/17.

  As the ratio of the first image 51 and the ratio of the second image 52 decreases toward the end of each connected area 56 (shown in FIG. 8), as shown in FIG. 9 and FIG. A natural gradation is formed in the region 60, and the connected portion can be seen more naturally.

  In the above-described embodiment, the blending processing unit 24 calculates the RGB values of the sub areas based on the mixing ratio. Thereby, blending of images in the sub-region can be easily executed.

  In the above-described embodiment, the imaging apparatus 1 includes the image correction unit 22, and the image correction unit 22 projects a plurality of images 51, 52, and 53 onto the spherical surfaces 75, 76, and 77 of the virtual sphere 70, respectively. The plurality of images 51, 52, and 53 are corrected by acquiring a spherical projection image, projecting the plurality of spherical projection images onto a virtual cylindrical circumferential surface, and acquiring a plurality of cylindrical projection images. Thereby, the perspective abnormality and distortion of the plurality of images 51, 52, 53 can be corrected, and the plurality of images 51, 52, 53 can be smoothly connected.

  Moreover, according to this embodiment, since the connection area | region 56 preset by the pre-processing part 5 should just be connected, it is not necessary to perform high-load processes, such as pattern matching. For example, for a high frame rate video (for example, 30 frames / second), it is difficult to perform a process of updating the synthesis point every 1/30 seconds at a sufficient processing speed even with a current PC. On the other hand, according to the present embodiment, it is possible to perform image connection processing at a lower load and at a higher speed.

<Other embodiments>
Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications and changes can be made based on the technical idea of the present invention. For example, in the above-described embodiment, the imaging apparatus 1 including the imaging unit 2 has been described, but the present invention is not limited to this. The present invention may be realized as an image processing apparatus that does not include the imaging unit 2. The imaging device 1 described above can be realized as a multi-lens camera that captures a panoramic image, but is not limited thereto. A plurality of images 51, 52, and 53 may be connected to generate a single image 63 other than the panoramic image. In the above-described embodiment, the imaging device 1 includes the first imaging unit 2A to the third imaging unit 2C and connects the three images 51, 52, and 53, but is not limited thereto. For example, two imaging units 2 or four or more imaging units 2 may be provided and two or four or more images may be connected. In the above-described embodiment, the sub area divided into 8 or 16 has been described. However, the present invention is not limited to this, and another number of sub areas may be set.

  The mixing ratio of the images 51, 52, and 53 in the sub-region is not limited to the ratio in the above-described embodiment, and other mixing ratios may be used. In the imaging device 1 described above, blending of the images 51, 52, and 53 in the sub-region is performed by adjusting the RGB value. However, the present invention is not limited to this, and expression values of colors other than the RGB value are used. You may perform the blend of the images 51, 52, and 53 in a sub area | region. In the imaging device 1 described above, the unused upper and lower end portions 67 are cut by the image shaping unit 29 after the images 51, 52, and 53 are connected, but the present invention is not limited to this. As an example, in step S15 (shown in FIG. 7), before connecting the images 51, 52, and 53, the unused upper and lower end portions 67 are cut together with the unused connection ends 51b, 52b, and 53b. Also good. In this case, the image shaping unit 29 becomes unnecessary. In the imaging device 1 described above, various image processing such as white balance adjustment may be executed together with the image connection processing.

DESCRIPTION OF SYMBOLS 1 Imaging device 2 Imaging part 3 Image input part 6 Image connection part 22 Image correction part 24 Blending process part 46,56,60,61 Connection area 51,52,53 Multiple images 70 Virtual sphere

Claims (8)

An image input unit for inputting a plurality of images;
An image connecting unit that converts the plurality of images input by the image input unit from a planar image to a spherical projection image, corrects the images , and superimposes a part of the corrected images as a connection region to connect the plurality of images. And
The image connection unit divides the connection region into a plurality of sub-regions that are overlapped with each other, and changes a mixing ratio for each sub-region so that a change in color difference in the connection region is moderate. An image processing apparatus comprising a blending processing unit that superimposes the sub-regions on each other.   The image processing apparatus according to claim 1, wherein the blending processing unit sets the plurality of sub-regions by dividing the connection region in a connection direction for connecting the plurality of images.   The blending processing unit decreases the mixing ratio of the plurality of images as the ratio of the one image in the plurality of sub-regions approaches the tip of the connected region, and the other image in the plurality of sub-regions. The image processing apparatus according to claim 1, wherein the ratio is determined so as to decrease as it approaches the tip of the connection region.   4. The image processing apparatus according to claim 1, wherein the blending processing unit calculates an RGB value of the sub-region based on the mixing ratio. 5. The image connecting unit further includes an image correcting unit that corrects the plurality of images,
The image correction unit projects the plurality of images onto a spherical surface of a virtual sphere to obtain a plurality of sphere projection images, and projects the plurality of sphere projection images onto a virtual cylindrical circumferential surface to project a plurality of sphere projection images. Correcting the plurality of images by acquiring a cylindrical projection image;
The image processing apparatus according to claim 1, wherein the blending processing unit is configured to connect the plurality of images corrected by the image correction unit. An imaging unit that captures the plurality of images;
An imaging device comprising the image processing device according to any one of claims 1 to 5. An image input step for inputting a plurality of images;
An image linking step of connecting the plurality of images by converting the plurality of images input in the image input step from a planar image to a spherical projection image and correcting the images and superimposing a part of the corrected images as a connection region. Including
The image connection step divides the connection region into a plurality of sub-regions that are overlapped with each other, and changes a mixing ratio for each sub-region so that a change in color difference in the connection region becomes gradual. An image processing method including a blending processing step of superimposing the sub-regions on each other. On the computer,
An image input step for inputting a plurality of images;
An image linking step of connecting the plurality of images by converting the plurality of images input in the image input step from a planar image to a spherical projection image and correcting the images and superimposing a part of the corrected images as a connection region. An image processing program for executing
The image connection step divides the connection region into a plurality of sub-regions that are overlapped with each other, and changes a mixing ratio for each sub-region so that a change in color difference in the connection region becomes gradual. An image processing program including a blending processing step of superimposing the sub-regions on each other.