JP2007334558A - Image processor, method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a more natural panoramic image more simply. <P>SOLUTION: In an image processor for processing a plurality of images taken by changing angles of photographing directions, a brightness magnification approximate function calculating part 106 calculates an approximate function that approximates the change of the exposure magnification to the angle of the photographic directions of the respective images from the photographic conditions of the plurality of images, a luminance adjustment part 107 adjusts the luminance of the plurality of images in a manner of changing the luminance continuously in the direction of joining the images according to the approximate function, and an image composition part 108 composites the plurality of images, the luminance of which have been adjusted, into a single composite image by joining them together. The present invention can be applied to digital still cameras. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and method, and a program, and more particularly, to an image processing apparatus and method for synthesizing images, and a program.

  2. Description of the Related Art Conventionally, panoramic photography has been widely performed in which a plurality of images are photographed and the photographed images are combined to form a single image.

  For example, in Patent Document 1, when performing panoramic shooting, the same exposure value is held, and exposure is controlled based on the same exposure value until panoramic shooting is completed, whereby a non-natural panoramic image is obtained. An obtained camera is disclosed.

  Patent Document 2 discloses an electronic camera system that captures a series of images with the same exposure, focus, zoom value, white balance, and the like when performing panoramic shooting.

  Patent Document 3 discloses an apparatus for panoramic composition after reducing the dynamic range of individual captured images by linearly or logarithmically converting a plurality of images having different exposures.

  Patent Document 4 discloses an imaging system that synthesizes an image by controlling image quality parameters set in each imaging device so that the image quality of the composite image is uniform when the image quality of the composite image is not uniform. Has been.

  Patent Document 5 discloses a method for performing dynamic range compression using logarithmic transformation.

  In addition, many studies have been conducted on dynamic range compression while preserving local contrast.

Japanese Patent Laid-Open No. 3-145635

Japanese Patent Laid-Open No. 9-322054

JP 2004-180308 A

JP 2001-320616 A

JP 2002-514359 A

  However, since a panoramic image has a very wide horizontal angle of view, a dynamic range with an extremely wide luminance is required. For example, when a user performs panoramic shooting indoors instead of outdoors, a single panoramic image includes a dark place behind the room, such as the back of furniture, and a bright place outside the window. Become. That is, it can be said that overexposure and underexposure of the image are inevitable.

  Further, when the dynamic range is reduced by linear transformation or logarithmic transformation, the local contrast of the processed image is reduced by such a monotone function transformation.

  Also, it is not possible to create an image with a wide dynamic range of luminance simply by changing the image quality parameter set in the imaging apparatus.

  Also, the dynamic range compression by logarithmic transformation does not preserve the local contrast of the image. An image with reduced contrast does not cause a problem in use as an image taken with a surveillance camera, but when an image is taken as an expression means pursuing artisticity, its appreciation value is extremely low.

  In addition, when dynamic range compression is performed with the local contrast preserved, the low frequency component of the luminance change is obtained from the image, so when incorporated in a mobile device, the amount of calculation is large and the power consumption is high. It becomes unsuitable.

  In this way, when panoramic synthesis is performed, the panorama synthesized image is overexposed and blacked out, and further, the local contrast of the image is reduced. Extremely low.

  For this reason, there is an increasing need for a technique for reducing overexposure and underexposure in a panorama synthesized image and maintaining local contrast of the image.

  The present invention has been made in view of such a situation, and makes it possible to generate a more natural panoramic image more easily.

  An image processing apparatus according to an aspect of the present invention is an image processing apparatus that processes a plurality of images captured by changing an angle of a capturing direction, and is configured to capture each of the images based on a plurality of image capturing conditions. A calculation means for calculating an approximate function that approximates a change in exposure magnification with respect to an angle of a shooting direction, and a brightness of the plurality of images is adjusted so as to be continuously changed in a stitching direction according to the approximate function. And adjusting means and combining means for combining the plurality of images whose luminance has been adjusted to combine them into one combined image.

  A determining unit is further provided for determining a parameter for determining a dynamic range of brightness after adjustment from a plurality of dynamic ranges of the luminance of the image, and the adjusting unit is connected in a direction in accordance with the approximate function and the parameter. The brightness of the plurality of images can be adjusted so as to continuously change.

  A conversion means for converting the RGB color system of the plurality of images into a color system represented by a luminance component and a color component is further provided, and the adjustment means is configured to change the luminance component. The brightness of the plurality of images can be adjusted.

  It is possible to further provide correction means for applying inverse gamma correction processing to the plurality of images and applying gamma correction processing to the composite image before the brightness adjustment.

  An image processing method according to an aspect of the present invention is an image processing method for processing a plurality of images captured by changing an angle of a shooting direction, and is based on the shooting conditions of the plurality of images. An approximate function that approximates the change in the magnification of exposure with respect to the angle of the shooting direction is calculated, and the brightness of the plurality of images is adjusted so as to continuously change in the stitching direction according to the approximate function. The method includes a step of combining the plurality of adjusted images into a single composite image by joining them together.

  A program according to one aspect of the present invention is a program for causing a computer to perform image processing for processing a plurality of images captured by changing an angle of a shooting direction. An approximate function that approximates a change in exposure magnification with respect to the angle of the image capturing direction of the image is calculated, and the brightness of the plurality of images is adjusted so as to continuously change in the stitching direction according to the approximate function And combining the plurality of images with adjusted brightness into one composite image.

  In one aspect of the present invention, an approximate function that approximates a change in the magnification of exposure with respect to an angle in the shooting direction of each of the images is calculated from a plurality of shooting conditions of the images, and according to the approximate function, The brightness of the plurality of images is adjusted so as to be continuously changed in the stitching direction, and the plurality of images whose brightness has been adjusted are stitched together to be combined into one synthesized image.

  As described above, according to one aspect of the present invention, a panoramic image can be generated.

  Also, according to one aspect of the present invention, a more natural panoramic image can be generated more easily.

  Embodiments of the present invention will be described below. Correspondences between the constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  An image processing apparatus according to an aspect of the present invention is an image processing apparatus that processes a plurality of images captured by changing an angle of a capturing direction, and that captures each of the images based on a plurality of image capturing conditions. A calculation means for calculating an approximate function that approximates a change in exposure magnification with respect to a direction angle (for example, the luminance magnification approximate function calculation unit 106 in FIG. 6), and continuously changes in the stitching direction according to the approximate function. Adjusting means (for example, the brightness adjusting unit 107 in FIG. 6) for adjusting the brightness of the plurality of images so as to combine the plurality of images whose brightness has been adjusted to combine them into one composite image. (For example, the image composition unit 108 in FIG. 6).

  A determining unit (for example, an adjustment parameter determining unit 121 in FIG. 6) for determining a dynamic range of luminance after adjustment from the dynamic range of luminance of the plurality of images may be further provided, and the adjusting unit In accordance with the approximate function and the parameter, the brightness of the plurality of images can be adjusted so as to continuously change in the stitching direction.

  Conversion means (for example, the color system conversion unit 105 in FIG. 6) for converting the RGB color system of the plurality of images into a color system represented by a luminance component and a color component may be further provided. The adjusting means can adjust the luminance of the plurality of images by changing the luminance component.

  Before the brightness adjustment, a correction unit (for example, a gamma correction unit 83 in FIG. 6) that applies inverse gamma correction processing to the plurality of images and applies gamma correction processing to the composite image may be further provided. it can.

  The image processing method or program according to an aspect of the present invention calculates an approximate function that approximates a change in the magnification of exposure with respect to an angle of a shooting direction of each of the images from a plurality of shooting conditions of the images (for example, In step S44 in FIG. 12, the brightness of the plurality of images is adjusted so as to be continuously changed in the stitching direction according to the approximate function (for example, step S45 in FIG. 12), and the brightness is adjusted. Are combined to form a single composite image (for example, step 46 in FIG. 12).

  FIG. 1 is a diagram showing the appearance of an embodiment of a digital still camera 1 of the present invention. The digital still camera 1 performs panoramic shooting. The digital still camera 1 is an example of an information processing apparatus. The digital still camera 1 includes a lens unit 11 including an image sensor and a lens block, a lens unit 11 that pivotally supports the lens unit 11, and a main body 12 that controls the entire digital still camera 1. The motor drive unit 13 is configured to rotate.

  The lens unit 11 is provided with a lens 21 constituting a lens block that forms an optical image of a subject on an image sensor. The lens unit 11 is provided with an image sensor (not shown). The imaging device is, for example, a CCD (Charge Coupled Devices) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

  FIG. 2 is a diagram showing an internal configuration of the motor drive unit 13. The motor drive unit 13 includes a position detection sensor 31, a motor 32, and a reduction gear 33.

  The position detection sensor 31 includes, for example, a potentiometer, a resolver, or an encoder, and detects a rotation angle of the lens unit 11 with respect to the main body 12. The motor 32 generates a rotational driving force for rotating the lens unit 11. The reduction gear 33 decelerates the driving force of rotation generated by the motor 32. The lens unit 11 is rotated with respect to the main body 12 by the rotational driving force of the motor 32 decelerated by the reduction gear 33.

  FIG. 3 is a block diagram illustrating a hardware configuration of the digital still camera 1. The main body 12 is provided with a control unit 51, and an external recording medium 52 is detachably attached thereto.

  The control unit 51 controls the lens unit 11 and the motor drive unit 13 to control the entire digital still camera 1. The control unit 51 directs the lens unit 11 in a predetermined direction by the motor drive unit 13 and causes the lens unit 11 to perform photographing.

  The external recording medium 52 includes a semiconductor memory, a hard disk, an optical disk, and a drive thereof, and records image data of captured images.

  The control unit 51 includes a CPU (Central Processing Unit) 61, an A / D (Analog to Digital) converter 62, a motor driver 63, a lens controller 64, an image sensor control circuit 65, an image processing unit 66, a built-in memory 67, and a display 68. Consists of The CPU 61, the lens controller 64, the image sensor control circuit 65, the image processing unit 66, the built-in memory 67, the display 68, and the external recording medium 52 are connected to each other by a bus.

  The CPU 61 executes the program to control the A / D converter 62, the motor driver 63, the lens controller 64, the image sensor control circuit 65, the image processing unit 66, the built-in memory 67, the display 68, and the external recording medium 52.

  The A / D converter 62 converts, for example, an analog signal that is supplied from the position detection sensor 31 that is a potentiometer and indicates the rotation angle of the lens unit 11 with respect to the main body 12 into a digital signal. The A / D converter 62 supplies the CPU 61 with a digital signal obtained by the conversion and indicating the rotation angle of the lens unit 11 with respect to the main body 12.

  The motor driver 63 drives the motor 32 based on the control of the CPU 61 so as to rotate the drive shaft of the motor 32.

  Based on the control of the CPU 61, the lens controller 64 controls the lens block of the lens unit 11 so as to control focus, zoom, or aperture. The imaging element control circuit 65 causes the imaging element to photograph the subject based on the control of the CPU 61, that is, causes the imaging element to convert an optical subject image formed by the lens block into an electrical signal. The image sensor provided in the unit 11 is controlled. Further, the image sensor control circuit 65 acquires an electrical signal output from the image sensor as a result of photographing the subject, and stores it in the built-in memory 67 via the bus. For example, when an analog electrical signal of an image from the image sensor is output, the image sensor control circuit 65 converts the analog electrical signal from the image sensor into a digital electrical signal, and then converts the digital electrical signal. The image data is stored in the built-in memory 67.

  The image processing unit 66 applies various types of image processing to image data obtained by photographing with an image sensor, which is stored in the built-in memory 67.

  The built-in memory 67 is composed of SDRAM (Synchronous Dynamic Random Access Memory) or the like, and stores image data and the like.

  The display 68 includes an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display, and displays an image corresponding to the image data stored in the built-in memory 67 under the control of the CPU 61. For example, the display 68 displays an image of a subject photographed on the image sensor or an image corresponding to image data recorded on the external recording medium 52.

  FIG. 4 is a diagram showing a shooting field angle and a horizontal field angle of the digital still camera 1. 4A shows the center of rotation of the lens unit 11. FIG. The center A of the rotation of the lens unit 11 is the position of the lens NP point at the zoom lens wide-angle end of the lens block. The NP point is a point where a chief ray passing through the Gaussian region is selected from chief rays passing through the center of the stop, and a linear component in the object space of the chief ray is extended to intersect the optical axis. The rotation center A of the lens unit 11 is determined so as to minimize the parallax of the captured image.

  The horizontal angle of view β indicates the horizontal angle of view of the lens unit 11. The imaging angle of view α indicates a range of angles in the horizontal direction that can be captured by rotating the lens unit 11 that rotates relative to the main body 12.

  For example, if the shooting angle of view α and the horizontal angle of view β are known in advance, the number of images required for panoramic synthesis of images with the shooting angle of view α can be calculated.

  FIG. 5 is a diagram illustrating the difference in image brightness depending on the shooting direction. When a shutter button (not shown) is pressed, the digital still camera 1 rotates the lens unit 11 with respect to the main body 12 and displays the image when the lens unit 11 is at a predetermined angle with respect to the main body 12. This is repeated within the range of the shooting angle of view α, and a plurality of images are continuously shot.

  That is, in this case, the lens unit 11 is rotated with respect to the main body 12 around the rotation center A of the lens unit 11. For example, the digital still camera 1 captures the image 71 when the lens unit 11 is at the position of the angle facing the leftmost side with respect to the main body 12. The digital still camera 1 rotates the lens unit 11 clockwise.

  The digital still camera 1 captures an image 72 when the lens unit 11 is at a central angle between the position of the lens unit 11 facing the left side and the position facing the front (center) with respect to the main body 12. Further, the digital still camera 1 rotates the lens unit 11 clockwise.

  The digital still camera 1 captures an image 73 when the lens unit 11 is at an angle with respect to the main body 12 facing the front. Further, the digital still camera 1 rotates the lens unit 11 clockwise.

  The digital still camera 1 captures an image 74 when the lens unit 11 is at a central angle between the position facing the front and the position facing the rightmost side with respect to the main body 12. Further, the digital still camera 1 rotates the lens unit 11 clockwise.

  The digital still camera 1 captures an image 75 when the lens unit 11 is at a position that is at the most right angle with respect to the main body 12.

  For example, when the left side is dark with respect to the front of the digital still camera 1 and the right side is bright with respect to the front of the digital still camera 1, a dark landscape is shot in the left half of the image 71, the image 72, and the image 73. A bright scene is photographed in the right half of the image 73, the image 74, and the image 75.

  Since the images 71 to 75 are finally stitched together and synthesized into one synthesized image, the images at the left and right end portions are photographed so as to overlap each other.

  FIG. 6 is a diagram for explaining functions realized by the CPU 61 of the control unit 51 for executing a program. When the CPU 61 executes the program, an imaging control unit 81, a panoramic image generation processing unit 82, and a gamma correction unit 83 are realized.

  The shooting control unit 81 controls the rotation of the lens unit 11 and shooting by the lens unit 11. The imaging control unit 81 includes a rotation control unit 101, an exposure adjustment unit 102, and a parameter storage control unit 103.

  The rotation control unit 101 controls the rotation of the lens unit 11 relative to the main body 12 by controlling the rotation of the motor 32. For example, the rotation control unit 101 indicates the position of the angle of the lens unit 11 with respect to the main body 12 indicated by a digital signal obtained by analog-digital conversion of the signal output from the position detection sensor 31 by the A / D converter 62. However, the lens unit 11 is rotated with respect to the main body 12 by causing the motor driver 63 to drive the motor 32 by transmitting a predetermined control command to the motor driver 63 so that the position is at a desired angle. .

  The exposure adjustment unit 102 adjusts the exposure in image shooting. Specifically, the exposure adjustment unit 102 adjusts the exposure according to the shooting direction by adjusting the shutter speed, ISO (International Organization for Standardization) sensitivity, and the lens aperture.

  The parameter storage control unit 103 controls storage of exposure parameters indicating the shutter speed, ISO sensitivity, lens aperture, and the like in the built-in memory 67 when an image is taken.

  The panoramic image generation processing unit 82 calculates an approximation function that approximates the change in the magnification of the exposure with respect to the angle in the shooting direction of each image from the shooting conditions of the plurality of images shot by rotating the lens unit 11. Then, in accordance with the approximate function, the brightness of the plurality of images is adjusted so as to be continuously changed in the joining direction, and the plurality of images having the adjusted brightness are joined to form a single composite image.

  The panorama image generation processing unit 82 includes a bit width adjustment unit 104, a color system conversion unit 105, a luminance magnification approximation function calculation unit 106, a luminance adjustment unit 107, and an image composition unit 108.

  The bit width adjusting unit 104 adjusts the bit width of the stitched image or the stitched composite image. For example, the bit width adjustment unit 104 adjusts the bit width of the stitched images so that each pixel of the stitched images is changed from 8 bits to 16 bits, and each pixel of the stitched composite image is 16 bits. The bit width of the combined composite image is adjusted so that the bit is changed to 8 bits.

  The color system conversion unit 105 converts the color system of the stitched image or the stitched composite image. For example, the color system conversion unit 105 converts the RGB color system of the images to be joined into a color system represented by a luminance component and a color component, more specifically, a color system that is YCrCb. Convert. The color system conversion unit 105 converts the color system represented by the luminance component and color component of the combined composite image, more specifically, the color system that is YCrCb into the color system that is RGB. Convert. By converting the color system of the image, it is possible to apply processing focusing on luminance to the image.

  The luminance magnification approximation function calculation unit 106 calculates an approximation function that approximates the change in the magnification of exposure with respect to the angle of the shooting direction of each image from the shooting conditions of each of the plurality of images. For example, the luminance magnification approximate function calculation unit 106 calculates an approximate function that represents a spline curve or an approximate function that is a high-order polynomial. By calculating the approximate function, it is possible to smoothly adjust the change in the exposure magnification.

  The luminance magnification approximation function calculation unit 106 includes an adjustment parameter determination unit 121. The adjustment parameter determination unit 121 determines a parameter for adjusting the luminance of the image based on the parameter stored in the internal memory 67 by the parameter storage control unit 103. For example, the adjustment parameter determination unit 121 determines a parameter that determines the dynamic range of brightness after adjustment from the dynamic range of brightness of a plurality of images.

  The brightness adjusting unit 107 adjusts the brightness of each of the plurality of images so as to continuously change in the stitching direction according to the calculated approximate function.

  The image synthesizing unit 108 synthesizes a plurality of images with adjusted brightness into one synthesized image. For example, the image synthesizing unit 108 combines a plurality of images with adjusted brightness and combines them into one to generate a panoramic image.

  The gamma correction unit 83 applies inverse gamma correction processing to a plurality of images and applies gamma correction processing to the composite image before adjusting the luminance. For example, by applying inverse gamma correction processing to multiple images, it is possible to obtain a captured image as an image to which synthesis processing is applied, and by applying gamma correction processing, The displayed image is displayed as an image closer to the actual color.

  Note that the inverse gamma correction or the gamma correction process may be executed by the image processing unit 66.

  Next, photographing processing by the digital still camera 1 executed when a shutter button (not shown) is pressed will be described.

  FIG. 7 is a flowchart for explaining a photographing process by the digital still camera 1. In step S11, the rotation control unit 101 of the imaging control unit 81 drives the motor drive unit 13 to rotate the lens unit 11 to the front of the main body 12, that is, the lens unit 11 to the center position.

  In step S12, the exposure adjustment unit 102 of the imaging control unit 81 measures the front surface based on the electrical signal corresponding to the image of the front subject supplied from the imaging device of the lens unit 11 to obtain the optimum exposure. decide. For example, the exposure adjustment unit 102 determines the shutter speed, ISO sensitivity, and lens aperture as exposure.

  Note that an exposure meter may be provided in the lens unit 11 separately from the image sensor, and the brightness in the photographing direction may be measured by the exposure meter.

  In step S13, the parameter storage control unit 103 of the imaging control unit 81 stores the exposure condition, which is the exposure condition determined in step S12, in the built-in memory 67. For example, the parameter storage control unit 103 stores the shutter speed, ISO sensitivity, and lens aperture in the built-in memory 67.

  In step S14, the rotation control unit 101 of the imaging control unit 81 drives the motor drive unit 13 to rotate the lens unit 11 in the imaging direction. For example, in the first step S14, the rotation control unit 101 rotates the lens unit 11 in the shooting direction, which is the position of the leftmost angle toward the subject, which captures the image 71.

  As will be described later, step S14 is the number of images to be synthesized, and is repeated by a predetermined number of times, and the rotation control unit 101 takes the motor drive unit so as to capture the image to be synthesized. 13 is driven to rotate the lens unit 11 in the photographing direction. Specifically, the rotation control unit 101 rotates the lens unit 11 in the shooting direction in which the image 72 is shot in the second step S14, that is, the direction in which the image 73 is shot in the third step S14. The lens unit 11 is rotated to the front, and in the fourth step S14, the lens unit 11 is rotated in the shooting direction for shooting the image 74, and in the fifth step S14, the image 75 is shot toward the subject. The lens unit 11 is rotated in the photographing direction which is the position of the rightmost angle.

  In step S <b> 15, the exposure adjustment unit 102 of the shooting control unit 81 measures the brightness in the shooting direction based on the electrical signal that is supplied from the image sensor of the lens unit 11 and that corresponds to the image of the subject.

  In step S16, the exposure adjustment unit 102 of the imaging control unit 81 adjusts the exposure. For example, the exposure adjustment unit 102 adjusts the shutter speed and the ISO sensitivity so as to obtain an optimum exposure based on the brightness measured in step S15 while keeping the lens aperture set to the value determined in step S12.

  In step S17, the exposure adjustment unit 102 of the photographing control unit 81 determines whether or not the difference between the exposure determined in step S12 and the exposure adjusted in step S16 exceeds a preset limit.

  If it is determined in step S17 that the difference between the exposure determined in step S12 and the exposure adjusted in step S16 exceeds a preset limit, the procedure needs to be adjusted, so the procedure goes to step S18. move on.

  In step S18, the exposure adjustment unit 102 of the photographing control unit 81 adjusts the exposure within the limit by adjusting the shutter speed, and the process proceeds to step S19. When the ISO sensitivity is increased, the noise included in the image increases. In step S18, the exposure adjustment unit 102 adjusts the exposure within the limits by adjusting the shutter speed while avoiding so-called gain increase as much as possible. To do.

  If it is determined in step S17 that the difference in exposure does not exceed the preset limit, there is no need to further adjust the exposure, so the process of step S18 is skipped and the process proceeds to step S19.

  In step S19, the photographing control unit 81 causes the lens unit 11 to perform photographing. For example, the photographing control unit 81 causes the lens unit 11 to photograph a subject in the photographing direction of the lens unit 11 with the exposure adjusted in step S16 or step S18.

  In step S <b> 20, the parameter storage control unit 103 of the shooting control unit 81 stores the shooting exposure parameter indicating the shooting exposure in step S <b> 19 in the built-in memory 67. For example, the parameter storage control unit 103 causes the built-in memory 67 to store values indicating shutter speed, ISO sensitivity, lens aperture, and the like in the shooting at step S19.

  In step S21, the shooting control unit 81 determines whether or not a predetermined number of images have been shot. For example, the shooting control unit 81 determines whether or not the number of shots determined by the shooting angle of view α and the horizontal angle of view β of FIG.

  If it is determined in step S21 that the predetermined number of images has not been shot, it is necessary to take images in different shooting directions in order to generate a panoramic image, so the process returns to step S14 and the next shooting is performed. The above-described processing is repeated so as to capture an image of the direction.

  If it is determined in step S21 that a predetermined number of images have been shot, the processing ends because all the images necessary for generating a panoramic image are available.

  In order to combine panoramic images by stitching together the images thus captured, the brightness of the stitched images must be matched. First, the brightness magnification of an image is obtained from the shutter speed, ISO sensitivity, and exposure of each image indicated by the lens aperture when each image is captured. The image luminance magnification refers to the magnification of the image luminance with respect to the luminance of the image whose exposure is adjusted when it is assumed that all images are captured with the same exposure as that of the reference image.

  FIG. 8 is a diagram illustrating the relationship between the position of the image and the luminance magnification.

  In FIG. 8, the horizontal axis indicates the pixel position in the direction of combining the captured images and the vertical axis indicates the luminance magnification of each image. It can be said that the horizontal axis indicates the position of the shooting angle.

  For example, the image 71 and the image 72 are photographed at a shutter speed of 1/60 seconds because the subject is a dark scene, and are bright images. The image 75 is an image captured at an optimal exposure because the subject is a bright scene and is captured at a shutter speed of 1/500 seconds. The image 73 is photographed at a shutter speed of 1/125 seconds because the subject is not as dark as the scenery photographed in the images 71 and 72 and is not as bright as the scenery photographed in the image 75. It is a photographed image. The image 74 was shot at an optimum exposure because the subject was not as dark as the landscape photographed in the image 73 and was not as bright as the landscape photographed in the image 75. It is an image.

  When the image 73 is used as a reference, the length of the exposure period at the shutter speed when the image 71 and the image 72 are taken is 2 as the length of the exposure period at the shutter speed when the image 73 is taken. Since the magnification is double, the luminance magnification of the image 71 and the image 72 is 0.5.

  In addition, since the length of the period exposed at the shutter speed when the image 74 is captured is half the period exposed at the shutter speed when the image 73 is captured, the luminance magnification of the image 74 is 2.0.

  Furthermore, since the length of the period exposed at the shutter speed when the image 75 is captured is a quarter of the period exposed at the shutter speed when the image 73 is captured, The luminance magnification is 4.0.

  In this case, since the image 73 is the reference, the luminance magnification of the image 75 is 1.0.

  By multiplying the brightness of the image by the brightness magnification due to the difference in shutter speed with respect to each of the images 71 to 75 photographed with almost optimum exposure, the photographing conditions of the images 71 to 75 become the same. The images 75 can be joined.

  FIG. 9 is a diagram illustrating an example of a panoramic image synthesized by multiplying each of the images 71 to 75 by multiplying the respective luminance magnifications.

  In FIG. 9, when the image 73 is used as a reference, if the brightness of the image 71 and the image 72 is 0.5 times, the brightness of the image 74 is 2 times, and the brightness of the image 75 is 4 times, the shooting conditions are uniform. Panorama images can be combined. However, when the brightness of the image 71 is increased by 0.5, the dark part of the image 71 is blacked out, and when the brightness of the image 75 is increased by 4, the bright part of the image 75 is overexposed. The maximum value of the magnification by which the luminance of the images 71 to 75 is multiplied is 4.0, the minimum value is 0.5, and 8 times when expressed in the dynamic range. That is, the ratio of the maximum luminance magnification to the minimum luminance magnification is 8.

  Consider the reduction of the luminance ratio in order to prevent blackout and whiteout.

  FIG. 10 is a diagram illustrating an example of an image brightness distribution when shooting is performed with a fixed exposure.

  In FIG. 10, the horizontal axis indicates the position of the shooting angle, that is, the horizontal shooting direction, and the vertical axis indicates the luminance level of the image. Since the luminance is usually expressed by 8 bits, overexposure occurs in an image portion where the luminance is 256 or more. In addition, an image portion with low luminance is buried in noise.

  For example, overexposure occurs in a portion where the luminance indicated by the oblique line rising to the right in FIG. For example, the image is buried in noise at a portion where the luminance indicated by the diagonally downward slanting line in FIG.

  There is a portion of an image having a luminance of 10 or less in the range of the horizontal photographing direction from 1 to 4, and the image of that portion is buried in noise. In addition, there is an image portion with a luminance of 256 or more in the range of the horizontal photographing direction, which is 23 to 24 and 26 to 30, and the image of the portion is overexposed.

  For example, the exposure when the images 71 to 75 are photographed is adjusted so that there are fewer portions where over-exposure occurs or noise is buried.

  FIG. 11 is a diagram illustrating an example of the brightness distribution when the image is divided into five and photographed with optimum exposure in each direction.

  In FIG. 11, a portion where the brightness is 256 or more and overexposure is indicated by a diagonal line rising to the right, and a portion where the luminance is 10 or less and buried in noise is indicated by a diagonal line that descends to the right.

  White square marks (□) indicate the luminance when the range of the horizontal shooting direction, which is 1 to 10, is taken with the luminance magnification set to 10 times that shown in FIG.

  A white circle (◯) indicates the luminance when the horizontal shooting direction range of 6 to 15 is taken with the luminance magnification set to 3 times that shown in FIG.

  A white triangle (Δ) indicates the luminance when the horizontal photographing direction range of 11 to 20 is photographed with a luminance magnification of 1.4 times compared to the case shown in FIG.

  The crosses (x) indicate the luminance when the horizontal shooting direction range of 16 to 25 is taken with the luminance magnification set to 0.75 with respect to the case shown in FIG. The asterisk (*) indicates the luminance when the horizontal photographing direction range of 21 to 30 is photographed with a luminance magnification of 0.45 with respect to the case shown in FIG.

  FIG. 12 is a flowchart for explaining dynamic range compression processing by the digital still camera 1. In this process, by adjusting the dynamic range of the brightness, a panoramic image can be synthesized while maintaining local contrast without causing blackout or overexposure.

  In step S41, the gamma correction unit 83 performs inverse gamma correction on each of the images 71 to 75, for example, to return the relationship between the luminance of the subject and the pixel RGB to linear. For example, normally, a gamma correction is applied to an image captured and stored by the digital still camera 1 so that the image is naturally displayed when displayed and printed. In this state, the actual luminance of the subject and the pixel value have a non-linear relationship. For this reason, for example, when the pixel value is simply doubled when the luminance is doubled, a pixel value indicating the correct luminance cannot be obtained. Therefore, the gamma correction unit 83 restores the once recorded image to the captured state by performing inverse gamma correction on the image before the process of combining the images.

  More precisely, it can be said that the gamma correction unit 83 applies, to the image, a process opposite to the image correction process performed by the digital still camera 1 that is applied when the captured image is stored.

  Further, when the digital still camera 1 records the image data of the photographed image on the external recording medium 52 without applying the gamma correction process to the photographed image, the pixel value of the image based on the recorded image data is Since there is a linear relationship with the luminance of the subject, step S41 is skipped.

  In step S42, the bit width adjustment unit 104 of the panoramic image generation processing unit 82 extends the RGB value, which is the pixel value of each pixel of each image, from 8 bits to 16 bits. For example, the bit width adjustment unit 104 adjusts the bit width of the stitched images so that the 8-bit RGB value of each pixel of the stitched images that are the images 71 to 75 is a 16-bit RGB value. .

  In step S43, the color system conversion unit 105 of the panoramic image generation processing unit 82 separates the RGB values of the pixels of each image into luminance components and color components. For example, the color system conversion unit 105 converts the color system of RGB, which is the pixel value of each pixel of the images 71 to 75, into a color system of YCrCb. As a result, it is possible to apply processing focusing on the luminance component to the image.

  In step S44, the luminance magnification approximation function calculation unit 106 of the panoramic image generation processing unit 82 obtains a dynamic range compression function f (x), which is an example of an approximation function that approximates a change in exposure magnification, from the shooting exposure parameters. .

  That is, first, in step S44, the luminance magnification approximation function calculation unit 106 calculates an approximation function that approximates the change in exposure magnification with respect to the angle in the shooting direction of each image from the conditions for shooting a plurality of images. For example, the luminance magnification approximate function calculation unit 106 obtains an approximate function indicating a spline curve or an approximate function that is a high-order polynomial.

  FIG. 13 is a diagram for explaining the approximate curve k (x). In FIG. 13, the horizontal axis indicates the pixel position in the image pasting direction, and the vertical axis indicates the common logarithmic value of the luminance magnification.

  The luminance magnification approximation function calculation unit 106 determines the luminance magnification k1 of the image 71, the luminance magnification k2 of the image 72, the luminance magnification k3 of the image 73, and the luminance magnification k4 of the image 74 from the respective shooting exposure parameters of the images 71 to 75. And the luminance magnification k5 of the image 75 is calculated. Then, as illustrated in FIG. 13, the luminance magnification approximation function calculation unit 106 associates the luminance magnification k1 of the image 71 with the center position in the left-right direction of the image 71, and sets the luminance magnification k2 of the image 72 to the left and right of the image 72. Corresponding to the center position in the direction, the brightness magnification k3 of the image 73 is correlated to the center position in the left-right direction of the image 73, the brightness magnification k4 in the image 74 is correlated to the center position in the left-right direction of the image 74, The luminance magnification k5 of 75 is associated with the center position of the image 75 in the left-right direction.

  When attention is paid to the image 71 and the luminance component at the position x on the image 71 is represented by P1 (x), it is assumed that the image 71 is shot with the same exposure as the exposure of the reference image 73. The luminance P (x) at the pixel position x in the direction can be expressed by P1 (x) × k1. Focusing on the image 72, if the luminance component at the position x on the image 72 is represented by P2 (x), it is assumed that the image 72 is taken with the same exposure as that of the reference image 73. The luminance P (x) at the pixel position x in the direction can be expressed by P2 (x) × k2. When attention is paid to the image 73 and the luminance component at the position x on the image 73 is represented by P3 (x), the luminance P (x) at the pixel position x in the image pasting direction is P3 (x) × k3. Can be represented.

  Similarly, when attention is paid to the image 74 and the luminance component at the position x on the image 74 is represented by P4 (x), it is assumed that the image 74 is captured with the same exposure as that of the reference image 73. The luminance P (x) at the pixel position x in the pasting direction can be expressed as P4 (x) × k4. When attention is paid to the image 75 and the luminance component at the position x on the image 75 is represented by P5 (x), it is assumed that the image 75 is shot with the same exposure as that of the reference image 73. The luminance P (x) at the pixel position x in the direction can be expressed by P5 (x) × k5.

  The luminance magnification approximation function calculation unit 106 smoothly approximates the luminance P (x) represented by P1 (x) × k1 to the luminance P (x) represented by P5 (x) × k5. Ask for.

  In FIG. 13, a straight line connecting luminances P (x) to P5 (x) × k5 represented by P1 (x) × k1 is indicated by a dotted line, and an approximate curve k (x) Is shown by a dashed line.

  The approximate curve k (x) is a curve that passes through the vicinity of the point indicating the luminance at the center position in the left-right direction of each of the images 71 to 75. This curve may be a spline curve or a curve generated by a high-order polynomial.

  In this way, the approximate curve k (x) is calculated.

  Next, the dynamic range compression function f (x) created using the approximate curve k (x) will be described with reference to FIG.

FIG. 14 is a diagram showing a change in the dynamic range compression function f (x) according to the adjustment parameter m for compression of the luminance dynamic range. When the pixel position in the image pasting direction is x, the dynamic range compression function f (x) is expressed by Expression (1).
f (x) = ((k (x) −1) / m + 1) / k (x) (1)

  In FIG. 14, a black diamond symbol (♦) indicates the value of the dynamic range compression function f (x) obtained by Expression (1) when the adjustment parameter m = 1, (■) indicates the value of the dynamic range compression function f (x) obtained by Equation (1) when the adjustment parameter m = 1.2, and the black triangle (▲) indicates the adjustment parameter. When m = 2, the value of the dynamic range compression function f (x) obtained by Expression (1) is shown, and the black circle (●) indicates the expression (1) when the adjustment parameter m = 100. The value of the dynamic range compression function f (x) obtained by

  In step S45, the luminance adjustment unit 107 of the panoramic image generation processing unit 82 adjusts the luminance level of each image by multiplying the luminance component Pn (x) of each image by (kn / T) × f (x).

The luminance Q (x) at the position x is the image number n, the luminance component at the position x of each image is Pn (x), the luminance magnification with respect to the image number n is kn, and the luminance magnification of the reference image after synthesis is T Then, it is calculated | required by the calculation represented by Formula (2).
Q (x) = Pn (x) × (kn / T) × (((k (x) −1) / m + 1) / k (x))
= Pn (x) × (kn / T) × f (x)
... (2)

  For example, the luminance adjustment unit 107 adjusts the luminance level of the images 71 to 75 by multiplying the luminance component Pn (x) by (kn / T) × f (x), thereby compressing the luminance dynamic range. Q (x) is calculated.

  When the adjustment parameter m = 1, the dynamic range of the luminance of the image is not compressed, and the compression ratio of the dynamic range of the luminance of the image increases as the adjustment parameter m increases.

  Here, compression of the dynamic range of the luminance of the image will be described with reference to FIGS.

  FIG. 15 is a diagram illustrating the characteristics of (kn / T) × f (x) when the adjustment parameter m = 1. (Kn / T) × f (x) is a value for adjusting the luminance multiplied by the luminance of each pixel of each image.

  A white square mark (□) indicates a value of (kn / T) × f (x) that is constant at 0.2 in the range of the position x that is 1 to 10, corresponding to the range of one image. Show.

  A white circle (◯) indicates a value of (kn / T) × f (x) that is constant at 0.5 in the range of position x that is 6 to 15, corresponding to the range of one image. Show.

  A white triangle (Δ) indicates a value of (kn / T) × f (x) that is constant at 1.0 in the range of the position x that is 11 to 20, corresponding to the range of one image. Show.

  The cross mark (×) indicates a value of (kn / T) × f (x) that is constant at 2.1 in the range of position x that is 16 to 25, which corresponds to the range of one image.

  An asterisk (*) indicates a value of (kn / T) × f (x) that is constant at 3.4 in the range of the position x that is 21 to 30 corresponding to the range of one image.

  However, T = k3.

  In this way, when the adjustment parameter m = 1, the value for adjusting the brightness, which is multiplied by the brightness of each pixel of each image, is determined for each image, does not change depending on the position x in the image, and The dynamic range of luminance is not compressed.

  FIG. 16 is a diagram illustrating the characteristics of the luminance Q (x) when the adjustment parameter m = 1.

  As shown in FIG. 16, in the low luminance portion of the image, the image is buried in noise, and in the high luminance portion, the luminance is 256 or more and is saturated.

  For example, the luminance of pixels in the range of position x from 22 to 30 is 256 or more, and overexposure occurs. For example, the luminance of the pixel in the range of position x from 1 to 4 is 10 or less, and the image is buried in noise.

  There is a portion of an image with a luminance of 10 or less in the range of the position x that is 1 to 4, and the image of that portion is buried in noise. Further, there is an image portion having a luminance of 256 or more in the range of the position x that is 22 to 30, and the image of the portion is overexposed.

  The value of the adjustment parameter m is adjusted so that there are fewer such portions where overexposure occurs or where the portion is buried in noise.

  FIG. 17 is a diagram illustrating the characteristics of (kn / T) × f (x) when the adjustment parameter m = 4.

  In FIG. 17, a white square mark (□), a white circle mark (◯), a white triangle mark (Δ), a cross mark (×), and an asterisk (*) represent one image, respectively. (Kn / T) × f (x), which is a value for adjusting the luminance multiplied by the luminance of each pixel of each image in the range, is shown. However, T = k3, and an image in which (kn / T) × f (x) is indicated by a white triangle (Δ) is used as a reference.

  As indicated by the white square mark (□), the luminance of the pixel in the range of the position x that is 1 to 10 in the first image decreases as the value of the position x increases. A value for adjusting the luminance of each pixel in the range of 0.25 is multiplied.

  As indicated by a white circle (◯), the luminance of the pixel in the range of the position x that is 6 to 15 in the second image decreases as the value of the position x increases. A value for adjusting the luminance of each pixel in the range of 0.38 is multiplied.

  Further, as indicated by a white triangle (Δ), the luminance of the pixel in the range of the position x that is 11 to 20 in the third image decreases as the value of the position x increases. A value for adjusting the luminance of each pixel in the range of 19 to 0.50 is multiplied.

  Further, as indicated by a cross (x), the luminance of the pixels in the range of the position x that is 16 to 25 in the fourth image decreases as the value of the position x increases, 1.39 to 0. A value for adjusting the luminance of each pixel in the range of .81 is multiplied.

  As indicated by an asterisk (*), the luminance of the pixels in the range of the position x that is 21 to 30 in the fifth image decreases as the value of the position x increases. A value for adjusting the luminance of each pixel in the range of 19 is multiplied.

  As shown in FIG. 17, when the adjustment parameter m = 4, the dynamic range is compressed at a constant rate.

  FIG. 18 shows the characteristics of the luminance Q (x) when the adjustment parameter m = 4.

  As shown in FIG. 18, it is not buried in noise even when the luminance is low, is not saturated even when the luminance is high, and the high-frequency component of the luminance change that is local contrast is maintained. ing.

  For example, the luminance of the pixel in the range of position x that is 22 to 30 is less than 256, and overexposure does not occur. For example, the luminance of the pixel in the range of position x from 1 to 4 exceeds 10, and the image is not buried in noise.

  In this way, for example, the dynamic ranges of the images 71 to 75 are adjusted so that there is no portion where whiteout occurs or portions which are buried in noise. The adjustment parameter determining unit 121 determines the value of the adjustment parameter m for compressing the dynamic range of the luminance of the image so that there is no portion where overexposure occurs or a portion where noise is buried.

  Note that when the adjustment parameter m is set to a large value such as 10 to 100, the dynamic range of the luminance of the image is further reduced, and the image after panorama synthesis is unnatural, the adjustment parameter m The image quality can be adjusted by decreasing the value of m and decreasing the compression ratio of the dynamic range of the luminance of the image.

Further, the luminance Q (x) may be calculated by the equation (3).
Q (x) = Pn (x) × (kn / T) / k (x) (3)
In this case, the low frequency component of the image is completely removed.

  Returning to FIG. 12, in step S46, the image composition unit 108 of the panorama image generation processing unit 82 performs panorama composition of the images. For example, the image composition unit 108 performs panorama composition of the images 71 to 75 whose luminance dynamic range is compressed.

  FIG. 19 is a diagram illustrating an example of a panoramic image created by stitching together the images 71 to 75 whose luminance has been adjusted.

  In FIG. 19, blackening or whiteout of the image does not occur. Each of the images 71 to 75 is smoothly stitched and synthesized without causing blackout or whiteout of the image.

  In step S47, the color system conversion unit 105 of the panorama image generation processing unit 82 combines the luminance component and the color component of the panorama combined image and converts them into RGB values. More specifically, the color system conversion unit 105 converts the color system of the pixel values of the panorama synthesized image from the color system of YCrCb to the color system of RGB.

  In step S48, the bit width adjustment unit 104 of the panoramic image generation processing unit 82 converts the RGB value of the pixel of the composite image from 16 bits to 8 bits. For example, the bit width adjustment unit 104 combines the bits of the composite image obtained by joining the images 71 to 75 so that the 16-bit pixels of the composite image obtained by joining the images 71 to 75 become 8-bit pixels. Adjust the width. For example, the bit width adjustment unit 104 sets all values exceeding 255 to 255 by saturation processing.

  In step S49, the gamma correction unit 83 performs gamma correction on the composite image, and the process ends. By performing gamma correction, an image photographed and stored by the digital still camera 1 can be naturally seen during display and printing.

  In this way, panorama images can be synthesized while maintaining local contrast of the images while avoiding blackout and overexposure, so that a panoramic image with high appreciation value can be obtained.

  Furthermore, since the brightness is adjusted by a simple calculation, the amount of calculation can be reduced, power consumption can be reduced, and it can be incorporated into a mobile device.

  In this way, when a plurality of images are connected, a panoramic image can be generated. Further, an approximate function that approximates a change in exposure magnification with respect to an angle in the direction of capturing each of the images is calculated from a plurality of image capturing conditions, and continuous in the stitching direction according to the approximate function. When the brightness of the plurality of images is adjusted so as to be changed to one, and the plurality of images with adjusted brightness are connected to form a single composite image, it is easier and more natural panoramic image Can be generated.

  FIG. 20 is a block diagram showing an example of the configuration of a personal computer that executes the above-described series of processing by a program. A CPU (Central Processing Unit) 201 executes various processes according to a program stored in a ROM (Read Only Memory) 202 or a storage unit 208. A RAM (Random Access Memory) 203 appropriately stores programs executed by the CPU 201 and data. The CPU 201, ROM 202, and RAM 203 are connected to each other via a bus 204.

  An input / output interface 205 is also connected to the CPU 201 via the bus 204. Connected to the input / output interface 205 are an input unit 206 composed of a keyboard, mouse, microphone, and the like, and an output unit 207 composed of a display, a speaker, and the like. The CPU 201 executes various processes in response to commands input from the input unit 206. Then, the CPU 201 outputs the processing result to the output unit 207.

  The storage unit 208 connected to the input / output interface 205 includes, for example, a hard disk, and stores programs executed by the CPU 201 and various data. The communication unit 209 communicates with an external device via a network such as the Internet or a local area network.

  Further, a program may be acquired via the communication unit 209 and stored in the storage unit 208.

  The drive 210 connected to the input / output interface 205 drives a removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives programs and data recorded there. Get etc. The acquired program and data are transferred to and stored in the storage unit 208 as necessary.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  As shown in FIG. 20, a program recording medium for storing a program that is installed in a computer and is ready to be executed by the computer includes a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only). Memory), DVD (including Digital Versatile Disc), magneto-optical disk), or removable media 211, which is a package medium made of semiconductor memory, or ROM 202 where programs are temporarily or permanently stored, The storage unit 208 is configured by a hard disk or the like. The program is stored in the program recording medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 209 that is an interface such as a router or a modem as necessary. Done.

  In the present specification, the step of describing the program stored in the program recording medium is not limited to the processing performed in time series in the order described, but is not necessarily performed in time series. Or the process performed separately is also included.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a figure which shows the external appearance of one Embodiment of the digital still camera of this invention. It is a figure which shows the internal structure of a motor drive unit. It is a block diagram which shows the structure of the hardware of a digital still camera. It is a figure which shows the imaging | photography field angle and horizontal field angle of a digital still camera. It is a figure which shows the difference in the brightness of the image by an imaging | photography direction. It is a figure explaining the function implement | achieved by CPU of a control unit which performs a program. It is a flowchart explaining the process of imaging | photography with a digital still camera. It is a figure which shows the relationship between the position of an image, and luminance magnification. It is a figure which shows the example of the panoramic image synthesized by multiplying each of the images by multiplying each luminance magnification. It is a figure which shows an example of the brightness distribution of an image at the time of imaging | photography with fixing exposure. It is a figure which shows an example of the distribution of the brightness | luminance at the time of dividing | segmenting into 5 and image | photographing with the optimal exposure in each direction. It is a flowchart explaining the process of compression of the dynamic range by a digital still camera. It is a figure explaining the approximate curve k (x). It is a figure which shows the change of the function f (x) for dynamic range compression by the parameter m for adjustment of compression of the dynamic range of a brightness | luminance. It is a figure which shows the characteristic of (kn / T) * f (x) in case adjustment parameter m = 1. It is a figure which shows the characteristic of the brightness | luminance Q (x) in case adjustment parameter m = 1. It is a figure which shows the characteristic of (kn / T) * f (x) in case adjustment parameter m = 4. It is a figure which shows the characteristic of the brightness | luminance Q (x) in case adjustment parameter m = 4. It is a figure which shows the example of a panoramic image. And FIG. 11 is a block diagram illustrating an example of a configuration of a personal computer.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 11 Lens unit, 12 Main body, 13 Motor drive unit, 21 Lens, 31 Potentiometer, 32 Motor, 33 Reduction gear, 51 Control unit, 52 External recording medium, 61 CPU, 62 A / D converter, 63 Motor driver, 64 Lens controller, 65 Image sensor control circuit, 66 Image processing unit, 67 Built-in memory, 68 Display, 81 Shooting control unit, 82 Panorama image generation processing unit, 83 Gamma correction unit, 101 Rotation control unit, 102 Exposure adjustment unit, 103 Parameter storage control unit , 104 bit width adjustment unit, 105 color system conversion unit, 106 luminance magnification approximation function calculation unit, 107 luminance adjustment unit, 108 image composition unit, 121 adjustment parameter determination unit, 201 CPU, 20 ROM, 203 RAM, 208 storage unit, 211 a removable media

Claims (6)

In an image processing apparatus that processes a plurality of images shot by changing the angle of the shooting direction,
A calculating means for calculating an approximation function for approximating a change in exposure magnification with respect to an angle of a shooting direction of each of the images from a plurality of shooting conditions of the images;
Adjusting means for adjusting the luminance of the plurality of images so as to continuously change in the joining direction according to the approximation function;
An image processing apparatus comprising: a combining unit configured to combine a plurality of the images adjusted in luminance into one combined image. Determining means for determining a parameter for determining a dynamic range of brightness after adjustment from a plurality of dynamic ranges of brightness of the image;
The image processing apparatus according to claim 1, wherein the adjustment unit adjusts the luminance of the plurality of images so as to continuously change in a stitching direction according to the approximation function and the parameter. A conversion unit that converts the color system of RGB of the plurality of images into a color system represented by a luminance component and a color component;
The image processing apparatus according to claim 1, wherein the adjustment unit adjusts the luminance of the plurality of images by changing a luminance component. The image processing apparatus according to claim 1, further comprising: a correction unit that applies inverse gamma correction processing to the plurality of images and applies gamma correction processing to the composite image before brightness adjustment. In an image processing method for processing a plurality of images taken by changing the angle of the shooting direction,
From an imaging condition of the plurality of images, an approximate function that approximates a change in the magnification of exposure with respect to the angle of the imaging direction of each of the images is calculated,
In accordance with the approximate function, the brightness of the plurality of images is adjusted so as to continuously change in the stitching direction,
An image processing method including a step of combining a plurality of images with adjusted brightness to combine them into one combined image. In a program for causing a computer to perform image processing for processing a plurality of images shot by changing the angle of the shooting direction,
From an imaging condition of the plurality of images, an approximate function that approximates a change in the magnification of exposure with respect to the angle of the imaging direction of each of the images is calculated,
In accordance with the approximate function, the brightness of the plurality of images is adjusted so as to continuously change in the stitching direction,
A program comprising a step of combining a plurality of the images adjusted in brightness into one composite image by joining together.

Images