JP2016040883A - Image processing device, image processing method, image processing system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve image quality of an output image.SOLUTION: An image processing device for performing image processing on image data executes: acquisition processing of acquiring two or more partial images under each exposure condition of two or more exposure conditions including a first exposure condition and a second exposure condition; calculation processing of calculating a connection condition for connecting partial images under the first exposure condition; synthesis processing of creating, with respect to partial images acquired by the acquisition processing, at least two synthesis images of a first synthesis image created from two or more partial images under the first exposure condition on the basis of the connection condition calculated by the calculation processing and a second synthesis image created from two or more partial images under the second exposure condition on the basis of the connection condition; and creation processing of creating an output image on the basis of at least the first synthesis image and the second synthesis image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, an image processing system, and a program.

  Conventionally, a method of creating a panoramic image by combining and combining is known. When creating a panoramic image by combining images, a method is known in which a subject is divided and photographed, and exposure is corrected for each of the divided and photographed images, and then the images are combined (for example, Patent Document 1). reference).

  In addition, a method of synthesizing a plurality of divided images with different exposures is known in order to generate a panoramic image without overexposure and underexposure (see, for example, Patent Document 2).

  However, in the conventional method, since the output image is not generated from the synthesized image synthesized under the same connection condition, the image quality of the output image may be deteriorated.

  One aspect of the present invention aims to improve the quality of an output image.

  An image processing apparatus that performs image processing on image data according to an aspect, wherein an acquisition process of acquiring two or more partial images under each exposure condition of two or more exposure conditions including a first exposure condition and a second exposure condition Obtaining means for performing calculation processing for calculating a joining condition for joining the partial images of the first exposure condition, and a joining condition calculated by the computing means for the partial image obtained by the obtaining means A first composite image generated from two or more partial images of the first exposure condition, and a second composite image generated from two or more partial images of the second exposure condition based on the connection condition. Synthesis means for performing synthesis processing for generating at least two composite images of the composite image, and generation processing for generating an output image based on at least the first composite image and the second composite image Characterized in that it has a generation unit that performs, the.

  The image quality of the output image can be improved.

1 is a cross-sectional view illustrating an example of an overall configuration of an image processing system according to an embodiment of the present invention. It is a block diagram explaining an example of the hardware constitutions of the image processing system which concerns on one Embodiment of this invention. It is the schematic explaining another example of the whole structure of the image processing system which concerns on one Embodiment of this invention. It is a flowchart explaining an example of the whole process by the image processing system which concerns on one Embodiment of this invention. It is a block diagram which shows an example of the calculation process which concerns on one Embodiment of this invention, and a synthetic | combination process. It is a flowchart which shows an example of the calculation process which concerns on one Embodiment of this invention, and a synthetic | combination process. It is a figure explaining an example of the projection relationship of the lens which concerns on one Embodiment of this invention. It is a figure explaining an example of a calculation process of a connection position concerning one embodiment of the present invention, and a calculation result. It is a figure explaining an example of the conversion table correction process based on the calculation result which concerns on one Embodiment of this invention. It is a figure explaining an example of the distortion correction process for synthetic | combination processes based on the conversion table for synthetic | combination processes which concerns on one Embodiment of this invention. It is a figure explaining an example of the connection condition calculation conversion table which concerns on one Embodiment of this invention. It is a figure which shows an example of the synthetic | combination image produced | generated by the distortion correction process for synthetic | combination processes based on the conversion table for synthetic | combination processes which concerns on one Embodiment of this invention, and a synthetic | combination process. It is a figure which shows an example of the production | generation process which concerns on one Embodiment of this invention. It is a figure explaining an example of the effect concerning one embodiment of the present invention. It is a figure explaining an example of the matching process which concerns on one Embodiment of this invention. It is a functional block diagram which shows an example of a function structure of the image processing system which concerns on one Embodiment of this invention.

  Embodiments of the present invention will be described below.

<Outline of image processing system>
FIG. 1 is a cross-sectional view showing an example of the overall configuration of an image processing system according to an embodiment of the present invention.

  The image processing system 10 includes an imaging device 12, a housing 14, and a shutter button 18.

  The imaging device 12 includes, for example, an imaging optical system 20A, an imaging optical system 20B, a solid-state imaging element 22A, and a solid-state imaging element 22B as illustrated.

  The solid-state imaging device 22A and the solid-state imaging device 22B are optical sensors such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

  The image forming optical system 20A and the image forming optical system 20B are, for example, so-called fish-eye lenses of six groups and seven elements. It is desirable that the imaging optical system 20A and the imaging optical system 20B have an imaging range of two optical systems of 360 ° or more. The imaging optical system 20A and the imaging optical system 20B each have an angle of view of 180 ° or more, and more preferably, each of the angle of view is 185 ° or more, or 190 ° or more. It is desirable that there is a range. The overlapping imaging range is a reference for the joining process when joining the images.

  The imaging optical system 20A and the imaging optical system 20B include a prism, a filter, a stop, and the like.

  The solid-state imaging device 22A and the solid-state imaging device 22B correspond to the imaging optical system 20A and the imaging optical system 20B, respectively. For example, as shown in the figure, the solid-state image pickup element 22A and the solid-state image pickup element 22B have positions where the light incident on each imaging optical system is orthogonal to the center of the light-receiving area of the corresponding solid-state image pickup element, It is positioned at a position that becomes a coupling surface of the corresponding lens. Each solid-state imaging device has a light receiving area, converts light incident on the light receiving area into an image signal, and outputs the image signal. In the case of FIG. 1, the imaging optical system 20A, the imaging optical system 20B, the solid-state imaging device 22A, and the solid-state imaging device 22B have the same specifications, and are installed in opposite directions so that their optical axes match. The

  Note that the imaging device is not limited to the case where the solid-state imaging device and the two imaging optical systems are provided. The imaging apparatus may include three or more solid-state imaging elements and an imaging optical system. Hereinafter, a case where the solid-state imaging device and the imaging optical system illustrated in FIG. 1 are two pairs will be described as an example.

  The housing 14 includes components such as a controller for the imaging device 12 and a battery that serves as a power source. The shutter button 18 is a button provided on the housing 14 for performing an operation of releasing the shutter of the imaging device 12.

  The image processing apparatus 100 performs a process of connecting each captured image (hereinafter referred to as a captured image) input from the imaging apparatus 12. The image processing apparatus 100 performs a process of connecting and synthesizing a plurality of captured images to generate a composite image having a solid angle of 4 rad (radian) (hereinafter referred to as an omnidirectional image). The omnidirectional image is an image obtained by photographing all directions that can be seen from the photographing point. Note that the omnidirectional image may be a so-called panorama image in which a horizontal plane is imaged 360 °.

  By matching the scanning directions of the solid-state imaging element 22A and the solid-state imaging element 22B, the image processing apparatus 100 can facilitate each process. In other words, by matching the scanning direction and the order at the places where the joining process is performed, the joining process of the subject of each solid-state imaging device, in particular, the moving object can be facilitated. For example, when the upper left part of the image of the solid-state image sensor 22A and the lower left part of the image of the solid-state image sensor 22B coincide with each other at the place where the joining process is performed, the scanning direction of the solid-state image sensor 22A is changed from the top to the bottom and the right From left to right. When the upper left part of the image of the solid-state image sensor 22A and the lower left part of the image of the solid-state image sensor 22B coincide with each other at the position where the joining process is performed, the scanning direction of the solid-state image sensor 22B is from the lower side and the left side Use the right direction. By matching the scanning direction based on the location where the stitching process is performed, the image processing apparatus 100 can easily perform the stitching process.

  The image processing apparatus 100 performs a process of outputting the processed image to an output device such as a display device or an image forming device, and the output image is output by the output device. The image processing apparatus 100 performs a process of outputting the processed image to a storage medium such as an SD (registered trademark) card or a compact flash (registered trademark).

<Hardware configuration of image processing system>
FIG. 2 is a block diagram illustrating an example of a hardware configuration of the image processing system according to the embodiment of the present invention.

  The image processing system 10 includes an image processing apparatus 100 and a lens barrel unit 102.

  The image processing apparatus 100 includes an image data transfer processing circuit 126 and an SDRAMC (Synchronous Dynamic Random Access Memory Controller) 128.

  The image processing apparatus 100 also includes an ISP (Image Signal Processor) 108A, an ISP 108B, and a DMAC (Direct Memory Access Controller) 110.

  Further, the image processing apparatus 100 includes a memory access arbitration circuit (hereinafter referred to as an ARB MEMC (Arbitration Memory Controller)) 112 and a MEMC (Memory Controller) 114.

  In addition, the image processing apparatus 100 includes a distortion correction / image synthesis processing circuit 118, a DMAC 122, an image processing circuit 124, a CPU (Central Processing Unit) 130, and a resizing processing circuit 132.

  The image processing apparatus 100 includes a JPEG (Joint Photographic Exports Group) processing circuit 134, H.264 processing circuit 136, memory card control processing circuit 140, USB (Universal Serial Bus) processing circuit 146, and peripheral communication processing circuit 150.

  In addition, the image processing apparatus 100 includes an audio unit 152, a serial communication processing circuit 158, an LCD (Liquid Crystal Display) driver 162, and a bridge circuit 168.

  The image processing apparatus 100 is connected to an SDRAM (Synchronous Random Access Memory) 138, a memory card slot 142, and a flash ROM (Read-Only Memory) 144.

  In addition, a USB connector 148, a speaker 154, a microphone 156, a wireless NIC (Network Interface Card) 160, an LCD monitor 164, and a power switch 166 are connected to the image processing apparatus 100.

  The ISP 108A and ISP 108B perform so-called preprocessing such as white balance, gamma correction, distortion correction, and image synthesis on the image data based on the input image signal.

  The DMACs 110 and 122 control processing for the memory. The processing for the memory is, for example, processing for writing image data into the memory or reading image data from the memory.

  The ARBMEMC 112 performs processing for arbitrating processing for accessing each memory.

  The MEMC 114 performs processing for controlling the memory. An SDRAM 116 is connected to the MEMC 114. The SDRAM 116 stores data used for processing when the ISP 108A and the ISP 108B perform processing.

  The distortion correction / image synthesis processing circuit 118 processes the image signals output from the solid-state imaging devices 22A and 22B. The distortion correction / image composition processing circuit 118 performs processing such as distortion correction, for example.

  A triaxial acceleration sensor 120 is connected to the distortion correction / image composition processing circuit 118. The triaxial acceleration sensor 120 outputs information such as the posture and orientation of the image processing system 10. The distortion correction / image synthesis processing circuit 118 may perform so-called top-and-bottom correction processing based on information from the triaxial acceleration sensor 120. The triaxial acceleration sensor 120 may be a gyro sensor or the like. Furthermore, the processing performed by the distortion correction / image synthesis processing circuit 118 is not limited to the top / bottom correction processing. For example, camera shake correction processing or the like may be performed.

  The image processing circuit 124 performs various types of image processing. The image processing circuit 124 includes a resizing processing circuit 132, a JPEG processing circuit, The H.264 processing circuit 136 is controlled to process the image data.

  The image data transfer processing circuit 126 performs processing for transferring the image data output from the image processing circuit 124.

  The SDRAM C 128 controls the SDRAM 138. The SDRAM 138 stores various data such as image data processed by the image processing apparatus 100.

  The CPU 130 performs various processes and controls each hardware.

  The resizing processing circuit 132 performs processing for enlarging or reducing the size of the image data.

  The JPEG processing circuit 134 performs codec processing that compresses and decompresses image data.

  H. The H.264 processing circuit 136 performs codec processing for moving image compression and expansion.

  Note that the codec processing of still images and moving images is JPEG and H.264. The format is not limited to H.264. For example, the codec processing of a still image may be GIF (Graphics Interchange Format), PDF (Portable Document Format), or the like. Further, the codec processing of the still image may be a format that does not perform compression processing such as Microsoft Windows (registered trademark) Bitmap Image or RAW image format. Furthermore, these may be combined and output in a plurality of formats.

  Also, the codec processing of the moving image may be MP4 (MPEG-4 Part 14) or the like. Moreover, the format which does not perform compression processing, such as AVI (Audio Video Interleave), may be sufficient. Furthermore, these may be combined and output in a plurality of formats.

  The memory card control processing circuit 140 controls processing such as reading or writing with respect to the memory card connected to the memory card slot 142. An external storage medium such as a memory card is connected to the memory card slot 142.

  The flash ROM 144 stores a program executed by the CPU 130 and various parameters. The program and various parameters stored in the flash ROM 144 are loaded into the main memory or a register of the CPU 130 and executed by the CPU 130.

  The USB processing circuit 146 controls USB communication with an external device such as a PC (Personal Computer) connected via the USB connector 148.

  The peripheral communication processing circuit 150 controls input by the power switch 166.

  The audio unit 152 controls the speaker 154 and the microphone 156 to control audio input / output. The voice unit 152 controls voice input from the microphone 156. The audio unit 152 controls audio output to the speaker 154.

  The serial communication processing circuit 158 controls serial communication with an external device such as a PC. The serial communication processing circuit 158 controls the connected wireless NIC 160 to communicate with an external device via a network.

  The connection with the external device is not limited to USB communication and serial communication. For example, communication conforming to standards such as IEEE (The Institute of Electrical and Electronics Engineers, Inc.) 1394 and Thunderbolt (registered trademark) may be used. Further, wireless communication such as Bluetooth (registered trademark) or NFC (Near Field Communication) may be used.

  The LCD driver 162 is a circuit that controls the connected LCD monitor 164. The LCD driver 162 controls the image signal output to the LCD monitor 164. The LCD monitor 164 performs various displays based on the output image signal.

  Transmission / reception between the elements may be performed via the bridge circuit 168.

  The lens barrel unit 102 includes imaging optical systems 20A and 20B, and solid-state imaging elements 22A and 22B. The solid-state imaging elements 22A and 22B are controlled by the CPU 130.

  Note that the processing performed by the distortion correction / image synthesis processing circuit 118 is not limited to processing such as distortion correction. The distortion correction / image synthesis processing circuit 118 may pre-process the captured image. The pre-processing may be performed by at least one type of processing such as white balance processing, shading correction, camera shake correction, edge enhancement, and defective pixel correction. The pre-processing includes at least one type of processing such as optical black correction, linear correction, gamma correction, color space conversion processing, and image format change processing. May be. In the preprocessing, processing for deforming the image such as rotation and enlargement / reduction of at least one of the images may be performed. By performing the preprocessing, the image processing apparatus 100 can improve the image quality of the image, or can easily perform the processing in the subsequent processing.

  Note that the processing performed by the distortion correction / image synthesis processing circuit 118 is not limited to the processing performed by the distortion correction / image synthesis processing circuit 118. For example, the image processing circuit 124 or the CPU 130 may perform all or part of the processing by the distortion correction / image synthesis processing circuit 118. Similarly, all or part of the processing by the ISP 108A, ISP 108B, and the image processing circuit 124 may be processed by the distortion correction / image synthesis processing circuit 118, the CPU 130, or the like.

  In addition, each process by a circuit is not restricted to be processed by a circuit. For example, a part or all of each process may be processed by software or firmware.

  Further, the hardware configuration of the image processing apparatus 100 is not limited to the configuration illustrated in FIG. For example, the audio unit 152 or the like is not an essential configuration.

  Further, the image processing apparatus 100 is not limited to being integrated with the lens barrel unit 102. The image processing apparatus 100 and the lens barrel unit 102 may be configured as separate apparatuses.

  FIG. 3 is a schematic diagram illustrating another example of the overall configuration of the image processing system according to the embodiment of the present invention.

  The image processing apparatus 100 is an information processing apparatus such as a so-called smartphone or a PC, and may have a configuration in which the lens barrel unit 102 that performs imaging is a separate apparatus.

<Overall processing by image processing system>
FIG. 4 is a flowchart for explaining an example of overall processing by the image processing system according to the embodiment of the present invention.

  In step S0401, the image processing system 10 performs processing for setting the first exposure condition. The exposure condition is a condition for determining an exposure value related to the imaging optical system and the solid-state imaging device. The exposure condition is a set value based on, for example, an aperture, a shutter speed, a gain, or a combination thereof. The exposure condition is determined based on the light source, the subject, and the like.

  In the following, an example will be described in which the first exposure condition is used as a reference, two other exposure conditions are set, and imaging processing is performed under a total of three conditions. The second exposure condition is an exposure condition in which a so-called plus correction is performed, in which the first exposure condition is changed to underexposure for a white subject or the like. The third exposure condition is an exposure condition in which the first exposure condition is changed for overexposure of a black subject or the like, so-called minus correction is performed. Hereinafter, a case where imaging processing is performed under the above-described first exposure condition, second exposure condition, and third exposure condition will be described as an example.

  Steps S0402 to S0405 are examples of acquisition processing for acquiring a captured image under the first exposure condition.

  In step S0402, the image processing system 10 performs the imaging process of the first captured image by the solid-state imaging element 22A. In step S0402, the image processing apparatus 100 performs processing for causing the imaging device 12 in FIG. 1 to capture an image of the first exposure condition set in step S0401.

  In step S0403, the image processing system 10 performs preprocessing on the first captured image. In step S0403, when preprocessing is performed, the image processing apparatus 100 inputs the first captured image captured in step S0402 and performs preprocessing. That is, when preprocessing is performed, the image processing apparatus 100 performs preprocessing on the first captured image and acquires the first partial image.

  In step S0404, the image processing system 10 performs the imaging process of the second captured image by the solid-state imaging element 22B. In step S0404, the image processing apparatus 100 performs processing for causing the imaging device 12 in FIG. 1 to capture an image of the first exposure condition set in step S0401.

  In step S0405, the image processing system 10 performs preprocessing on the second captured image. In step S0405, when performing preprocessing, the image processing apparatus 100 inputs the second captured image captured in step S0404 and performs preprocessing. That is, when preprocessing is performed, the image processing apparatus 100 performs preprocessing on the second captured image and acquires a second partial image.

  The processes in step S402 and step S0403 are an example of an acquisition process for acquiring a first partial image that is necessary in the synthesis process for creating a subsequent-stage composite image. The processes in step S404 and step S0405 are an example of an acquisition process for acquiring a second partial image that is necessary in the synthesis process for creating a subsequent-stage composite image. Note that when the image processing system 10 performs synthesis processing using three or more partial images, the image processing system 10 repeatedly performs the process of step S0402. When the preprocessing is performed, the process of step S0403 is repeated as in the first partial image.

  In step S0406, the image processing system 10 performs a calculation process for calculating a connection condition between the first partial image and the second partial image.

  In step S0407, the image processing system 10 performs a composition process for generating a first composite image.

  FIG. 5 is a block diagram illustrating an example of calculation processing and synthesis processing according to an embodiment of the present invention.

  FIG. 6 is a flowchart illustrating an example of calculation processing and synthesis processing according to an embodiment of the present invention.

  The image processing apparatus 100 combines at least two partial images to generate a combined image. Hereinafter, a case where the first composite image is generated from the first partial image and the second partial image will be described as an example.

  In step S0501, the image processing apparatus 100 performs a connection condition calculation distortion correction process on the first partial image and the second partial image using the connection condition calculation conversion table. In steps S0406 and S0407 in FIG. 4, the first partial image D1 is the first partial image acquired by the acquisition processing in steps S0402 to S0403 in FIG. The second partial image D2 in steps S0406 and S0407 in FIG. 4 is the second partial image acquired by the acquisition processing in steps S0404 to S0405 in FIG. In step S0501, the image processing apparatus 100 performs a process of generating a first connection condition calculation correction image D3 by performing a connection condition calculation distortion correction process on the first partial image D1. In step S0501, the image processing apparatus 100 performs processing to generate a second connection condition calculation correction image D4 by performing a connection condition calculation distortion correction process on the second partial image D2. The connection condition calculation distortion correction process is a process of correcting an image using the connection condition calculation conversion table D6. The connection condition calculation conversion table D6 is data of a conversion table generated based on the design data of the lenses of the imaging optical systems 20A and 20B in FIG. The conversion table is data in which coordinates after conversion are stored in correspondence with coordinates before conversion. The process using the table is a process of rearranging the pixels input to the coordinates before conversion to the position of the coordinates after conversion.

  FIG. 7 is a diagram for explaining an example of the projection relationship of lenses according to an embodiment of the present invention.

  FIG. 7 shows a case where the lenses of the imaging optical systems 20A and 20B in FIG. 1 are lenses for imaging a wide range of angles, so-called fisheye lenses, and the like.

  FIG. 7A is a cross-sectional view illustrating an example of an incident angle λ to a lens according to an embodiment of the present invention.

  FIG. 7B is a diagram illustrating an example of the image height h according to an embodiment of the present invention.

  When light is incident on the lenses of the imaging optical systems 20A and 20B in FIG. 1 illustrated in FIG. 7A at an incident angle λ, the incident light is at the position of the image height h illustrated in FIG. 7B. To form an image. The solid-state image sensor 22A and the solid-state image sensor 22B in FIG. 1 photoelectrically convert the imaged light to capture a partial image. As shown in FIG. 7B, the picked-up image has distortion aberration particularly when picked up with a wide-angle lens such as a fisheye lens. The connection condition calculation distortion correction process is a process for correcting distortion such as distortion.

  In step S0502, the image processing apparatus 100 performs a calculation process for obtaining a connection condition using an overlapping area of the first connection condition calculation correction image and the second connection condition calculation correction image.

  The connection condition is that, during the synthesis process, the first partial correction image D8 obtained by performing the distortion correction process for the first partial image D1 and the second partial image D2 obtained by performing the distortion correction process for the synthesis process on the second partial image D2. The connecting position for connecting the corrected image for composite processing D9. The joining position is calculated using a subject or the like imaged in a range where the first joining condition calculation correction image D3 and the second joining condition calculation correction image D4 are imaged in an overlapping manner.

  FIG. 8 is a diagram for explaining an example of a connection position calculation process and a calculation result according to an embodiment of the present invention.

  FIG. 8A is a diagram illustrating an example of a first connection condition calculation correction image D3 according to an embodiment of the present invention.

  FIG. 8B is a diagram illustrating an example of a second connection condition calculation correction image D4 according to an embodiment of the present invention.

  Hereinafter, as shown in the drawing, a case will be described as an example where the subject TG in the overlapping region is imaged in the first connection condition calculation correction image D3 and the second connection condition calculation correction image D4.

  When the subject TG in the overlapping region is captured in the first connection condition calculation correction image D3, the subject TG in the overlapping region is similarly captured in the second connection condition calculation correction image D4. Yes. The image processing apparatus 100 detects at which position of the second connection condition calculation correction image D4 the subject TG in the overlapping region is imaged by matching processing or the like in order to calculate the shift amount (Δθ, Δφ). To do. Details of the matching process will be described later. As illustrated in FIG. 8B, the image processing apparatus 100 calculates the shift amounts (Δθ, Δφ) for the coordinates of the pixel indicated by “☆” in the first connection condition calculation correction image D3. .

  The shift amounts (Δθ, Δφ) are obtained by moving the arbitrary coordinates (θ, φ) of the first connection condition calculation correction image D3 by the shift amounts (Δθ, Δφ) and the second connection condition calculation correction image D4. The matching correction amount is indicated. The image processing apparatus 100 outputs the shift amounts (Δθ, Δφ) as the calculation result data D5 in FIG.

  In step S0503, the image processing apparatus 100 performs a conversion table correction process for correcting the connection condition calculation conversion table so that the images are aligned on the spherical coordinates based on the result of the calculation process.

  The conversion table correction process includes a process of adding the shift amount (Δθ, Δφ) to the connection condition calculation conversion table D6 when the calculation result data D5 is the shift amount (Δθ, Δφ).

  FIG. 9 is a diagram illustrating an example of a conversion table correction process based on the calculation result according to an embodiment of the present invention.

  FIG. 9A is a diagram illustrating an example of a conversion table before correction in the conversion table correction process based on the calculation result according to an embodiment of the present invention.

  FIG. 9B is a diagram for explaining an example of the conversion table after the correction of the conversion table correction process based on the calculation result according to the embodiment of the present invention.

  When the connection condition calculation conversion table D6 of FIG. 5 is FIG. 9A, the conversion table correction processing corrects the values of the connection condition calculation conversion table D6 of FIG. 5 based on the deviation amounts (Δθ, Δφ). This is a process of generating the corrected connection condition calculation conversion table D62.

  In step S0504, the image processing apparatus 100 performs conversion table generation processing for performing rotation conversion on the corrected conversion table to convert coordinates and generating a conversion table for synthesis processing. In step S0504, the image processing apparatus 100 performs a process of generating the composition processing conversion table D7 based on the corrected connection condition calculation conversion table D62.

  In step S0505, the image processing apparatus 100 performs the distortion correction process for the synthesis process on the first partial image and the second partial image using the synthesis process conversion table.

  In step S0505, the image processing apparatus 100 performs processing for correcting the distortion of the first partial image D1 based on the conversion table for combination processing D7 and generating the first correction image for combination processing D8. In step S0505, the image processing apparatus 100 performs a distortion correction process on the second partial image D2 based on the synthesis process conversion table D7 to generate a second synthesis process correction image D9.

  In step S0506, the image processing apparatus 100 performs a combining process for generating a composite image by combining the first composite processing correction image and the second composite processing correction image generated in the composite processing distortion correction process. Do. In step S0506, the image processing apparatus 100 performs a process of generating a composite image D10 by a composite process of combining the first composite processing correction image D8 and the second composite processing correction image D9. The composite image D10 is an omnidirectional image.

  FIG. 10 is a diagram for explaining an example of the distortion correction process for the synthesis process and the synthesis process based on the conversion table for the synthesis process according to the embodiment of the present invention.

  FIG. 10A is a diagram illustrating an example of the conversion table for combination processing D7.

  FIG. 10B is a diagram illustrating an example of the synthesis process based on the synthesis process conversion table D7.

  As shown in the figure, the composition processing conversion table D7 is a conversion table for processing to generate a composite image D10 by mapping the first composite processing correction image D8 and the second composite processing correction image D9. .

  The composite image D10 shown in FIG. 10 is generated by mapping the central portions of the first composite processing correction image D8 and the second composite processing correction image D9 in the vicinity of the equator with little distortion. Is the case.

  FIG. 11 is a diagram for explaining an example of a connection condition calculation conversion table according to an embodiment of the present invention.

  As shown in FIG. 11, the connection condition calculation conversion table D6 used for the calculation process in step S0502 is a connection table calculation process that converts the calculation process so that the calculation process is performed at coordinates with less distortion. The accuracy can be improved. As shown in FIG. 10B, the conversion table for combination processing D7 is a conversion table in which the central portion of each partial image is mapped to the vicinity of the equator where the composite image D10 is less distorted. Can be generated.

  Further, the composition processing conversion table D7 is generated by performing rotation conversion or the like, so that it is possible to align the case where the top and bottom of the partial images are inverted.

  FIG. 12 is a diagram illustrating an example of a distortion correction process for synthesis processing based on a conversion table for synthesis processing according to an embodiment of the present invention, and a synthesized image generated by the synthesis processing.

  FIG. 12A is a diagram showing an example of the first partial image D1 according to the embodiment of the present invention.

  FIG. 12B is a diagram showing an example of the second partial image D2 according to the embodiment of the present invention.

  FIG. 12C shows the first partial image D1 of FIG. 12A and the second partial image D2 of FIG. 12B in the composition processing conversion table D7 according to an embodiment of the present invention. It is a figure which shows an example of the synthetic | combination process distortion correction process based on this, and the synthesized image D10 produced | generated by the synthesis process.

  As shown in the figure, when the partial images in FIGS. 12A and 12B whose top and bottom are mutually inverted are rotationally converted, the top and bottom when generating the composite image in FIG. You can align the direction.

  Here, returning to FIG.

  In step S0408, the image processing system 10 performs processing for setting the second exposure condition.

  In step S0409, the image processing system 10 performs an acquisition process for acquiring the third partial image and the fourth partial image. In step S0409, the image processing system 10 performs the acquisition process of steps S0402 to S0405 under the second exposure condition. The third partial image is a first captured image captured under the second exposure condition. The third partial image is an image obtained by performing the preprocessing of Step S0403 on the first captured image captured under the second exposure condition when the preprocessing is performed. The fourth partial image is a second captured image captured under the second exposure condition. The fourth partial image is an image obtained by performing the preprocessing in step S0405 on the second captured image captured under the second exposure condition when the preprocessing is performed.

  In step S0410, the image processing system 10 performs a composition process for generating a second composite image under the same connection condition as the first composite image. In step S0410, the image processing system 10 performs the synthesis process similar to that in step S0407 using the connection condition obtained in step S0406, and generates a second synthesized image. That is, the image processing system 10 does not perform the calculation process based on the third partial image and the fourth partial image acquired in step S0409, and the first partial image obtained in step S0406 and the second partial image. A second composite image is generated under the condition for joining the partial images.

  In step S0411, the image processing system 10 performs processing for setting the third exposure condition.

  In step S0412, the image processing system 10 performs an acquisition process for acquiring the fifth partial image and the sixth partial image. In step S0412, the image processing system 10 performs the acquisition processing of steps S0402 to S0405 under the third exposure condition. The fifth partial image is a first captured image captured under the third exposure condition. The fifth partial image is an image obtained by performing the preprocessing in step S0403 on the first captured image captured under the third exposure condition when the preprocessing is performed. The sixth partial image is a second captured image captured under the third exposure condition. The sixth partial image is an image obtained by performing the preprocessing of Step S0405 on the second captured image captured under the third exposure condition when the preprocessing is performed.

  In step S0413, the image processing system 10 performs a composition process for generating a third composite image under the same connection conditions as the first composite image. In step S0413, similarly to step S0410, the image processing system 10 performs the synthesis process under the connection condition obtained in step S0406 to generate a third synthesized image. That is, the image processing system 10 does not perform the calculation process based on the fifth partial image and the sixth partial image acquired in step S0412, and the first partial image calculated in step S0406 and the second partial image A third composite image is generated under the connection condition of the partial images.

  In step S0414, the image processing system 10 performs generation processing for generating an output image from the first composite image, the second composite image, and the third composite image. In step S0414, the image processing system 10 outputs an output image from the first composite image generated in step S0407, the second composite image generated in step S0410, and the third composite image generated in step S0413. Generate.

  FIG. 13 is a diagram illustrating an example of a generation process according to an embodiment of the present invention.

  FIG. 13 is an example of a histogram indicating the brightness of the pixels included in the captured image. In the histogram, the horizontal axis indicates brightness and the vertical axis indicates the number of pixels.

  FIG. 13A is a diagram illustrating an example of the dynamic range DR1 of the first exposure condition.

  As shown in the drawing, an example of a case where the exposure range that can be captured by the solid-state imaging device and reproduced in an image, that is, a so-called dynamic range is the dynamic range DR1 of the first exposure condition will be described as an example. As shown in the figure, the dynamic range DR1 of the first exposure condition has a narrow dynamic range in order to show all the brightness of the pixels included in the captured image. That is, in the dynamic range DR1 of one first exposure condition, there is a case where the dynamic range is narrow to indicate the brightness of the pixels included in the captured image. When the dynamic range is narrow, the image may have a so-called blackout pixel in a dark part or a so-called whiteout part in a bright part.

  FIG. 13B is a diagram illustrating an example of a generation process that generates an output image from the first composite image, the second composite image, and the third composite image.

  When the second exposure condition is an exposure condition that is positively corrected from the first exposure condition, the dynamic range is, for example, the dynamic range DR2 of the second exposure condition as illustrated. The dynamic range DR2 of the second exposure condition is a dynamic range in which a bright part can be imaged as compared with the dynamic range DR1 of the first exposure condition. That is, the second composite image generated from the captured image captured under the second exposure condition can show a brighter portion as compared to the first composite image.

  When the third exposure condition is an exposure condition in which the first exposure condition is negatively corrected, the dynamic range is, for example, the dynamic range DR3 of the third exposure condition as illustrated. The dynamic range DR3 of the third exposure condition is a dynamic range in which a dark part can be imaged as compared with the dynamic range DR1 of the first exposure condition. That is, the third composite image generated from the captured image captured under the third exposure condition can show a darker portion than the first composite image.

  Each exposure condition may be set so that there is no range where the dynamic ranges partially overlap as shown in the figure.

  The generation process is a process for generating an output image by combining, for example, a first composite image, a second composite image, and a third composite image. As illustrated, the dynamic range of the output image is the dynamic range DR4 of the output image indicated by the dynamic ranges of the first composite image, the second composite image, and the third composite image. Since the output image is generated by combining the first composite image, the second composite image, and the third composite image, the dynamic range DR4 of the output image has the first composite image, the second composite image, and It can be wider than the dynamic range of the third composite image. By the generation processing, the image processing apparatus 100 can generate a so-called HDR (High Dynamic Range) output image having a wider dynamic range than the dynamic range of the solid-state imaging device.

  In the above description, the generation process is described when the imaging apparatus and the subject do not move. However, when the imaging apparatus and the subject move, the generation process is realized by a method shown in, for example, the special table 2006-525747. .

  Note that the generation process is not limited to the process of generating from the three composite images. The generation process may be a process that uses at least two composite images. That is, the generation process may be a process of generating from two composite images. Further, the generation process may be a process of generating from four or more composite images. By generating from four or more combined images, an output image with a wider dynamic range can be generated.

  FIG. 14 is a diagram for explaining an example of the effect according to the embodiment of the present invention.

  FIG. 14A is a diagram illustrating an example of an imaging state according to an embodiment of the present invention.

  Hereinafter, the case where the imaging range of each partial image and the overlapping area are illustrated in FIG. 14A will be described as an example. FIG. 14A shows an example in which the imaging range of each partial image and the subject TG in the overlapping region in the overlapping region are imaged as the subject. The subject TG in the overlapping area may be a moving subject such as moving from the position of the subject TGN at the short distance position to the position of the subject TGF at the long distance position. In the case where the subject TG in the overlapping region is a moving subject, in each partial image, the imaged position changes according to the distance of the subject TG in the overlapping region.

  FIG. 14B is a view for explaining an example of an imaging state of the subject TGN at a short distance according to an embodiment of the present invention.

  FIG. 14C is a view for explaining an example of an imaging state of a subject TGF at a long distance position according to an embodiment of the present invention.

  When the subject TG in the overlapping area is the position of the subject TGN at the short distance position, the subject TG in the overlapping area is imaged on the outer portion of each partial image as illustrated in FIG. When the subject TG in the overlapping area is the position of the subject TGF at the long distance position, the subject TG in the overlapping area is imaged in the inner part of each partial image as illustrated in FIG. That is, the position where the subject TG in the overlapping area is imaged changes due to a change in distance due to the movement of the subject TG in the overlapping area.

  Here, in the calculation process of step S0406 in FIG. 4, the connection condition is calculated by the matching process described in FIG. 8, for example.

  FIG. 15 is a diagram illustrating an example of matching processing according to an embodiment of the present invention.

  FIG. 15A is a diagram illustrating an example in which a subject TGF at a long distance position and a subject TGN at a short distance position are captured in the same partial image.

  In FIG. 15A, the long-distance subject TGF and the short-distance subject TGN, that is, the distant subject and the near subject are simultaneously imaged in the overlapping area, and both subjects are shown in FIG. , The first partial image and the second partial image.

  The matching process is a process for extracting, for example, a feature point of a subject shown in an overlapping area in one partial image and calculating where the extracted feature point is located in the other partial image. For example, when the subject TGF at a long distance is extracted with the feature points SP1 to SP3 in the first partial image, the matching process determines where the feature points SP1 to SP3 are located in the second partial image. This is a calculation process. Similarly, when the feature point SP4 to SP7 is extracted from the first partial image of the subject TGN at the short distance position, the matching process calculates where the feature points SP4 to SP7 are located in the second partial image. It is processing to do.

  The connection condition is, for example, a connection position where the partial images are connected such that the feature points SP1 to SP3 of the first partial image overlap with the feature points SP1 to SP3 of the second partial image. Each partial image is connected by being deformed for each whole area or mesh area so that the extracted feature points overlap each other. Note that the connection condition may be a shift amount indicating a difference in coordinates connecting the partial images as described in FIG.

  FIG. 15B is a diagram illustrating an example of a composite image generated by combining partial images when a subject TGF at a long distance position and a subject TGN at a short distance position are captured in the same partial image.

  When the combination processing described with reference to FIG. 12 is performed using the partial images illustrated in FIG. 15A, the combined image illustrated in FIG. 15B is generated.

  As illustrated in FIG. 15A, when the distance or the like of the subject to be subjected to the matching process changes, the position where the subject is imaged changes. Therefore, even if the subject TGF at the long distance position and the subject TGN at the short distance position are similar, the result of the matching process, that is, the connection condition changes depending on the position where the subject is imaged. In addition, when deformation or the like is performed during the matching process, the peripheral portion of the subject is often deformed together. Therefore, the result of the matching process may change due to deformation or the like.

  When imaging is performed under the first exposure condition and the second exposure condition, each exposure condition is set, and imaging is performed by continuous shooting or the like. That is, in the case of captured images with different exposure conditions, the frames in which each captured image is captured are often different. When the captured frames are different from each other, the captured images are captured even when the same subject is captured, or when the subject or the image processing system 10 is moving, when the subject is captured differently as described with reference to FIG. There is. If the captured frames are different, a new object may enter the overlap area between the previous frame and the next frame in the captured image, and a new subject may be added.

  When the subject is imaged differently or when a new subject is added, the result of the matching process often changes due to a change in the feature point extraction result or the like. When the connection condition changes, the composite image generated by the composite process often generates a different composite image even when processed based on the same partial image. A composite image generated under different connection conditions may cause inconsistency in the generation processing described with reference to FIG. 13 such that the same subject is not captured at the same position. The image processing apparatus 100 generates the first composite image and the second composite image under the same connection condition, thereby reducing inconsistencies during the generation process for generating the output image.

<Functional configuration>
FIG. 16 is a functional block diagram illustrating an example of a functional configuration of an image processing system according to an embodiment of the present invention.

  The image processing system 10 includes an imaging unit 10F1 and an image processing device 100. The image processing apparatus 100 includes an acquisition unit 100F1, a calculation unit 100F2, a synthesis unit 100F3, a generation unit 100F4, and an exposure condition setting unit 100F5.

  The imaging unit 10F1 includes a first imaging unit 10F11 and a second imaging unit 10F12.

  The first imaging unit 10F11 performs an imaging process for capturing a first captured image under the exposure condition set by the exposure condition setting unit 100F5. The second imaging unit 10F12 performs an imaging process for capturing a second captured image under the exposure condition set by the exposure condition setting unit 100F5. The first imaging unit 10F11 is realized by, for example, the solid-state imaging device 22A and the imaging optical system 20A shown in FIG. The second imaging unit 10F12 is realized by the solid-state imaging device 22B and the imaging optical system 20B in FIG.

  Note that the imaging unit 10F1 may capture three or more captured images. The imaging unit 10F1 may capture each captured image under three or more exposure conditions.

  The acquisition unit 100F1 performs an acquisition process for acquiring a partial image. When performing the preprocessing, the acquisition unit 100F1 may perform a process of preprocessing the captured image and acquiring the preprocessed image as a partial image.

  When the exposure condition setting unit 100F5 sets the first exposure condition, the first captured image is acquired as the first partial image. When performing the preprocessing, the acquisition unit 100F1 may perform a process for preprocessing the first captured image and acquiring the first captured image as the first partial image. When the exposure condition setting unit 100F5 sets the first exposure condition, the second captured image is acquired as the second partial image. When performing the preprocessing, the acquisition unit 100F1 may perform a process for preprocessing the second captured image and acquiring the second captured image as a second partial image.

  When the exposure condition setting unit 100F5 sets the second exposure condition, a process of acquiring the first captured image as the third partial image is performed. When performing preprocessing, the acquisition unit 100F1 may perform processing for preprocessing the first captured image and acquiring the first captured image as a third partial image. When the exposure condition setting unit 100F5 sets the second exposure condition, a process of acquiring the second captured image as the fourth partial image is performed. When performing the preprocessing, the acquisition unit 100F1 may perform a process for preprocessing the second captured image and acquiring the second captured image as a fourth partial image.

  The acquisition unit 100F1 is realized by the ISP 108A in FIG. 2, the ISP 108B in FIG.

  The calculation unit 100F2 performs a calculation process for obtaining a connection condition for connecting the first partial image and the second partial image. The calculation unit 100F2 performs calculation processing using the first partial image and the second partial image among the partial images acquired by the acquisition unit 100F1 in the acquisition processing. The calculation unit 100F2 performs a process of calculating a connection condition necessary for generating a composite image from each partial image by the composite process.

  The calculation unit 100F2 is realized by the CPU 130, the image processing circuit 124, and the like in FIG.

  The synthesizing unit 100F3 performs a synthesizing process for synthesizing the first partial image and the second partial image based on the connection condition calculated by the calculating unit 100F2 to generate a first synthesized image. The synthesizing unit 100F3 performs a synthesizing process of synthesizing the third partial image and the fourth partial image based on the connection condition calculated by the calculating unit 100F2 to generate a second synthetic image. The synthesizing unit 100F3 performs a synthesizing process for generating a first synthesized image by synthesizing the first partial image calculated by the calculating unit 100F2 and a connection condition for connecting the second partial image. Similar to the first combined image, the combining unit 100F3 generates a second combined image by combining the first partial image calculated by the calculating unit 100F2 and the connection condition for connecting the second partial image. Perform synthesis processing.

  The composition unit 100F3 connects the first partial image and the second partial image even when the partial image to be used is the third partial image and the fourth partial image when the composite image is generated by the composition processing. A synthesis process is performed based on the connection condition.

  Note that the combining unit 100F3 may generate three or more combined images. Even when three or more composite images are generated, the compositing unit 100F3 performs the compositing process based on the connection condition for connecting the first partial image and the second partial image in common.

  The synthesizing unit 100F3 is realized by the CPU 130, the image processing circuit 124, and the like in FIG.

  The generation unit 100F4 performs generation processing for generating an output image based on at least the first composite image and the second composite image. The generation unit 100F4 performs a process of generating an output image based on two or more combined images.

  Note that the generation unit 100F4 may generate an output image based on three or more combined images.

  The generation unit 100F4 is realized by the CPU 130, the image processing circuit 124, and the like in FIG.

  The exposure condition setting unit 100F5 performs a process of setting exposure conditions for the imaging unit 10F1 to perform imaging processing. The exposure condition setting unit 100F5 is realized by the CPU 130 in FIG. The exposure condition setting unit 100F5 may perform a process of calculating the brightness of the captured image in order to set the exposure condition. In addition, each exposure condition may be realized by a user or the like previously inputting respective setting values such as an aperture, a shutter speed, and a gain for each exposure condition, and setting the input setting values.

  The image processing apparatus 100 obtains, by calculation processing, a connection condition for generating a first composite image from the first partial image based on the imaging under the first exposure condition and the second partial image by combining processing. The image processing apparatus 100 generates a first combined image based on the connection condition in the combining process based on the connection condition calculated in the calculation process.

  Since the image processing apparatus 100 generates a second composite image from the third partial image and the fourth partial image based on the imaging of the second exposure condition, the first partial image and the second partial image A second synthesized image is generated by the synthesis process based on the connection condition.

  The first synthesized image and the second synthesized image are generated by the synthesis process based on the same connection condition. The image processing apparatus 100 reduces the influence of the change in the matching processing result because the connecting condition is the same when the captured frames of the first synthesized image and the partial image for generating the second synthesized image are different. it can. Therefore, the image processing apparatus 100 generates the first composite image and the second composite image under the same connection condition, thereby reducing inconsistencies during the generation process for generating the output image. The image quality of the generated output image can be improved.

  All or part of each process is realized by a program that can be executed by a computer written in a legacy programming language such as an assembler, C, C ++, C #, and Java (registered trademark) or an object-oriented programming language. May be. The program can be stored and distributed in a storage medium such as ROM or EEPROM (Electrically Erasable Programmable ROM). The program can be stored and distributed in a storage medium such as an EPROM (Erasable Programmable ROM). The program can be stored and distributed in a storage medium such as a flash memory, a flexible disk, a CD-ROM, a CD-RW, a DVD-ROM, a DVD-RAM, or a DVD-RW. The program can be stored in a device-readable storage medium such as a Blu-ray disc, SD (registered trademark) card, or MO, or can be distributed through a telecommunication line.

  Note that the image in the embodiment is not limited to a still image. For example, the image may be a moving image.

  Moreover, a part or all of each processing of the embodiment may be realized by a programmable device (PD) such as a field programmable gate array (FPGA). Furthermore, a part or all of each process of the embodiment may be realized by an ASIC (Application Specific Integrated Circuit).

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the present invention described in the claims. Can be changed.

DESCRIPTION OF SYMBOLS 10 Image processing system 10F1 Image pick-up part 10F11 First image pick-up part 10F12 Second image pick-up part 12 Image pick-up device 14 Case 18 Shutter button 20A, 20B Imaging optical system 22A, 22B Solid-state image pick-up element 100 Image processing apparatus 100F1 Acquisition part 100F2 Calculation part 100F3 Composition unit 100F4 Generation unit 100F5 Exposure condition setting unit 102 Lens barrel unit 126 Image data transfer processing circuit 128 SDRAMC
108A, 108B ISP
110, 122 DMAC
112 ARBMEMC
114 MEMC
118 Distortion Correction / Image Synthesis Processing Circuit 124 Image Processing Circuit 130 CPU
132 Resize processing circuit 134 JPEG processing circuit 136 H.264 processing circuit 140 Memory card control processing circuit 146 USB processing circuit 150 Peripheral communication processing circuit 152 Audio unit 158 Serial communication processing circuit 162 LCD driver 168 Bridge circuit 138 SDRAM
142 Memory card slot 144 Flash ROM
148 USB connector 154 Speaker 156 Microphone 160 Wireless NIC
164 LCD monitor 166 Power switch D1 First partial image D2 Second partial image D3 First connection condition calculation correction image D4 Second connection condition calculation correction image D5 Calculation result data D6 Connection condition calculation conversion table D62 Modified connection condition calculation conversion table D7 Composite processing conversion table D8 First composite processing correction image D9 Second composite processing correction image D10 Composite image DR1 Dynamic range DR2 of the first exposure condition Dynamic of the second exposure condition Range DR3 Dynamic range DR3 of the third exposure condition Dynamic range TG of the output image Subject TGF in the overlapping area Subject TGN in the long distance position Subject SP1, SP2, SP3, SP4, SP5, SP6, SP7 in the short distance position

Japanese Patent Laid-Open No. 11-205648 JP 2012-80432 A

Claims (9)

An image processing apparatus for image processing image data,
Acquisition means for performing an acquisition process of acquiring two or more partial images under each exposure condition of two or more exposure conditions including a first exposure condition and a second exposure condition;
Calculation means for calculating a connection condition for connecting the partial images of the first exposure condition;
For the partial image acquired by the acquisition unit, based on the connection condition calculated by the calculation unit, the first composite image generated from two or more partial images of the first exposure condition, and the connection condition And a combining means for performing a combining process for generating at least two combined images of the second combined image generated from two or more partial images of the second exposure condition,
An image processing apparatus comprising: generation means for performing generation processing for generating an output image based on at least the first composite image and the second composite image.   The image processing apparatus according to claim 1, wherein the exposure condition is a condition including at least one set value of an aperture, a shutter speed, and a gain when capturing a partial image acquired in the acquisition process. .   The image processing apparatus according to claim 1, wherein the combining process is a process of generating the combined image from three or more partial images.   The image processing apparatus according to claim 1, wherein the generation process is a process of generating the output image based on three or more combined images.   The image processing apparatus according to claim 1, wherein the generation process is a process of generating the output image having a dynamic range wider than that of the first composite image or the second composite image.   The image processing apparatus according to claim 1, wherein the connection condition is a position where one partial image and the other partial image are connected or a shift amount in the composition processing. An image processing system having an imaging device and an image processing device,
The imaging device
An imaging unit that captures two or more captured images under each exposure condition including two or more exposure conditions including a first exposure condition and a second exposure condition;
The image processing apparatus includes:
Exposure condition setting means for setting an exposure condition when the imaging means captures an image;
An acquisition unit that inputs the captured image of each exposure condition from the imaging device and performs an acquisition process of acquiring two or more partial images based on the captured image of each exposure condition;
Calculation means for calculating a connection condition for connecting the partial images of the first exposure condition;
For the partial image acquired by the acquisition unit, based on the connection condition calculated by the calculation unit, the first composite image generated from two or more partial images of the first exposure condition, and the connection condition And a combining means for performing a combining process for generating at least two combined images of the second combined image generated from two or more partial images of the second exposure condition,
An image processing system comprising: generation means for performing generation processing for generating an output image based on at least the first composite image and the second composite image. An image processing method for causing an image processing apparatus to perform image processing on image data,
An acquisition procedure for causing the image processing apparatus to perform an acquisition process of acquiring two or more partial images under each exposure condition of two or more exposure conditions including a first exposure condition and a second exposure condition;
A calculation procedure for calculating a connection condition for connecting the partial images of the first exposure condition to the image processing apparatus;
A first composite image generated from two or more partial images of the first exposure condition based on the connection condition calculated in the calculation procedure for the partial image acquired in the acquisition process by the image processing apparatus. And a combining procedure for performing a combining process for generating at least two combined images of the second combined image generated from two or more partial images of the second exposure condition based on the connection condition;
An image processing method comprising: a generation procedure for causing the image processing apparatus to perform a generation process for generating an output image based on at least the first composite image and the second composite image. A program for causing an image processing apparatus to perform image processing of image data,
An acquisition procedure for causing the image processing apparatus to perform an acquisition process of acquiring two or more partial images under each exposure condition of two or more exposure conditions including a first exposure condition and a second exposure condition;
A calculation procedure for causing the image processing apparatus to calculate a connection condition for connecting the partial images of the first exposure condition;
A first composite image generated from two or more partial images of the first exposure condition based on the connection condition calculated in the calculation procedure for the partial image acquired in the acquisition process by the image processing apparatus. And a combining procedure for performing a combining process for generating at least two combined images of the second combined image generated from two or more partial images of the second exposure condition based on the connection condition;
A program for causing the image processing apparatus to execute a generation procedure for performing a generation process for generating an output image based on at least the first composite image and the second composite image.