JP2021047608A - Image processing device, imaging device, image processing method, and program - Google Patents

Abstract

To provide an image processing device capable of appropriately performing alignment between images when synthesizing a plurality of images.SOLUTION: An image processing device includes: a reference image serving as a position reference of image synthesis; an object image having an object to be included in a composite image obtained by the image synthesis; an image processing part which generates a difference image by selecting the reference image to be compared with the object image from a plurality of images and aligning the reference image and the object image to extract a difference; and an image composition part which generates the composite image by aligning the difference image with the reference image and combining them.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image processing device, an imaging device, an image processing method, and a program for synthesizing a plurality of images to obtain a composite image.

Astronomical photography is widely used to photograph celestial bodies such as stars, planets, nebulae, and galaxies, and astronomical phenomena such as meteors, solar eclipses, and lunar eclipses. When shooting a meteor shower, which is a phenomenon in which a large number of meteors occur intensively, in the captured image, celestial bodies other than the meteor do not draw a trajectory, more meteors are contained, and the relative of the meteor to the star. It is important that the position is accurate. It is also important that the captured image captures how each meteor radiates from the radiant point of the meteor shower, which is a point on the celestial sphere.

In astronomical photography, the amount of light of the subject is often very small, so exposure for a relatively long time (for example, 30 seconds, 1 minute, etc.) is often performed to acquire one image. Since the earth is rotating, the stars on the celestial sphere seen from the earth appear to be in the apparent motion of diurnal motion. Therefore, when the imaging device is fixed and exposure is performed for a long time in astronomical photography, the star draws an arc-shaped (substantially straight line) trajectory in the photographed image instead of forming a point image.

Therefore, in astronomical photography, a star image is acquired as a point image by using an equatorial mount that moves the image pickup device in the right ascension direction so as to track a star moving by diurnal motion. However, equatorial mounts are generally expensive and heavy, and it is difficult for some users to use them because they require polar alignment and other settings to orient the polar axis toward the celestial pole (north pole or south pole). .. Therefore, a plurality of images are acquired by repeating a short exposure so that the star does not draw a locus, and the deviation of the star image due to the diurnal motion occurring between the acquired images is corrected and combined. It is preferable to acquire one composite image.

Patent Document 1 describes the extracted difference by aligning the target image in which the subject exists with respect to the background image in which the subject does not exist and then extracting the difference in photographing a general subject (automobile or the like) other than the celestial body. Discloses a technique for synthesizing a background image.

Japanese Unexamined Patent Publication No. 2011-72035

In the technique of Patent Document 1 relating to a general subject, the background itself does not move over a plurality of shots, so if the images are combined by aligning based on the relationship between the background and the subject (shooting object). It is enough. However, in astronomical photography, the stellar image itself, which should be the background, moves due to diurnal motion, so it is considered inappropriate to simply apply the alignment technique of Patent Document 1 in consideration of only the background and the subject.

For example, as described above, since the stellar image as the background moves due to diurnal motion, the image including the object to be photographed such as a meteor is aligned with the background as compared with the case where the background is fixed. Difficult to do. Further, since the stellar image is a set of light spots and is less complicated than a general image, it is difficult to align with high accuracy even by using a simple image comparison.

In view of the above circumstances, it is an object of the present invention to provide an image processing device, an imaging device, an image processing method, and a program capable of appropriately aligning images when synthesizing a plurality of images.

In order to achieve the above object, the image processing apparatus of the present invention compares a reference image that serves as a position reference for image composition, a target image having an object to be included in the composite image obtained by image composition, and the target image. An image processing unit that generates a difference image by selecting the reference image to be used from a plurality of images, aligning the reference image and the target image and extracting the difference, and aligning the difference image with respect to the reference image. It is characterized by including an image compositing unit that generates a compositing image by compositing the images.

According to the present invention, it is possible to appropriately perform alignment between images when synthesizing a plurality of images.

It is a block diagram which shows the structure of the digital camera which is an example of the image processing apparatus which concerns on 1st Embodiment of this invention. It is explanatory drawing about an example of the motion vector acquisition processing in 1st Embodiment of this invention. It is explanatory drawing about another example of the motion vector acquisition processing in 1st Embodiment of this invention. It is a flowchart which shows the synthesis process in 1st Embodiment of this invention. It is explanatory drawing which shows the relationship of a plurality of images concerning the synthesis process in 1st Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Each embodiment described below is merely an example of a configuration in which the present invention can be realized. Each of the following embodiments can be appropriately modified or modified according to the configuration of the apparatus to which the present invention is applied and various conditions. In addition, not all combinations of elements included in the following embodiments are essential for realizing the present invention, and some of the elements can be omitted as appropriate. Therefore, the scope of the present invention is not limited by the configurations described in each of the following embodiments. Further, as long as there is no mutual contradiction, a configuration in which a plurality of configurations described in the embodiment are combined can also be adopted.

<First Embodiment>
FIG. 1 is a block diagram showing a configuration of a digital camera 100 which is an example of an image processing device according to the first embodiment of the present invention. The digital camera 100 is an imaging device having the following elements shown in the dotted line. Generally, the digital camera 100 of the present embodiment performs alignment and composition processing on a plurality of images to acquire one composite image.

The photographing lens 103 is a lens group used for photographing a subject, and includes a lens such as a focus lens and a zoom lens. The shutter 101 is a mechanical shutter having an aperture mechanism. The image pickup unit 22 has an image pickup element that converts an optical image formed through the photographing lens 103 into an electric signal by photoelectric conversion and outputs the image sensor. The A / D conversion unit 23 is a conversion circuit that converts an analog signal output from the imaging unit 22 into a digital signal (image data) and outputs the digital signal (image data). The barrier 102 is a barrier that prevents stains and damage by covering the photographing lens 103 and the imaging unit 22.

The image sensor described above has red (R), green (G1, G2), and blue (B) color filters that are regularly arranged according to the Bayer arrangement. An image sensor that follows a filter arrangement other than the Bayer arrangement may be adopted. The digital camera 100 may have an electronic shutter realized by controlling the reset timing of the image pickup device in place of or in addition to the above shutter 101.

The image processing unit 24 is, for example, an integrated circuit such as an image processing engine, and executes various image processing as illustrated below on the images supplied from the A / D conversion unit 23 and the memory control unit 15. Then, the image after image processing is output.

-Pixel interpolation processing for image data-Various correction processing including shading correction. In the shading correction of the present embodiment, the image data is corrected so that the shading caused by the characteristics of the photographing lens 103, the aberration of the imaging unit 22, etc. is corrected by applying a gain for each pixel according to the two-dimensional coordinates of the image sensor. Brightness level is adjusted.

-White balance (WB) processing. In the auto white balance processing of the present embodiment, different gains are applied to the image data corresponding to the Bayer array color filters (R, G1, G2, B), so that the reference portion (maximum brightness portion) in the image data is applied. ) Is corrected so that it becomes a predetermined color (white). The above auto white balance processing (AWB processing) is executed on the image data to which shading correction has been applied.

-Γ correction processing, color conversion processing-Electronic zoom processing by cropping and scaling of image data-Development processing (conversion processing of RAW data to YUV color space and encoding processing to a predetermined image format (JPEG format, etc.))
-Decoding process from image format-Difference calculation process (process to calculate the difference of color information such as brightness value in each pixel from two images)
-Mask processing, low-pass filter processing-Threshold processing (processing to truncate the color information of image data having a value equal to or less than the set predetermined threshold value for each pixel)
-Motion vector acquisition process (process to calculate motion vector from multiple image data. Details will be described later)

The image composition unit 54 is a composition processing circuit that executes a composition process for synthesizing a plurality of image data, and may be provided separately from the image processing unit 24, or the image processing unit 24 physically performs the image composition unit 54. It may have a physical or functional effect. The image compositing unit 54 can execute comparative bright compositing processing in addition to simple additive compositing and additive average compositing. The comparative bright composition process is a process of generating one image data by comparing a plurality of image data to be synthesized, selecting a portion (pixel or area) having the brightest value, and synthesizing the data. .. Further, the image compositing unit 54 can execute an embedding compositing process in which additional information such as a mask image generated by the image processing unit 24 and a region having a difference in color information is embedded in a reference image. In addition, when synthesizing a plurality of image data, the image synthesizing unit 54 can perform the compositing process after performing the alignment based on the motion vector information supplied from the image processing unit 24.

The system control unit 50 is a control unit that integrally controls the operation of each element of the digital camera 100, and is, for example, a CPU (Central Processing Unit). The system control unit 50 realizes the above-mentioned integrated control and realizes each process of the present embodiment by executing the program stored in the non-volatile memory 56 described later. The system control unit 50 controls the exposure control means 40 and the focus control means 42 based on the calculation result acquired based on the calculation process for the image data by the image processing unit 24, thereby performing the TTL method automatic focus processing and the focus control unit 42. Achieve automatic exposure processing.

The system memory 52 is a storage medium that functions as a working memory for storing constants and variables during operation of the system control unit 50, and is, for example, a RAM (Random Access Memory). The non-volatile memory 56 is a storage medium for storing the above-mentioned program code, system information, and user setting information, and is, for example, a flash memory. The system control unit 50 sequentially reads the program from the non-volatile memory 56 and executes the program while expanding the program into the system memory 52. Further, the system control unit 50 restores the system environment of the digital camera 100 by reading various setting information stored in the non-volatile memory 56 at startup.

The system timer 53 is a timing generation unit used for various controls and a time measuring unit for measuring the time of a built-in clock (not shown). The system control unit 50 controls the operations of the photographing lens 103, the shutter 101, the imaging unit 22, and the like based on the timing generated by the system timer 53.

The memory control unit 15 controls input / output of image data related to the A / D conversion unit 23, the image processing unit 24, and the memory 32, and reading / writing to / from the memory 32. The memory 32 is a memory for storing captured still images and moving images. Under the control of the memory control unit 15, the image data output from the A / D conversion unit 23 is supplied to the memory control unit 15 directly or through image processing in the image processing unit 24. The memory control unit 15 writes the supplied image data to the memory 32.

Since the memory 32 has a sufficient capacity and band for storing a plurality of still images and moving images, it is suitable for writing a large amount of high-speed image data by continuous shooting in which a plurality of still images are continuously photographed. It fits. Further, the memory 32 can store temporary data at the time of image processing by the image processing unit 24, for example, an image ID that identifies a selected reference image, an acquired motion vector, and the like. The memory 32 can also function as a working memory of the system control unit 50.

The display unit 28 is an element that displays image data stored in the memory 32 and various setting information of the digital camera 100, and is composed of, for example, a TFT liquid crystal. The image data stored in the memory 32 is displayed on the display unit 28 via the memory control unit 15 under the control of the system control unit 50. The live view display and the electronic viewfinder function can be realized by sequentially displaying the image data output from the imaging unit 22 by the exposure and reading in a predetermined cycle on the display unit 28 via the image processing unit 24 and the memory control unit 15. .. The display unit 28 can switch between an ON state and an OFF state based on an instruction from the system control unit 50. By turning off the display unit 28, the power consumption of the entire digital camera 100 can be reduced.

The operation unit 70 is an element including various buttons and devices such as a touch panel to be operated by the user, and for example, the following elements are structurally or functionally (for example, a software button displayed on the touch panel). As) including.

− Shutter button − Menu button − Set button − Macro button − Flash setting button − Single shot / continuous shooting / self-timer switching button − Menu move + (plus) button − Menu move − (minus) button − Shooting image quality selection button − Exposure Correction button-Date / time setting button

The menu activated by pressing the menu button included in the operation unit 70 and displayed on the touch panel or the like may include an item for selecting and setting an arbitrary image as a reference image.

The shutter button included in the operation unit 70 has a first shutter switch SW1 that is turned on during the pressing operation (that is, by pressing halfway) and a first shutter switch SW1 that is turned ON by completing the pressing operation (that is, by pressing the button fully). 2 Includes shutter switch SW2. When the first shutter switch SW1 is turned on, the system control unit 50 is instructed to start shooting preparation operations such as AF processing, AE processing, and AWB processing. On the other hand, when the second shutter switch SW2 is turned on, the system control unit 50 is instructed to start shooting operations such as exposure processing, development processing, compression / decompression processing, and recording processing.

The power supply unit 30 is an element that supplies power to the digital camera 100, and is at least one of a primary battery (alkaline battery, lithium battery, etc.), a secondary battery (NiCd battery, NiMH battery, Li battery, etc.), and an AC adapter. Consists of. The power supply unit 30 is controlled by the power supply control unit 80.

The interface 18 is a connection interface with a recording medium (memory card, hard disk drive, etc.) that can be connected to the digital camera 100, and is, for example, a connector.

An example of the motion vector acquisition process according to the first embodiment of the present invention will be described with reference to FIG. The motion vector is calculated by the vector processing unit realized by executing the program by the image processing unit 24 based on a plurality of image data. In FIG. 2, motion vectors are acquired from the first and second image data illustrated in the upper portion. The second image data is taken after the first image data in chronological order, and a plurality of feature points (stellar images) appear on the image from the first image data to the second image data. It is moving to the lower right.

In order to calculate the motion vector, the vector processing unit divides each image data into a plurality of blocks (for example, 80 blocks of 8 rows and 10 columns) as shown in the figure. Each block is further divided into a plurality of subblocks. That is, each block contains a plurality of subblocks. Hereinafter, description will be made with reference to the block (a) shown on the first sheet. The block (a) contains one feature point.

The vector processing unit sets one sub-block including the feature points as a template for difference detection in the first image data (lower left portion (first image) in FIG. 2). Next, the vector processing unit scans one pixel at a time using the template in the second block corresponding to the first block including the above template, and calculates the difference (lower right of FIG. 2). part). In this example, as shown in the figure, the difference when the template is displaced by m pixels to the right and n pixels downward from the original position is the smallest. The smaller the difference, the greater the similarity between the template and the subblock. Therefore, in this example, by the above difference detection, a motion vector indicating that the feature point has moved m pixels to the right and n pixels downward from the first image data to the second image data is acquired. ..

The vector processing unit performs the above-mentioned difference detection processing (that is, for a block in which a luminance difference exists (that is, a feature point is assumed to exist) in all the blocks included in the image data or in the block. Motion vector acquisition process) is executed.

The vector acquisition unit is adjacent to the block in which the feature point exists on the first sheet but the motion vector is not detected, that is, the block in which the feature point may have moved to another block is adjacent to the block. The motion vector is detected by scanning the block additionally. Preferably, the vector acquisition unit should scan (ie, by predicting the direction in which the feature points in the block have moved, based on the motion vector previously detected by the movement of the feature points in the block. Narrow down adjacent blocks (to be searched).

The vector acquisition unit acquires a motion vector by interpolation processing using neighboring blocks included in a wider range than the adjacent block for the block for which the motion vector is not acquired even by the above-mentioned search for the adjacent block.

In the above example described with reference to FIG. 2, the image data is divided into blocks of 8 rows and 10 columns, but the image data can be divided by blocks of an arbitrary number and shape. For example, when the photographing lens 103 is a wide-angle lens having a short focal length and therefore a large amount of apparently small stellar images are included in the image data, it is preferable that the block size is set small. Therefore, it is preferable that the image processing unit 24 changes the number of divided blocks of the image data according to the focal length of the photographing lens 103. The shape of the block may be a square as shown in FIG. 2, a rectangle, or a hexagon.

By the way, if the degree of similarity (sum of absolute values of differences) is calculated by scanning the template pixel by pixel as in the vector acquisition process described above, the processing load may become excessive. Further, when the shooting time difference between the two images to be compared (for example, the above-mentioned first and second image data) is large, there is a possibility that the arrangement pattern of similar feature points may be erroneously detected. is assumed. Therefore, in the present embodiment, it is possible to execute the motion vector acquisition process that reduces the possibility of erroneous detection as follows.

Another example of the motion vector acquisition process according to the first embodiment of the present invention will be described with reference to FIG. In the upper part of FIG. 3, a series of captured images (image data) from the first image to the nth image are shown. Time has passed each time the number of sheets is chased. That is, the second image is taken at a time later than the first image, the third image is taken at a time later than the second image, and similarly, the nth image is taken after the (n-1) th image. Was filmed at a later time. Similar to FIG. 2, each image data is divided into a plurality of blocks.

In this example, it will be described with reference to the feature point (b) which is a stellar image shown in each image. As shown in the figure, the feature point (b) moves toward the lower right on the images in the plurality of captured images. The 4 rows and 5 columns grid shown in the lower part of FIG. 3 is an enlargement of the 4 rows and 5 columns block located at the upper left of the blocks shown in the upper part.

The movement of the feature point (b) from the first sheet to the nth sheet is shown in the above grid. The solid arrows in the grid correspond to each motion vector of the feature point (b) acquired by the image processing unit 24 that functions as the vector processing unit. (V1: 2) is a motion vector corresponding to the movement of the feature point (b) from the first to the second sheet, and (V2: 3) is the feature point (b) from the second to the third sheet. It is a motion vector corresponding to the movement of. The same applies hereinafter, and (Vn-1: n) indicating the motion vector corresponding to the movement of the feature point (b) from the (n-1) th sheet to the nth sheet is located at the end.

The image processing unit 24 acquires motion vectors sequentially in chronological order for two consecutive image data included in a series of all image data. That is, each time the image processing unit 24 acquires the image data, the above-mentioned (V1: 2), (V2: 3), ..., (Vn-1: n) are sequentially acquired. In the above time-series motion vector calculation process, when new image data is acquired, the image processing unit 24 corresponds to the sum of the motion vectors indicating the motions of a series of feature points separately from each motion vector. The value (total vector) to be processed is stored in the memory 32. For example, the image processing unit 24 that has acquired the third image data, in addition to acquiring the current motion vector (V2: 3), changes the previous motion vector (V1: 2) to (V2: 3). The added total vector (totalV3) is stored in the memory 32. As a generalization, when the xth image data (3 ≦ x ≦ n) is acquired, the image processing unit 24 as the vector processing unit first acquires the motion vector (Vx-1: x) this time. .. Then, the image processing unit 24 includes the total vector (totalVx-1) from (V1: 2) to (Vx-2: x-1) already stored in the memory 32 and the current (Vx-1: x). And are added to obtain a new total vector (totalVx) and store it in the memory 32. In the processing of the third and subsequent sheets, at least the previous total vector (totalVx-1) and the current total vector (totalVx) are stored in the memory 32. The above total vector is used for alignment of images, which will be described later.

The image data compositing process executed by the digital camera 100 of the present embodiment will be described with reference to FIGS. 4 and 5. FIG. 4 is a flowchart showing a synthesis process according to the first embodiment of the present invention. FIG. 5 is an explanatory diagram showing the relationship between a plurality of images in the above compositing process. Generally, in the composition process of the present embodiment, the difference image obtained by detecting the difference between the target image and the reference image is combined with the reference image that defines the angle of view.

In step S401, the user operates the operation unit 70 to instruct the digital camera 100 to perform continuous shooting processing and compositing processing of a plurality of images.

In step S402, the system control unit 50 controls each unit of the digital camera 100 so as to execute continuous shooting of a plurality of images based on an instruction by the user via the operation unit 70. As a result, a plurality of (n) images are acquired and stored in the memory 32.

In step S403, the image processing unit 24 operates as a reference setting unit for setting a reference image 501, which is a position reference in image composition, from the plurality of images acquired in step S402. The reference image 501 is an image that is a base for synthesizing an object to be synthesized such as a meteor in the image data synthesizing process, and is an image that defines the angle of view of the composite image finally obtained by the synthesizing process. .. The reference image 501 may be selected by any method. For example, the first image of a plurality of continuously captured images may be automatically set as the reference image 501, or the image designated by the user via the operation unit 70 may be set as the reference image 501. Good. Alternatively, the first selected target image 503 may be set as the reference image 501. In this example, the first image data is set as the reference image 501.

In step S404, the image processing unit 24 first selects an image (target image 503) including the object 505 and should be the extraction target of the difference image 507, and the reference image 502. Next, the image processing unit 24 executes the alignment of the selected reference image 502 and the target image 503. Specifically, the image processing unit 24 acquires a motion vector (first motion vector) between images as described above based on the feature points 504 that are stellar images included in each image, and the acquired motion vector. Is used to align the reference image 502 and the target image 503. The above motion vector indicates the amount of movement of the feature point 504 from the reference image 502 to the target image 503. The outline of the alignment is shown in the superimposed image 506. The alignment of the present embodiment means a process of digitally superimposing the target image 503 and the reference image 502 based on the motion vector.

The target image 503 may be designated, for example, by an instruction from the user via the operation unit 70, or is a feature of the target object 505 (an object that draws a substantially linear locus in which a predetermined number or more of high-luminance pixels exist). It may be selected based on the existence of an object, etc.). Further, as the reference image 502, it is preferable that image data adjacent to the target image 503 and captured between the reference image 501 and the target image 503 is selected. That is, it is preferable that the (n-1) th reference image 502 is selected with respect to the nth target image 503. When the reference image 502 is selected as described above, the overlapping area of the target image 503 and the reference image 502 and the overlapping area of the reference image 501 and the reference image 502 are further increased, so that the motion vector is calculated more easily and with high accuracy. Will be done.

If the selected (n-1) th image is not suitable for the reference image 502, the image processing unit 24 finds a suitable image until the (n-2) th image, the (n-3) th image, ... You may search retroactively. As an example of an image that is not suitable for the reference image 502, there is an image in which a region having a ratio exceeding a predetermined threshold value is detected as a difference in the vector calculation region that overlaps with the target image 503. More specifically, for example, it is an image in which an obstacle that is not in the target image 503 is reflected, or an image whose exposure condition is significantly different from that of the target image 503 due to lighting such as a headlight of an automobile. In addition, an image in which the vector calculation fails due to camera shake and an image in which a camera shake amount equal to or larger than a predetermined threshold value is detected by the gyro sensor are examples of images that are not suitable for the reference image 502.

In step S405, as described above, the image processing unit 24 adds the current motion vector (movement amount) acquired in step S404 to the past total vector and stores it in the memory 32.

In step S406, the image processing unit 24 compares the aligned reference image 502 with the target image 503, extracts the difference, and acquires the difference image 507. As the above difference extraction, for example, the image processing unit 24 extracts the difference between the luminance signal Y, the blue component difference signal U, and the red component difference signal V for each pixel or in the target image 503 and the reference image 502 having YUV data, respectively. It may be calculated for each area. That is, the image processing unit 24 acquires the following Y difference ΔY, U difference ΔU, and V difference ΔV for each pixel or each region.

ΔY = (Y data of target image 503)-(Y data of reference image 502)
ΔU = (U data of target image 503)-(U data of reference image 502)
ΔV = (V data of target image 503)-(V data of reference image 502)
When the Y difference ΔY is smaller than 0 (negative value), the image processing unit 24 sets each difference ΔY, ΔU, ΔV in the pixel or region to 0. Here, the fact that the Y difference ΔY of a certain pixel or region is smaller than 0 means that the brightness of the pixel or region in the reference image 502 is higher than that of the target image 503, that is, the reference image 502 includes an object 505 such as a meteor. It means that there is a high possibility of being affected. By setting the difference to 0 as described above, it is possible to prevent the pixels or regions of the target image 503 that do not include the object 505 from being extracted as differences, and by extension, the pixels or regions of the target image 503. Is prevented from being combined with the reference image 501.

In step S407, the image processing unit 24 determines whether or not there is a difference in the processing in step S406. When there is no difference between the reference image 502 and the target image 503 (S407: NO), the image processing unit 24 skips the subsequent steps S408 and S409 and proceeds to the process in step S410. By the above determination, it is possible to prevent unnecessary alignment and composition processing from being executed. On the other hand, when there is a difference between the reference image 502 and the target image 503 (S407: YES), the image processing unit 24 advances the process to step S408.

In step S408, the image processing unit 24 acquires a motion vector between the reference image 501 and the target image 503. More specifically, the image processing unit 24 acquires a motion vector (second motion vector) between the reference image 501 and the reference image 502, and between the reference image 502 and the target image 503 in the memory 32. A desired motion vector is obtained by adding to the motion vector (Vn-1: n). That is, the image processing unit 24 calculates the motion vector between the reference image 501 and the target image 503 indirectly, not directly, but through the reference image 502. By calculating the motion vectors using images that are closer to each other, the accuracy of the calculation is improved. The image processing unit 24 may directly acquire the motion vector between the reference image 501 and the target image 503 in consideration of the occupancy timing of the memory area and the like.

When acquiring the motion vector between the reference image 501 and the reference image 502, the image processing unit 24 preliminaryly prepares two images by using the previous motion vector (totalVn-1) stored in the memory 32. It is preferable to acquire the target motion vector after alignment. Since the reference image 501 and the reference image 502 are brought closer to each other by the preliminary alignment, it is possible to prevent the overlap of the feature point patterns in the image from being erroneously detected as compared with the star in which the two images are not brought close to each other. ..

In step S409, the image processing unit 24 aligns and synthesizes the difference image 507 acquired in step S406 with the reference image 501 using the motion vector acquired in step S408, and acquires the composite image 509. In addition, the image composition unit 54 is controlled. In the above synthesis process, a known synthesis method (for example, additive synthesis process, additive average synthesis process, comparative bright synthesis process) may be used.

In step S410, the image processing unit 24 determines whether or not the system control unit 50 has output a shooting completion command. If the shooting completion command is output (S410: YES), the image processing unit 24 outputs the composite image 509 acquired in the latest step S409 to the image synthesis unit 54 and ends the processing (step S411). ). On the other hand, if the shooting completion command is not output, the image processing unit 24 returns the processing to step S402, and repeats the above processing with the composite image 509 as a new reference image 501.

According to the configuration of the present embodiment described above, the difference image obtained by comparing the target image and the reference image and including the target object is aligned with the reference image and synthesized. That is, in image composition, alignment between a plurality of images is appropriately performed. For example, when shooting a meteor shower according to the configuration of the present embodiment, a normal star is depicted as a non-moving point in the composite image, while the meteor shower included in the meteor shower accurately determines the relative position with respect to the star. It is depicted as a linear trajectory while maintaining it.

Further, according to the configuration of the present embodiment described above, the alignment is executed based on the motion vector indicating the amount of movement of the feature points over the continuous image. Therefore, it is possible to perform the alignment more accurately than the alignment by a simple pattern comparison. The above technical effect has a relatively simple pattern of a star image, and is more remarkable when the configuration of the present embodiment is applied to a plurality of astronomical images taken over a long period of time. To. In addition, since the movement amount of the feature point can be specified by the motion vector, the need to store all the image data in the memory 32 is reduced.

<Second Embodiment>
Hereinafter, a second embodiment of the present invention will be described. In each of the embodiments illustrated below, for elements having the same functions and functions as those in the first embodiment, the reference numerals referred to in the above description will be used and the respective description will be omitted as appropriate.

In order to include more meteors in one composite image, it is desirable to perform shooting for a longer period of time to acquire more images and perform image composition. However, if the exposure is extended for a long time, the feature points included in the reference image (angle of view of the composite image) will eventually move due to diurnal motion and will not be included in the target image. Since the target image taken in such a state cannot be aligned with the reference image using the feature points, it cannot be the target of image composition.

In the second embodiment, the system control unit 50 drives the exposure control means 40 and the focus control means 42 to displace the angle of view so as to track the diurnal motion, so that the reference image and the target image overlap. (The period during which the target image includes the feature points of the reference image) is extended. For example, after the shooting of the nth image is completed and before the shooting of the (n + 1) th image is started (between step S410 and step S402), the image processing unit 24 causes the motion vector (Vn−). The direction of diurnal motion is estimated based on 1: n). Then, the system control unit 50 drives the exposure control means 40 and the focus control means 42 to track the correction optical system in the direction estimated by the image processing unit 24. In the present embodiment, in addition to the total vector described in the first embodiment, it is preferable to store at least the motion vectors corresponding to the two most recent target images 503 in the memory 32.

According to the configuration of the present embodiment described above, the same technical effect as that of the first embodiment can be obtained, and a plurality of images taken over a longer time can be combined as one composite image. It will be possible. As a result, for example, when a meteor shower is the object of photography, it is possible to fit more meteors in one composite image.

<Modification example>
Each of the above embodiments is variously modified. A specific mode of modification is illustrated below. Two or more embodiments arbitrarily selected from the above embodiments and the following examples can be appropriately merged as long as they do not contradict each other.

In the above embodiment, it is preferable that the image processing unit 24 executes noise reduction processing. In general, random noise is likely to be included in an image signal when the ISO sensitivity is high, the exposure time is long, the temperature of the image sensor is rising, and the like. When shooting astronomical objects, high ISO sensitivity (for example, ISO 1600 or higher) tends to be used to shoot an astronomical image that is a relatively dark subject, and the shooting time, that is, the exposure time tends to be long. The temperature of the image sensor tends to rise due to long-time shooting. That is, noise is likely to occur in astronomical photography.

Therefore, the image processing unit 24 of this modification executes noise reduction processing on the reference image 502 and the target image 503 prior to the difference extraction processing in step S406. The noise reduction processing removes, for example, punctate high-intensity random noise that occurs when the ISO sensitivity is high. The noise reduction method can be arbitrarily selected. For example, a method of applying a low-pass filter to the entire image, setting a high-luminance portion after applying the low-pass filter as a mask area, and applying a mask to the original image can be adopted. In addition, based on the finding that stars and meteors tend to have higher brightness than noise, a method is also adopted in which difference extraction is not performed for pixels or regions where the brightness signal Y in the target image 503 is less than or equal to the threshold value. obtain. However, according to the latter method, there is a possibility that celestial bodies (high-grade stars) having brightness below the threshold value will be removed.

In the above-described embodiment, the image processing unit 24 adopts the portion showing the difference only when the portion showing the extracted difference has a linear shape having a predetermined thickness and length with respect to the sensor size. The difference extraction process may be executed so as to be performed. In other words, when the extracted difference portion does not satisfy the above condition, the difference may not be adopted and the portion may be processed so as to exhibit 0 in the difference image. As a method for extracting a linear shape, a VanderBrug line detection method, an LSD method, a Paton line detection method, a Kasvand line detection method, or the like may be adopted. According to the above configuration, high-intensity subjects other than meteors (clouds, floating insects and birds, building window lights, automobile tail lamps, etc.) are suppressed from being extracted as differences.

In addition, the image processing unit 24 extends a plurality of linear shapes corresponding to the differences acquired from the plurality of target images to acquire intersections (corresponding to the radiant points of the meteor shower), and the linear shapes do not go to the above intersections. The difference corresponding to the shape may be removed. This is because the linear shape that does not face the radiant point is an object other than a meteor, such as an artificial satellite or an airplane. The image processing unit 24 can realize the above removal by, for example, replacing the difference portion to be removed with a corresponding portion in the reference image before the difference synthesis. In this modification, in addition to the reference image after the composition processing, the reference image before the composition processing is also stored in the memory 32, or the difference image in which the reference image and the difference image are sequentially combined is stored in the memory 32. It is preferable to hold it.

In the above-described embodiment, the digital camera 100 is exemplified as the image processing device, but any information processing device can also configure the image processing device of the present invention. For example, any device such as a mobile phone, a smartphone, a personal computer (laptop type, desktop type, tablet type, etc.), a game machine, etc., which has a built-in image pickup device or is externally connected to the image pickup device, is the image processing of the present invention. Can function as a device. In addition, a device such as a personal computer that does not have an image pickup function can form the image processing device of the present invention. For example, even if a personal computer having no imaging function reads a plurality of image data taken by another digital camera and executes the image processing in the above embodiment to acquire one composite image. Good. Therefore, the "image processing apparatus" in the present specification is a general concept that includes any electronic device having an image processing function.

The present invention can also be realized by the following configuration. A recording medium containing the program code of the software in which the procedure for realizing the function of the above-described embodiment is described is supplied to the system or the device, and the computer of the system or the device reads and executes the program code in the recording medium. A computer is configured to include, for example, one or more CPUs and / or arithmetic elements such as MPUs. In the present configuration, since the program code itself read from the recording medium realizes the new function according to the present invention, the storage medium on which the program code is recorded and the program itself also constitute the present invention.

As a recording medium for supplying the above program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, or the like can be adopted. In addition, recordings such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-R, magnetic tape, non-volatile memory card, ROM, etc. provide the program code. It can be adopted as a medium.

By making the program code read by the above computer executable, the processing function in the above-described embodiment is realized. Further, based on the instruction of the program code, the software such as the OS (operating system) running on the above computer executes a part or all of the above-mentioned processing, and the function in the above-described embodiment is realized. You may.

In addition, the present invention can also be realized by the following configuration. The program code of the above software is written to the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer. After that, based on the instruction of the program code written in the memory, an arithmetic element such as a CPU included in the function expansion board or the function expansion unit executes a part or all of the actual processing.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the gist thereof.

24 Image processing unit 50 System control unit 54 Image compositing unit 100 Digital camera (imaging device, image processing device)
501 Reference image 502 Reference image 503 Target image 504 Feature point 509 Composite image

Claims (14)

A reference image serving as a position reference for image composition, a target image having an object to be included in the composite image obtained by image composition, and a reference image to be compared with the target image are selected from a plurality of images, and the reference is made. An image processing unit that generates a difference image by aligning the image with the target image and extracting the difference.
An image processing apparatus comprising: an image synthesizing unit that generates the composite image by aligning and synthesizing the difference image with respect to the reference image. The image processing unit aligns the target image with the reference image based on a first motion vector indicating the amount of movement of the feature point from the reference image to the target image.
The first aspect of claim 1, wherein the image synthesizing unit aligns the difference image with the reference image based on a total vector indicating the amount of movement of the feature points from the reference image to the target image. Image processing equipment. The image processing apparatus according to claim 2, wherein the image processing unit acquires the total vector using the first motion vector acquired by comparing the reference image with the target image. In the time series when the plurality of the images are taken, the reference image is taken after the reference image and before the target image.
The image processing unit has a second motion vector showing the amount of movement of the feature point from the reference image to the reference image, and the first motion showing the amount of movement of the feature point from the reference image to the target image. The image processing apparatus according to claim 2 or 3, wherein the total vector is acquired by adding the vector. Any one of claims 1 to 4, wherein the image compositing unit does not perform alignment and image compositing when the difference does not exist between the reference image and the target image. The image processing apparatus according to. The image processing apparatus according to any one of claims 1 to 5, wherein the reference image and the target image are adjacent to each other in a time series when a plurality of the images are taken. The image processing unit sequentially generates a plurality of the difference images from the plurality of target images, and sequentially generates the plurality of difference images.
The image processing apparatus according to any one of claims 1 to 6, wherein the image synthesizing unit sequentially synthesizes a plurality of the difference images into one reference image. The image processing apparatus according to claim 7, wherein the reference image is an image taken as the first image in a time series when a plurality of the images are taken. The image processing apparatus according to any one of claims 1 to 7, wherein the reference image is the target image for which the difference extraction process is first executed. The image processing apparatus according to any one of claims 1 to 9, wherein the image processing unit sets a difference smaller than 0 to 0 when generating the difference image. The image processing according to any one of claims 1 to 10, wherein each of the plurality of the images includes an astronomical image, and the object included in the target image is a meteor. apparatus. An image pickup apparatus that functions as the image processing apparatus according to any one of claims 1 to 11.
An imaging device including an imaging unit that acquires a plurality of the images by continuous shooting. Selecting from a plurality of images a reference image that serves as a position reference for image composition, a target image having an object to be included in the composite image obtained by image composition, and a reference image to be compared with the target image.
To generate a difference image by aligning the reference image and the target image and extracting the difference,
An image processing method comprising: generating the composite image by aligning the difference image with respect to the reference image and synthesizing the composite image. A program comprising causing a computer to execute the image processing method according to claim 13.

Images