JP2016082477A - Image processing device, method for controlling the same, control program, and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To synthesize a plurality of images, without increasing a storage capacity and an arithmetic amount.SOLUTION: When synthesizing the plurality of images obtained as a consecutive photographing result, to generate a synthetic image, a camera 100 extracts a predetermined area as an extraction area for each of the plurality of images and transmits the extraction area to a PC being an external device. The PC generates synthetic information used when synthesizing the plurality of images on the basis of the extraction area and transmits the synthetic information to the camera. The camera synthesizes the plurality of images, according to the synthetic information received from the PC, to obtain the synthetic image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, a control method thereof, a control program, and an imaging apparatus, and more particularly to an image processing apparatus for synthesizing a plurality of image data.

  In general, a technique is known in which a plurality of pieces of image data obtained as a result of shooting are combined to obtain combined image data having an angle of view that cannot be obtained with a single piece of image data. For example, there is a panorama synthesis process in which image data obtained by continuously capturing images while moving the imaging device while changing the angle of view is generated to generate wide-angle image data. Furthermore, there is a method of removing the disturbing moving subject existing in the angle of view by synthesizing image data obtained by continuously photographing.

  However, in the above-described method, all of a plurality of image data obtained as a result of photographing is stored in an imaging device or an external processing device and composite processing is performed. As a result, it is necessary as the number of images to be photographed increases. And the calculation time required for processing increases.

  For example, image data obtained as a result of continuous shooting by an imaging device is synthesized using a main storage device to remove a moving subject and generate a panoramic image (see Patent Document 1). ).

JP 2004-88678 A

  In the method described in Patent Document 1 described above, as a result of combining a plurality of pieces of image data using the main storage device, as the number of shots increases, the storage capacity required for the main storage device and the combination processing are increased. The amount of calculation will increase.

  Accordingly, an object of the present invention is to provide an image processing apparatus, a control method thereof, a control program, and an imaging apparatus capable of synthesizing a plurality of image data without increasing storage capacity and calculation amount. is there.

  In order to achieve the above object, an image processing apparatus according to the present invention is an image processing apparatus that obtains a composite image by combining a plurality of images obtained as a result of continuous shooting, and for each of the plurality of images. An extraction unit that extracts a predetermined region as an extraction region, a transmission unit that transmits the extraction region to an external device, and a synthesis that is obtained based on the extraction region in the external device and is used when combining the plurality of images Synthesizing means for receiving information from the external device and synthesizing the plurality of images in accordance with the synthesis information to obtain a synthesized image.

  An imaging apparatus according to the present invention includes an imaging unit that captures a plurality of images, the above-described image processing apparatus, and a recording unit that records the composite image on a recording medium.

  A control method according to the present invention is a control method in an image processing apparatus that obtains a composite image by combining a plurality of images obtained as a result of continuous shooting, wherein a predetermined region is extracted from each of the plurality of images. An extraction step for extracting the extracted region as an external device, and a transmission step for transmitting the extraction region to the external device, and combining information used when combining the plurality of images obtained based on the extraction region in the external device from the external device. And a synthesis step of synthesizing the plurality of images according to the synthesis information to obtain a synthesized image.

  The control program according to the present invention is a control program used in an image processing apparatus that obtains a composite image by combining a plurality of images obtained as a result of continuous shooting, and the computer provided in the image processing apparatus includes the plurality of the plurality of images. An extraction step of extracting a predetermined region as an extraction region for each of the images, a transmission step of transmitting the extraction region to an external device, and a step of combining the plurality of images obtained based on the extraction region in the external device And a synthesis step of receiving the synthesis information used in the above-mentioned external device and synthesizing the plurality of images according to the synthesis information to obtain a synthesized image.

  According to the present invention, when image synthesis is performed, synthesis information for performing image synthesis is obtained from an external device, and a plurality of images are synthesized based on the synthesis information. As a result, the image processing apparatus can synthesize a plurality of images without increasing the storage capacity and the calculation amount.

1 is a block diagram illustrating a configuration of an example of an imaging apparatus including an image processing apparatus according to a first embodiment of the present invention. 1 is a block diagram showing the configuration of a PC that is an example of an external device used with an image processing apparatus according to a first embodiment of the present invention. It is a figure which shows an example of the image data obtained as a result of continuous imaging | photography with the camera shown in FIG. 3 is a flowchart for explaining an example of an image composition process performed by the camera shown in FIG. 1. It is a figure which shows extraction of the marginal area | region performed with the camera shown in FIG. It is a figure for demonstrating the synthetic | combination process of the image performed with the camera shown in FIG. 1, (A) is a figure which shows a reference | standard image, (B) is a figure which shows images other than a reference | standard image, (C) is two figures The figure which shows the synthesized image which synthesize | combined the image, (D) is a figure which shows the synthesized image which synthesize | combined all the images. It is a flowchart for demonstrating the example about the calculation of the projection transformation coefficient performed with PC shown in FIG. FIG. 3 is a diagram for explaining calculation of a motion vector performed by the PC shown in FIG. 2, (A) is a diagram showing a reference image and its marginal region, and (B) is a diagram showing an image other than the reference image. It is a figure for demonstrating the example about the calculation of the projection transformation coefficient performed with PC shown in FIG. It is a flowchart for demonstrating an example of the image synthesis process using the projective transformation coefficient performed with the camera shown in FIG. It is a figure which shows an example of the final synthetic | combination image obtained by the image synthetic | combination process using the projective transformation coefficient performed with the camera shown in FIG. It is a flowchart for demonstrating an example of the baffle area extraction process performed with the camera by the 2nd Embodiment of this invention. It is a figure which shows an example of the image data obtained as a result of having performed continuous imaging | photography with the camera by the 2nd Embodiment of this invention. It is a figure which shows an example of the obstruction | occlusion area | region extracted with the camera by the 2nd Embodiment of this invention. It is a flowchart for demonstrating an example of the image composition process performed with PC used with the camera by the 2nd Embodiment of this invention. It is a flowchart for demonstrating the image-synthesis process performed with the camera by the 2nd Embodiment of this invention. It is a figure which shows an example of the synthesized image obtained with the camera by the 2nd Embodiment of this invention.

Hereinafter, an example of an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.
[First Embodiment]

  FIG. 1 is a block diagram showing a configuration of an example of an imaging apparatus including an image processing apparatus according to the first embodiment of the present invention. In the following description, a digital camera (hereinafter simply referred to as a camera) is taken as an example of an imaging device, and a personal computer (PC) is taken as an example of an external device that communicates with the camera.

  The camera 100 has a control unit 101, and this control unit 101 is, for example, a CPU. The control unit 101 reads out a control program (also referred to as an operation program) from the ROM 102, expands the control program in the RAM 103, and executes it. As a result, the control unit 101 controls the entire camera 100.

  The ROM 102 is, for example, a rewritable nonvolatile memory, and the ROM 102 stores parameters necessary for the operation of the camera 100 in addition to the operation program of the camera 100. The RAM 103 is a rewritable volatile memory, and the RAM 103 is used as a storage area for temporarily storing data such as image data obtained by the operation of the camera 100.

  The optical system 104 forms an optical image (subject image) on the imaging unit 105. The optical system 104 detects the movement of the camera 100 and outputs the direction and magnitude of the movement.

  The imaging unit 105 is an imaging device such as a CCD or CMOS image sensor, for example, and photoelectrically converts an optical image formed by the optical system 104 and outputs an analog image signal to the A / D conversion unit 106. The A / D converter 106 A / D converts the analog image signal and outputs the digital image signal to the RAM 103 as image data.

  The image processing unit 107 performs predetermined image processing on the image data stored in the RAM 103 and records the processed image data in the RAM 103. The recording medium 108 is a memory card that can be attached to and detached from the camera 100. The recording medium 108 records processed image data stored in the RAM 103 or image data A / D converted by the A / D converter 106 as recorded image data.

  The communication unit 109 transmits image data or the like recorded on the recording medium 108 to an external device (not shown) using a public mobile communication network such as a wireless LAN. Further, the communication unit 109 receives various data from the external device.

  FIG. 2 is a block diagram showing the configuration of a PC which is an example of an external apparatus used with the image processing apparatus according to the first embodiment of the present invention.

  The PC 200 has a control unit 201, and this control unit 201 is, for example, a CPU. The control unit 201 reads a control program (also referred to as an operation program) from the ROM 202, expands the control program in the RAM 203, and executes it. As a result, the control unit 201 controls the entire PC 200.

  The ROM 202 is, for example, a rewritable nonvolatile memory, and the ROM 202 stores parameters necessary for the operation of the PC 200 in addition to the operation program of the PC 200. The RAM 203 is a rewritable volatile memory, and the RAM 103 is used as a storage area for temporarily storing data obtained by the operation of the PC 200.

  The communication unit 204 receives data from an external device and transmits data to the external device using a public mobile communication network such as a wireless LAN. For example, the communication unit 204 receives image data from the camera 100 illustrated in FIG.

  The recording medium 205 is, for example, a built-in hard disk device, and image data received by the communication unit 204 is stored in the recording medium 205. Then, the image data stored in the recording medium 205 is expanded in the RAM 203, and predetermined image processing (for example, synthesis processing) is performed by the image processing unit 206.

  FIG. 3 is a diagram showing an example of image data obtained as a result of continuous shooting by the camera 100 shown in FIG.

  Now, assume that a subject is continuously photographed using the camera 100 to obtain a plurality of pieces of image data. Here, it is assumed that images are obtained in the order of images 300 to 303 shown in FIG.

  FIG. 4 is a flowchart for explaining an example of the image composition processing performed by the camera 100 shown in FIG. Note that the processing according to the illustrated flowchart is performed under the control of the control unit 101.

  In the camera 100, predetermined image processing is performed on the image data obtained as a result of continuous shooting to obtain a plurality of pieces of image data (hereinafter also simply referred to as images) 300 to 303 shown in FIG. 3 (step S400). ). The control unit 101 stores these images 300 to 303 in the recording medium 108.

  Subsequently, the control unit 101 selects a reference image from the images 300 to 303 (step S401). Here, the reference image refers to an image that is a conversion destination in a projective conversion process to be described later. In the illustrated example, the control unit 101 selects an image 300 that is first captured as a reference image. Note that any one of the images 301 to 303 other than the image 300 may be selected as the reference image. Hereinafter, the images 301 to 303 that have undergone the projective transformation processing are referred to as converted images 301 to 303, respectively.

  Subsequently, the control unit 101 determines whether or not predetermined image processing described later by the image processing unit 107 has been completed for all the images 301 to 303 other than the reference image (step S402). If image processing has not been completed for all of the images 301 to 303 (NO in step S402), the control unit 101 uses the image processing unit 107 to extract a marginal region from each of the reference image 300 and the images 301 to 303 as an extraction region. Then, it transmits to PC200 (step S403). Here, the marginal area refers to an image area having a common angle of view in two adjacent images.

  In the following description, the reference image 300 and the image 301 will be described as examples. For example, the control unit 101 determines that the camera 100 has moved in the horizontal right direction from the position of the camera 100 that has captured the reference image 300 and has captured the image 301 according to the motion detection result of the optical system 104. Then, the control unit 101 extracts the right end region of the reference image 300 and the left end region of the image 301 as a common margin region by the image processing unit 107.

  FIG. 5 is a diagram showing marginal area extraction performed by the camera 100 shown in FIG.

  In the example shown in FIG. 5, the reference image 300 and the image 301 are each divided into a plurality of block units 500 and 503 indicated by broken lines (here, the numbers and sizes of the block units 500 and 503 in the images 300 and 301 are The same). Block units 501 and 504 indicated by solid lines indicate marginal areas in the reference image 300 and the image 301, respectively.

  In FIG. 5, one third of all the block units is extracted as the marginal area, but the marginal area may be changed according to the movement of the camera 100 in the horizontal right direction. For example, if the magnitude of the horizontal rightward movement is greater than or equal to a predetermined value, the control unit 101 increases the margin range (for example, half of all block units). If the amount of horizontal rightward movement is less than a predetermined value, the control unit 101 reduces the marginal area (for example, a quarter of all blocks). Then, the control unit 101 transmits a marginal area indicated by a solid line to the PC 200 by the communication unit 109.

  Subsequently, the control unit 101 causes the image processing unit 107 to perform weighted addition synthesis processing on the images 300 and 301 based on the motion detection result of the optical system 104 (step S404: image connection). Here, the control unit 101 determines a synthesis region in the reference image 300 and the image 301 based on the motion detection result.

  FIG. 6 is a diagram for explaining an image composition process performed by the camera shown in FIG. 1. FIG. 6A shows a reference image, and FIG. 6B shows an image other than the reference image. FIG. 6C is a diagram showing a synthesized image obtained by synthesizing two images, and FIG. 6D is a diagram showing a synthesized image obtained by synthesizing all images.

  In the images 300 and 301 shown in FIGS. 6A and 6B, the image areas excluding the marginal areas 600 and 601 are approximate synthesis areas 602 and 603, respectively. Here, the boundary between the two synthesis regions 602 and 603 does not necessarily coincide with high accuracy. Then, the control unit 101 obtains a simple composite image 610 shown in FIG. 6C by connecting the two composite regions 602 and 603 by weighted average using the image processing unit 107. The simple composite image 610 is a confirmation composite image used for confirmation in the camera 100, and the recording composite image is generated as described later.

  Thereafter, the control unit 101 returns to the process of step S402. That is, the control unit 101 performs steps S402 to 404 until a simple composite image obtained by connecting all the images 300 to 303 is obtained. That is, after connecting the image 300 and the image 301, the control unit 101 extracts a marginal area for the images 301 and 302 in step S403 and transmits the marginal area to the PC 200. Then, as described above, the control unit 101 adds and averages the images 301 and 302 in step S404 to perform a combining process.

  Similarly, the control unit 101 performs the processes of steps S403 and S404 on the images 302 and 304, and then the control unit 101 performs the combining process on the three combined images in the same manner. As a result, all the images 300 to 303 are combined and a combined image 620 shown in FIG. 6D is obtained.

  When the image processing by the image processing unit 107 is completed for all of the images 301 to 303 (YES in step S402), that is, when the synthesis processing of the images 300 to 303 is completed, the control unit 101 stores the synthesized image 620 on the recording medium 108. Recording is performed (step S405). Then, the control unit 101 ends the image processing.

  FIG. 7 is a flowchart for explaining an example of the calculation of the projective transformation coefficient performed by the PC 200 shown in FIG. Note that the processing according to the illustrated flowchart is performed under the control of the control unit 201.

  First, the control unit 201 receives a marginal area of the images 300 to 303 through the communication unit 204 (step S410). Subsequently, the control unit 201 calculates a motion vector of the marginal area of the reference image 300 for each block unit described later (step S411).

  FIG. 8 is a diagram for explaining calculation of motion vectors performed by the PC 200 shown in FIG. FIG. 8A shows a reference image and a marginal area thereof, and FIG. 8B shows an image other than the reference image.

  In the reference image 300, the marginal area 701 is indicated by a thick line area. The control unit 201 divides the marginal area 701 in the reference image 300 into a plurality of blocks, performs pattern matching with the marginal area of the image 301 for each of these divided blocks, and calculates a motion vector for each block. That is, the control unit 201 sets the position where the sum of the absolute differences (SAD) is the smallest for each divided block unit as the movement destination coordinate in the block unit.

  However, since motion vectors with low reliability may be calculated depending on the block unit, it is necessary to exclude these motion vectors with low reliability. For example, in a low-contrast region that does not include an edge, many positions where the SAD is minimum may appear. For this reason, when the integral value of the edge signal is smaller than a predetermined value as is known, the motion vector is excluded.

  Furthermore, since the reference image 300 includes a region that does not exist in the reference image 301, it is necessary to remove a motion vector from the non-existing region. Here, the control unit 201 selects a predetermined number of motion vectors in ascending order of SAD values for each block unit, and excludes motion vectors related to blocks having no destination coordinates.

  Referring to FIG. 7 again, the control unit 201 obtains a projective transformation coefficient (synthesis information) based on the motion vector, and transmits the projective transformation coefficient to the camera 100 (step S412). Similarly, the control unit 201 calculates a motion vector for each block of the marginal area for the images 301 to 302 and transmits a projective transformation coefficient corresponding to the motion vector to the camera 100. Then, the PC ends the calculation of the projective transformation coefficient.

  Here, a method for calculating the projective transformation coefficient performed by the PC 200 shown in FIG. 2 will be described.

  FIG. 9 is a diagram for explaining an example of the calculation of the projective transformation coefficient performed by the PC 200 shown in FIG.

  Now, assume that the center position of the reference image 300 shown in FIG. 8A is (0, 0), and the center position of the block unit in the marginal area 701 is (x ′, y ′). In the illustrated example, the marginal area 701 is divided into 72 block units, the center position of the block unit located in the upper left corner is represented by (x0 ′, y0 ′), and the block unit located in the upper right corner is represented by (x0 ′, y0 ′). The center position is represented by (x5 ′, y0 ′). The center position of the block unit located at the lower left corner is represented by (x0 ′, y11 ′), and the center position of the block unit located at the lower right corner is represented by (x11 ′, y11 ′).

  In the image 301, the coordinates (x ′, y ′) of the reference image 300 (that is, the marginal area 701) corresponding to the movement destination coordinates (x, y) are expressed by the projective transformation represented by the following (1) and Expression (2). Is done.

  In Expressions (1) and (2), a0 to a7 indicate projective conversion coefficients, and the projective conversion coefficients a7 to a0 are expressed by the following Expressions (3) and (4).

In Formula (3) and Formula (4), h ₀ , h _k , X, and b _{0 to} b ₇ are represented by the following Formula (5) to Formula (15).

  Here, N indicates the number of elements for calculating a motion vector, and N = 5 in the illustrated example. Using the above equation, the control unit 201 calculates the projective transformation coefficient of the image 301 with respect to the reference image 300 as the first projective transformation coefficient. Similarly, the control unit 201 obtains projective transformation coefficients for the image 301, the image 302, the image 302, and the image 303, and sets them as second and third projective transformation coefficients, respectively. Then, the control unit 210 transmits the first to third projective transformation coefficients to the camera 100 through the communication unit 204.

  FIG. 10 is a flowchart for explaining an example of an image composition process using the projective transformation coefficient performed by the camera 100 shown in FIG. Note that the processing according to the illustrated flowchart is performed under the control of the control unit 101.

  In the camera 100, the control unit 101 receives the first to third projective transformation coefficients through the communication unit 109 (step S420). Subsequently, the control unit 101 determines whether or not the projective transformation process by the image processing unit 107 has been completed for all the images 301 to 303 other than the reference image (step S421). If the projection conversion processing has not been completed for all of the images 301 to 303 (NO in step S421), the control unit 101 performs the projection conversion processing for the images 301 to 303 using the first to third projection conversion coefficients. (Step S422).

  Here, the control unit 101 first performs a projective transformation process on the image 303 using the third projective transformation coefficient. As a result, the control unit 101 converts the marginal area of the image 303 so that the marginal area of the image 302 matches the marginal area of the image 302 and generates the converted image 303.

  Subsequently, the control unit 101 combines the converted image 303 and the image 302 by the image processing unit 107 (step S423). For example, the control unit 101 adds and averages the marginal areas and joins the image 302 and the converted image 303 to obtain a first composite image.

  Thereafter, the control unit 101 returns to the process of step S421. That is, the control unit 101 performs the processes of steps S421 to 423 until all the converted images 301 to 303 are connected to the reference image 300.

  Here, the control unit 101 performs a projective transformation process on the first synthesized image using the second projective transformation coefficient, and joins the synthesized image after the transformation process with the image 301 to obtain the second synthesized image. obtain. Then, the control unit 101 performs a projective transformation process on the second synthesized image using the first projective transformation coefficient, and joins the synthesized image after the transformation process with the reference image 301 to obtain a final synthesized image. obtain.

  FIG. 11 is a diagram illustrating an example of a final synthesized image obtained by the image synthesizing process using the projective transformation coefficient performed by the camera 100 illustrated in FIG.

  As shown in FIG. 11, it can be seen that the final composite image 710 has the images 301 to 303 joined together without causing a shift in the marginal area.

  When the projective transformation process is completed for all of the images 301 to 303 (YES in step S421), that is, when a final composite image is generated, the control unit 101 determines the final composite image (that is, a panoramic image). The simple composite image 620 recorded on the recording medium 108 is updated by (Step S424). Then, the control unit 101 ends the image composition process.

  As described above, in the first embodiment of the present invention, when a camera generates a composite image such as a panoramic image by combining a plurality of images, a projective transformation coefficient necessary for the combining process is set to an external device such as a PC. Therefore, even if the number of combined images increases, it is possible to prevent an increase in storage capacity and calculation amount when performing combining processing with the camera.

  Furthermore, in the camera 100, a simple composite image is generated by connecting a plurality of images in accordance with the marginal area before performing the composite processing using the projective transformation coefficient. The synthesis result can be confirmed in advance.

  Furthermore, since the camera 100 transmits the marginal area necessary for calculating the projective transformation coefficient to an external device such as a PC, an increase in the amount of communication data between the camera 100 and the external device can be suppressed.

In the above-described first embodiment, an example in which the camera 100 is swung in the horizontal direction and four images are continuously captured has been described. The synthesized image may be generated in the same manner when moving.
[Second Embodiment]

  Next, an example of a PC and a camera that are image processing apparatuses according to the second embodiment of the present invention will be described. The configurations of the camera and PC according to the second embodiment are the same as those of the camera and PC shown in FIGS. 1 and 2, respectively.

  In the second embodiment, a process of performing continuous shooting without moving the camera and removing a disturbing moving subject unintended by the user by the synthesis process will be described.

  FIG. 12 is a flowchart for explaining an example of an obstacle area extraction process performed by the camera 100 according to the second embodiment of the present invention.

  In the camera 100, predetermined image processing is performed on the image data obtained as a result of continuous shooting to obtain a plurality of pieces of image data (step S800).

  FIG. 13 is a diagram illustrating an example of image data obtained as a result of continuous shooting with the camera 100 according to the second embodiment of the present invention.

  In the illustrated example, it is assumed that images are captured in the order of the images 900 to 903, and the user continuously captures images without moving the camera 100.

  Subsequently, the control unit 101 selects, from the images 900 to 903, a reference image that is a reference image in the obstacle removal process described later (step S801). When selecting the reference image, for example, the control unit 101 selects the reference image according to the user's selection. Note that the control unit 101 may select the first image as the reference image. Here, it is assumed that the control unit 101 selects the image 900 as the reference image.

  Next, the control unit 101 extracts an obstacle area in the reference image and other images, and transmits the obstacle area to the PC 200 by the communication unit 109 (step S802). When extracting the obstacle area, for example, the user displays a reference image on an LCD provided in the camera 100 and designates at least one range with a rectangular frame. Based on the designated range, the control unit 101 extracts an area corresponding to the designated range from the reference image 900 and the other images 901 to 903 as an obstacle area.

  FIG. 14 is a diagram illustrating an example of a baffle area extracted by the camera 100 according to the second embodiment of the present invention.

  In the illustrated example, a baffle area 1000 including a car and a baffle area 1001 including a person are extracted from the reference image 900, and the control unit 101 includes a baffle area 1000 and 1001 in each of the images 901 to 903. A corresponding region is extracted at the same position. Then, as described above, the control unit 101 transmits all areas extracted as the disturbing areas 1000 and 1001 to the PC 200.

  Thereafter, the control unit 101 records the reference image 900 on the recording medium 108 (step S803), and ends the obstacle area extraction processing.

  FIG. 15 is a flowchart for explaining an example of an image composition process performed by a PC used with the camera according to the second embodiment of the present invention.

  First, the control unit 201 receives a disturbing area in the images 900 to 903 through the communication unit 204 (step S810). Subsequently, the control unit 201 determines whether or not the image processing has been completed for the obstacle area in all the images 901 to 903 other than the reference image 900 (step S811). If the image processing has not been completed (NO in step S811), the control unit 201 performs projective transformation on the obstacle area of the image 901 based on the obstacle area of the reference image 900 (step S812). Note that a method similar to the method described in the first embodiment is used when calculating the projective transformation coefficient.

  Subsequently, the control unit 201 performs processing for removing and compositing the moving subject area (that is, a car and a person) based on the reference image 900 and the projective transformed image (converted image) 901 (step S813). Mobile body removal synthesis).

  As a synthesis method used here, for example, a method described in Japanese Patent Application Laid-Open No. 2012-119817 is used. For example, the control unit 201 divides the reference image 900 into a plurality of blocks, and performs block matching with another image 901 for each block. Accordingly, the control unit 201 calculates a motion vector for each block and detects the contour of the moving subject.

  Next, the control unit 201 performs pattern matching processing on the moving subject in the reference image 900 and the obstacle area in the image 901 to determine whether or not the moving subject exists in the obstacle area in the image 901. Then, according to the determination result, the control unit 201 removes the moving subject by replacing the moving subject area (that is, the obstacle area) of the reference image 900 with an area where there is no moving subject. As a result, as the number of images other than the reference image increases, there is a higher possibility that there will be a region where no moving subject exists, and the probability that an obstacle will be removed increases.

  Referring to FIG. 15 again, after the process of step S813, control unit 101 returns to the process of step S811. That is, the control unit 101 performs the processes of steps S811 to 813 until projective transformation and moving object removal / combination are performed on the obstructive areas of the images 901 to 903. When the image processing is completed for all of the disturbing areas of the images 901 to 903 (YES in step S811), the control unit 201 causes the communication unit 204 to display the combined image (disturbed area combined image) generated in the process of step S813. It transmits to the camera 100 (step S814). Then, the control unit 201 ends the image composition process.

  FIG. 16 is a flowchart for explaining an image composition process performed by the camera 100 according to the second embodiment of the present invention.

  First, the control unit 101 receives a disturbing area composite image from the PC 200 through the communication unit 109 (step S820). Subsequently, the control unit 101 reads the recorded reference image 900 from the recording medium 108, replaces the disturbing area extracted in step S802 described above by the image processing unit 107 with the received disturbing area composite image, and A composite image from which the object has been removed is obtained (step S821).

  FIG. 17 is a diagram illustrating an example of a composite image obtained by the camera 100 according to the second embodiment of the present invention.

  As shown in the figure, it can be seen that the moving object (that is, the car and the person) that is an obstacle is removed from the composite image.

  Subsequently, the control unit 101 records the composite image obtained as described above as a stored image on the recording medium 108, and updates the recorded image (step S822). Then, the control unit 101 ends the image composition process.

  As described above, in the second embodiment of the present invention, the projection transformation coefficient necessary for generating a composite image by removing an obstacle area using a plurality of images is calculated by an external device such as a PC. Since the conversion is performed, the camera need only hold the reference image. As a result, it is possible to prevent an increase in the storage capacity and calculation amount of the camera even if the number of combined images increases.

  Furthermore, since the camera 100 is configured to transmit an obstacle area necessary for calculating the projective transformation coefficient to an external device such as a PC, an increase in the amount of communication data between the camera 100 and the external device can be suppressed.

  As is clear from the above description, in the example shown in FIG. 1, at least the control unit 101, the image processing unit 107, and the communication unit (transmission means) 109 constitute an image processing apparatus. The control unit 101 and the image processing unit 107 function as an extraction unit and a synthesis unit, and the control unit 101 functions as a recording unit. Furthermore, the optical system 104, the imaging unit 105, and the A / D conversion unit 106 function as an imaging unit.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, Various forms of the range which does not deviate from the summary of this invention are also contained in this invention. .

  For example, the function of the above embodiment may be used as a control method, and this control method may be executed by the image processing apparatus. In addition, a program having the functions of the above-described embodiments may be used as a control program, and the control program may be executed by a computer included in the image processing apparatus. The control program is recorded on a computer-readable recording medium, for example.

100 Camera 101, 201 Control unit 102, 202 ROM
103, 203 RAM
DESCRIPTION OF SYMBOLS 104 Optical system 105 Imaging part 107,206 Image processing part 108,205 Recording medium 109,204 Communication part 200 Personal computer (PC)

Claims (10)

An image processing apparatus that obtains a composite image by combining a plurality of images obtained as a result of continuous shooting,
Extraction means for extracting a predetermined area as an extraction area for each of the plurality of images;
Transmitting means for transmitting the extraction region to an external device;
The external device receives synthesis information obtained based on the extraction area and used to synthesize the plurality of images in the external device from the external device, and synthesizes the plurality of images according to the synthesis information. A synthesis means to obtain
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the extraction unit extracts a marginal area, which is a common area in adjacent images, as the predetermined area.   The external device divides the marginal area into a plurality of blocks, obtains a motion vector for each block in the adjacent images, and calculates a projective transformation coefficient used in a projective transformation process according to the motion vector. The image processing apparatus according to claim 2, wherein the image processing apparatus is obtained as composite information.   The synthesizing unit generates one of the plurality of images as a reference image serving as a conversion destination in the projective transformation process, and combines the plurality of images according to the projective transformation coefficient to generate the synthesized image. The image processing apparatus according to claim 3.   The extraction means extracts one of the plurality of images as a reference image and a specified range specified in the reference image as an obstacle area that is the predetermined area, and other images excluding the reference image The image processing apparatus according to claim 1, wherein an area corresponding to the specified range is extracted as an obstructive area.   The external device performs projective transformation on the disturbing areas of other images excluding the reference image based on the disturbing areas of the reference image, and removes the subject existing in the disturbing areas to generate the composite information. The image processing apparatus according to claim 5, wherein:   The image processing apparatus according to claim 6, wherein the synthesizing unit generates the synthesized image by replacing an area of an obstacle in the reference image with the synthesized information. Imaging means for imaging the plurality of images;
The image processing apparatus according to any one of claims 1 to 7,
Recording means for recording the composite image on a recording medium;
An imaging device comprising: There is a control method in an image processing apparatus that combines a plurality of images obtained as a result of continuous shooting to obtain a composite image,
An extraction step of extracting a predetermined area as an extraction area for each of the plurality of images;
A transmitting step of transmitting the extraction region to an external device;
The external device receives synthesis information obtained based on the extraction area and used to synthesize the plurality of images in the external device from the external device, and synthesizes the plurality of images according to the synthesis information. A synthesis step to obtain
A control method characterized by comprising: A control program used in an image processing apparatus that obtains a composite image by combining a plurality of images obtained as a result of continuous shooting,
In the computer provided in the image processing apparatus,
An extraction step of extracting a predetermined area as an extraction area for each of the plurality of images;
A transmitting step of transmitting the extraction region to an external device;
The external device receives synthesis information obtained based on the extraction area and used to synthesize the plurality of images in the external device from the external device, and synthesizes the plurality of images according to the synthesis information. A synthesis step to obtain
A control program characterized by causing