JP2013223080A - Image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of increasing an angle of view of a dynamic picture image obtained in parallel with a pan/tilt action.SOLUTION: A CPU 26, when a plurality of frames of image data each corresponding to a plurality of optical images captured by an image sensor 16 are obtained in parallel with a pan/tilt action of an imaging surface, designates one of the obtained plurality of frames of image data. The CPU 26 specifies an image data having one frame or two frames and more, among the obtained plurality of frames of image data, which has duplication with the designated image data greater than a standard, and creates composite image data which has enlarged angle of view corresponding to the standard by combining the specified image data and the designated image data. The CPU 26 also refreshes the designation target every time the creation of composite image data has been completed.

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that creates a moving image having an enlarged angle of view.

  An example of this type of device is disclosed in Patent Document 1. According to this background art, the plurality of imaging units have a structure capable of photographing different directions. Video signals taken from different directions at the same time output from the respective imaging units are synthesized, and panoramic video data is created by the synthesis processing unit. The panoramic signal data obtained from the synthesis processing unit is extracted and output.

JP 2002-359664 A

  However, in the background art, it is necessary to prepare a plurality of imaging devices and record a moving image with an enlarged angle of view from the time of shooting, and even when a wide field of view is shot by pan / tilt operation, it is already acquired. Enlarging the angle of view of a moving image is not described.

  Therefore, a main object of the present invention is to provide an image processing apparatus capable of expanding the angle of view of a moving image acquired in parallel with the pan / tilt operation.

  An image processing apparatus according to the present invention (10: reference numeral corresponding to the embodiment; hereinafter the same) obtains a plurality of images captured on the imaging surface in parallel with the pan / tilt operation of the imaging surface (16) , Designation means for designating any one of a plurality of images obtained by the obtaining means (S45), and the overlap with the image designated by the designation means among the plurality of images obtained by the obtaining means exceeds the reference 1 Alternatively, creating means (S61 to S81, S85 to S95, S99, S101, S111 to S131) that composes two or more images and an image designated by the designation means to create a composite image having an enlarged angle of view corresponding to the reference. , S135 to S145, S149, S151), and update means (S55) for updating the designation target of the designation means every time processing of the creation means is completed.

  Preferably, the creation means includes first image detection means (S61 to S79, S89 to S93) for detecting an image exceeding a reference from one or more images preceding the image designated by the designation means, and the designation means. Second image detecting means (S111 to S129, S139 to S143) for detecting an image exceeding the reference from one or two or more images delayed by the image designated by.

  Preferably, based on the direction detection means (S33 to S43) for detecting the operation direction of the pan / tilt operation using two or more of the plurality of images acquired by the acquisition means, and the detection result of the direction detection means. Position adjustment means (S83, S97, S133, S147) for adjusting the composite position of the composite image is further provided.

  Preferably, file creation means (S47, S53, S59) for creating an image file storing a plurality of composite images created by the creation means is further provided.

  Preferably, size adjustment means (S103, S107, S109, S153, S157, S159) for adjusting the size of the composite image created by the creation means is further provided.

  An image processing program according to the present invention is provided in a processor (26) of an image processing apparatus (10) including an acquisition unit (16) that acquires a plurality of images captured on an imaging surface in parallel with pan / tilt operations on the imaging surface. A designation step for designating any one of the plurality of images acquired by the acquisition means (S45), and the overlap with the image specified by the specification step among the plurality of images acquired by the acquisition means exceeds the reference 1 Alternatively, a creation step (S61 to S81, S85 to S95, S99, S101, S111 to S131) that creates a composite image having an enlarged angle of view corresponding to the reference by combining two or more images and the image specified in the specification step. , S135 to S145, S149, S151), and an update process (S55) for updating the designation target of the designation step every time the creation step is completed.

  The image processing method according to the present invention is an image processing executed by an image processing device (10) comprising an acquisition means (16) for acquiring a plurality of images captured on an imaging surface in parallel with a pan / tilt operation of the imaging surface. A specifying step (S45) for designating any one of a plurality of images acquired by the acquiring means, and an overlap with the image specified by the specifying step among the plurality of images acquired by the acquiring means. A creation step (S61 to S81, S85 to S95, S99, S101) that combines one or more images exceeding the reference and the image specified in the specifying step to create a composite image having an enlarged angle of view corresponding to the reference , S111 to S131, S135 to S145, S149, S151), and an update step (S55) for updating the designation target of the designation step every time the process of the creation step is completed.

  The external control program according to the present invention follows an acquisition means (16) for acquiring a plurality of images captured on the imaging surface in parallel with the pan / tilt operation of the imaging surface, and an internal control program stored in the memory (44). An external control program supplied to an image processing apparatus (10) comprising a processor (26) for performing processing, wherein one of a plurality of images acquired by the acquisition means is specified (S45); Of the plurality of images acquired by the acquisition means, one or two or more images whose overlap with the image specified by the specifying step exceeds the reference and the image specified by the specifying step are combined and an enlarged image corresponding to the reference Creation step (S61 to S81, S85 to S95, S99, S101, S111 to S131, S135 to S145, S149, S151) for creating a composite image with corners, and designation of the designated step every time the creation step is completed versus For execution by the processor updating step of updating (S55) in cooperation with the internal control program, which is an external control program.

  The image processing apparatus (10) according to the present invention includes an acquisition means (16) for acquiring a plurality of images captured on the imaging surface in parallel with the pan / tilt operation of the imaging surface, and an acquisition means (60) for acquiring an external control program. And an image processing apparatus comprising a processor (26) that executes processing according to the external control program captured by the capturing unit and the internal control program stored in the memory (44), wherein the external control program is acquired by the acquiring unit. A designation step (S45) for designating any one of the plurality of acquired images, and one or two or more of the plurality of images acquired by the acquisition means that overlap with the image specified by the specification step exceeds a reference A creation step (S61 to S81, S85 to S95, S99, S10) that creates a composite image having an enlarged angle of view corresponding to the reference by combining the image and the image specified in the specifying step. 1, S111 to S131, S135 to S145, S149, S151), and a program for executing the update step (S55) for updating the designated target of the designated step every time the creation step processing is completed in cooperation with the internal control program It corresponds to.

  According to the present invention, the image combined with the designated image corresponds to an image whose overlap with the designated image exceeds the reference, and the synthesized image corresponds to an image having an enlarged angle of view corresponding to the reference. The designated image is updated every time a composite image is created. As a result, the angle of view of the moving image acquired in parallel with the pan / tilt operation can be enlarged.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

It is a block diagram which shows the basic composition of one Example of this invention. It is a block diagram which shows the structure of one Example of this invention. It is an illustration figure which shows pan operation in video recording. It is an illustration figure which shows an example of the some frame image stored in the moving image file. It is an illustration figure which shows an example of a to-be-superimposed determination area | region. It is an illustration figure which shows an example of a superimposition determination area | region. It is an illustration figure which shows a part of image composition process. It is an illustration figure which shows the other part of an image synthesis process. It is an illustration figure which shows an example of the some frame image which comprises a panoramic video. It is a flowchart which shows a part of operation | movement of CPU applied to the FIG. 2 Example. It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; FIG. 10 is a flowchart showing yet another portion of behavior of the CPU applied to the embodiment in FIG. 2; It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; FIG. 10 is a flowchart showing yet another portion of behavior of the CPU applied to the embodiment in FIG. 2; It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; FIG. 10 is a flowchart showing yet another portion of behavior of the CPU applied to the embodiment in FIG. 2; It is a block diagram which shows the structure of the other Example of this invention.

Embodiments of the present invention will be described below with reference to the drawings.
[Basic configuration]

  Referring to FIG. 1, the image processing apparatus of this embodiment is basically configured as follows. The acquisition unit 1 acquires a plurality of images captured on the imaging surface in parallel with the pan / tilt operation of the imaging surface. The designation unit 2 designates any one of a plurality of images acquired by the acquisition unit 1. The creating unit 3 synthesizes one or more images whose overlap with the image specified by the specifying unit 2 out of the plurality of images acquired by the acquiring unit 1 exceeds the reference and the image specified by the specifying unit 2. Thus, a composite image having an enlarged angle of view corresponding to the reference is created. The updating unit 4 updates the designation target of the designation unit 2 every time the processing of the creation unit 3 is completed.

The image combined with the designated image corresponds to an image whose overlap with the designated image exceeds the reference, and the composite image corresponds to an image having an enlarged angle of view corresponding to the reference. The designated image is updated every time a composite image is created. As a result, the angle of view of the moving image acquired in parallel with the pan / tilt operation can be enlarged.
[Example]

  Referring to FIG. 2, the digital video camera 10 of this embodiment includes a focus lens 12 and an image sensor 16 driven by a driver 18. The optical image of the scene that has passed through these members is irradiated onto the imaging surface of the image sensor 16 and subjected to photoelectric conversion. As a result, a charge representing the scene is generated.

  When the entire system is activated, the CPU 26 determines the state of the mode change button 28md provided on the key input device 28 (that is, the current operation mode) under the main task, and activates the imaging task corresponding to the imaging mode. On the other hand, a playback task is started corresponding to the playback mode.

  When the imaging task is activated, the CPU 26 instructs the driver 18 to repeat the exposure operation and the charge readout operation in order to execute the moving image capturing process. In response to a vertical synchronization signal Vsync periodically generated from an SG (Signal Generator) (not shown), the driver 18 exposes the imaging surface and reads out the charges generated on the imaging surface in a raster scanning manner. From the image sensor 16, raw image data based on the read charges is periodically output.

  The signal processing circuit 20 performs processing such as white balance adjustment, color separation, and YUV conversion on the raw image data output from the image sensor 16, and the generated YUV format image data is sent to the SDRAM 32 through the memory control circuit 30. Write. The LCD driver 36 repeatedly reads out the image data stored in the SDRAM 32 through the memory control circuit 30 and drives the LCD monitor 38 based on the read image data. As a result, a moving image representing a scene is displayed on the monitor screen.

  When a recording start operation is performed toward the recording button 28rec provided on the key input device 28, the CPU 26 activates the MP4 codec 46 and the I / F 40 under the imaging task in order to start the recording process. The MP4 codec 46 reads the image data stored in the SDRAM 32 through the memory control circuit 30, and compresses the read image data according to the MPEG4 system. The compressed image data, that is, MP4 data is written into the SDRAM 32 by the memory control circuit 30. The I / F 40 reads the MP4 data stored in the SDRAM 32 through the memory control circuit 30 and writes the read MP4 data to the image file created on the recording medium 42.

  When a recording end operation is performed toward the key input device 28, the CPU 26 stops the MP4 codec 46 and the I / F 40 to end the recording process. Also, the end process is performed on the moving image file of the writing destination.

  When the reproduction task is activated, the CPU 26 designates the moving image file recorded on the recording medium 42 and instructs the I / F 40 to reproduce the first frame image. The I / F 40 reads the image data of the first frame from the designated moving image file, and writes the read image data to the SDRAM 32 through the memory control circuit 30.

  The LCD driver 36 reads the image data stored in the SDRAM 32 through the memory control circuit 30, and drives the LCD monitor 38 based on the read image data. As a result, the top frame image is displayed on the LCD monitor 38.

  When a reproduction start operation is performed toward the key input device 28, the CPU 26 instructs the I / F 40 to start moving image reproduction. The I / F 40 periodically reads image data of a plurality of frames stored in the designated moving image file, and writes the read image data to the SDRAM 32 through the memory control circuit 30. The image data stored in the SDRAM 32 is read by the LCD driver 36, and as a result, a moving image is displayed on the LCD monitor 38.

  When a playback end operation is performed toward the key input device 28 or when the playback image reaches the end frame image of the designated moving image file, the CPU 26 instructs the I / F 40 to end the playback of the moving image, and further plays back the top frame image. Is commanded to the I / F 40. As a result, the first frame image of the designated moving image file is displayed again on the LCD monitor 38.

  When a feed operation is performed toward the key input device 28, the CPU 26 designates the next image file and instructs the I / F 40 to reproduce the first frame image. The I / F 40 executes the same processing as described above, whereby the first frame image stored in the next moving image file is displayed on the LCD monitor 38.

  When a panoramic video creation operation is performed toward the key input device 28 in a state where the top frame image of the designated video file is displayed on the LCD monitor 38, the CPU 26 executes a panoramic video creation process.

  When a moving image is shot under the above-described imaging task while the operator rotates the housing of the digital video camera 10 in the pan direction, a through image displayed on the LCD monitor 38 during shooting is shown in FIG. Transition as shown. In this case, the moving image file created on the recording medium 42 stores a plurality of frame images FRa ... FRm ... FRz shown in FIG. For such a moving image file, panoramic moving image creation is executed as follows.

  First, the CPU 26 detects the total number of frames of the designated moving image file. Next, the CPU 26 performs an overlay process using the first frame image and the 10th frame image in order to detect the direction of the pan operation during shooting. In the overlay process, the overlay determination area AR1 is assigned to the center of one image (see FIG. 5), and the overlay determination area AR2 is assigned to the center of the other image (see FIG. 6). For example, 90% of the entire image is allocated to the overlay determination area AR1. An area that is narrower than the overlay determination area AR1 is allocated to the overlay determination area AR2, for example, an area that is 80% of the entire image is allocated.

  The CPU 26 executes a superimposition process using such two images. The position where the image belonging to the overlay determination area AR2 matches the part of the image belonging to the overlay determination area AR1 is specified by the overlay process. Based on the position specified in this way, the direction of the pan operation is detected. When the direction is detected, the CPU 26 newly creates a moving image file on the recording medium 42. The created movie file is opened. Next, image composition processing of the strip image RG in the forward direction and the backward direction with respect to the detected pan operation direction is executed for each frame image stored in the designated moving image file.

  In the image composition process, one or two or more strip images RG are synthesized into a frame image in each direction. In order to create such a strip image RG, an overlay process is executed using two frame images whose frames are adjacent to each other, and a positional deviation amount GP between both images is detected. The CPU 26 sequentially executes such a superimposition process in the advancing direction or the retreating direction from the target frame image of the image synthesis process, and calculates the cumulative sum RW of the obtained deviation amounts GP.

  When the calculated cumulative sum RW exceeds the strip width predetermined value RWFIX, the CPU 26 uses a frame image farther from the frame image to be subjected to the image synthesis process among the frame images used in the immediately preceding overlay process. Thus, a strip image RG having a strip width RWFIX is created. The created strip image RG is cumulatively combined with the end of the frame image.

  Referring to FIG. 7, when frame image FRm is a target of image synthesis processing, a shift amount GP1 between frame image FRm and frame image FRm + 1 adjacent in the traveling direction is first calculated. However, since the calculated shift amount GP1 is smaller than the strip width RWFIX, the shift amount GP2 between the frame image FRm + 1 and the frame image FRm + 2 is calculated. The sum of displacement GP1 + GP2 is larger than the strip width RWFIX. Therefore, the strip image RGf1 in the traveling direction is created using the frame image FRm + 2. The created strip image RGf1 is combined with the frame image FRm.

  The strip image RG is repeatedly created so that the total of the strip widths RWFIX in the traveling direction and the backward direction corresponds to the predetermined value SZFIX, and is cumulatively combined with the end of the frame image. The default value SZFIX is determined by the angle of view of the panoramic video to be created. For example, when creating a panoramic video with a 16: 9 angle of view based on a video file recorded with a 4: 3 angle of view, the default value SZFIX is 1/6 of the width of the original frame image. The strip width RWFIX is determined to be a value obtained by dividing the predetermined value SZFIX by an integer.

  Each time image composition processing in the forward direction and backward direction is completed for one frame image, the frame image subjected to the image composition processing is written to the open moving image file through the I / F 40.

  According to the example shown in FIG. 8, the strip images RGf1, RGf2, and RGf3 are synthesized on the right side that is the traveling direction, and the strip images RGr1, RGr2, and RGr3 are synthesized on the left side that is the backward direction with respect to the frame image FRm. The The total strip width RWFIX × 3 is equal to SZFIX.

  When the image synthesis processing in the advancing direction and the backward direction is completed for all the frame images stored in the designated moving image file, the open moving image file is closed and the moving image file storing the panoramic moving image is completed. The completed moving image file is in a reproduction waiting state as a specified moving image file under the reproduction task.

  When a moving image file storing a plurality of frame images FRa... FRm... FRz shown in FIG. 4 is designated and a panoramic moving image creation process is executed, a plurality of panoramic size frame images FRCa. A moving image file storing FRCm... FRCz is newly created on the recording medium 42.

  As shown in FIG. 9, for the frame images around the first frame and the last frame, blackout processing is performed on an area where a strip image cannot be created using the previous and next frame images.

  The CPU 26 executes a plurality of tasks including the reproduction task shown in FIGS. Note that control programs corresponding to these tasks are stored in the flash memory 44.

  Referring to FIG. 10, in step S1, the latest moving image file recorded on the recording medium 42 is designated, and in step S3, the I / F 40 is instructed to reproduce the first frame image. As a result, the first frame image of the designated moving image file is displayed on the LCD monitor 38.

  In step S5, it is determined whether or not a reproduction start operation has been performed. If the determination result is NO, it is determined in step S7 whether or not a feed operation has been performed. If the determination result in step S7 is NO, the process proceeds to step S17. If the determination result in step S7 is YES, the next moving image file is designated in step S15, and then the process returns to step S3. As a result, the first frame image of the next moving image file is displayed on the LCD monitor 38.

  If the determination result in step S5 is YES, the I / F 40 is commanded to start moving image reproduction in step S9. As a result, the moving image stored in the designated moving image file is displayed on the LCD monitor 38.

  In step S11, it is repeatedly determined whether or not a reproduction end operation has been performed or a logical sum condition that the reproduced image has reached the end frame image is satisfied. When the determination result is updated from NO to YES, the I / F 40 is commanded to end moving image reproduction in step S13, and when the process of step S13 is completed, the process returns to step S3.

  In step S17, it is determined whether or not a panoramic video creation operation has been performed. If the determination result is NO, the process returns to step S5. If the determination result is YES, a panoramic video generation process is executed in step S19.

  In step S21, it is determined whether or not the panoramic video has been successfully created by the process in step S19. If the determination result is NO, the process returns to step S3, whereas if the determination result is YES, the process proceeds to step S23. In step S23, the panoramic video file created in step S19 is designated, and the process returns to step S3. As a result, the top frame image of the panoramic video file is displayed on the LCD monitor 38.

  The panoramic video creation process in step S19 is executed according to a subroutine shown in FIGS. In step S31, the total number of frames of the designated moving image file is detected. In step S33, a superimposition process is executed using the first frame image and the tenth frame image in order to detect the direction of the pan operation during shooting.

  In step S35, it is determined whether or not the operation direction has been successfully detected by the process in step S33. If the determination result is NO, an error is notified in step S37, and then the process returns to the upper hierarchy routine. If a determination result is YES, it will progress to Step S39.

  In step S39, it is determined whether or not the operation direction detected by the process in step S33 is to move to the left. If the determination result is YES, the process proceeds to step S45 through the process in step S41, and the determination result is If NO, the process proceeds to step S45 through step S43.

  In step S41, “0” is set to the variable DRT, “1” is set to the variable DRT in step S43, and “1” is set to the variable N in step S45. In step S47, a moving image file is newly created on the recording medium. The created movie file is opened.

  In step S49, an image composition process in the traveling direction of the operation direction is performed on the Nth frame image, and in step S51, an image composition process in the backward direction of the operation direction is performed on the Nth frame image. In step S53, the frame image subjected to the processing in steps S49 and S51 is written to the open moving image file through the I / F 40.

  In step S55, the variable N is incremented, and it is determined in step S57 whether or not the variable N exceeds the total number of frames detected in step S31. If the determination result is NO, the process returns to step S49, while if the determination result is YES, the open moving image file is closed in step S59. When the processing in step S59 is completed, the process returns to the upper hierarchy routine.

  The image composition process in the advancing direction in step S49 is executed according to a subroutine shown in FIGS. Referring to FIG. 14, in step S61, the value indicated by variable N is set to variable M, and in step S63, variable SZ is set to “0”. In step S65, the value indicated by the variable M is set to the variable L, and in step S67, "0" is set to the variable RW.

  In step S69, it is determined whether or not the total number of frames of the designated moving image file detected in step S31 is equal to the variable L. If the determination result is YES, the process proceeds to step S103. If the determination result is NO, step S71 is performed. Proceed to

  In step S71, an overlay process is executed using the Lth frame image and the (L + 1) th frame image. In step S73, the variable RW and the sum of the deviation amounts obtained in step S71 are set in the variable RW. In step S75, it is determined whether or not the variable RW is equal to or larger than the strip width predetermined value RWFIX. If the determination result is YES, the process proceeds to step S81, and if the determination result is NO, the process proceeds to step S77.

  In step S77, the variable L is incremented. In step S79, it is determined whether or not the variable L + 1 exceeds the total number of frames of the designated moving image file detected in step S31. If the determination result is NO, the process returns to step S71, while if the determination result is YES, the process proceeds to step S95. If the determination result in step S75 is YES, a strip image having a width of RWFIX is created in step S81 using the L + 1th frame image.

  In step S83, it is determined whether or not “0” is set in the variable DRT. If the determination result is YES, the process proceeds to step S89 via step S85, whereas if the determination result is NO, the process proceeds to step S89 via step S87. Proceed to

  In step S85, the image composition process is performed so that the strip image created in step S81 is added to the left side of the Nth frame image. In step S87, the image composition process is performed so as to be added to the right side of the Nth frame image. Execute.

  In step S89, the sum of the variable SZ and the predetermined value RWFIX of the strip width is set in the variable SZ. In step S91, it is determined whether or not the variable SZ is equal to or greater than the predetermined value SZFIX. If the determination result is YES, the process returns to the upper hierarchy routine. If the determination result is NO, the process proceeds to step S93. In step S93, the variable M is set to the value indicated by the variable L + 1, and then the process returns to step S65.

  In step S95, a strip image having a width of RW is created using the Lth frame image. In step S97, it is determined whether or not “0” is set in the variable DRT. If the determination result is YES, the process proceeds to step S103 via step S99. If the determination result is NO, the process proceeds to step S103 via step S101. Proceed to

  In step S99, the image composition process is performed so that the strip image created in step S95 is added to the left side of the Nth frame image. In step S101, the image composition process is performed so as to be added to the right side of the Nth frame image. Execute.

  In step S103, a strip image having a predetermined width SZFIX-variable RW is created by blackout processing. In step S105, it is determined whether or not “0” is set in the variable DRT. If the determination result is YES, the process proceeds to step S107, and if the determination result is NO, the process proceeds to step S109.

  In step S107, the image composition process is performed so that the strip image created in step S103 is added to the left side of the Nth frame image. In step S109, the image composition process is performed so as to be added to the right side of the Nth frame image. Execute. When the process of step S107 or S109 is completed, the process returns to the upper hierarchy routine.

  The image composition process in the backward direction in step S51 is executed according to a subroutine shown in FIGS. Referring to FIG. 17, in step S111, the value indicated by variable N is set to variable P, and in step S113, variable SZ is set to “0”. In step S115, the value indicated by the variable P is set to the variable Q, and in step S117, "0" is set to the variable RW.

  In step S119, it is determined whether or not the variable Q is equal to “1”. If the determination result is YES, the process proceeds to step S153, and if the determination result is NO, the process proceeds to step S121.

  In step S121, a superimposition process is performed using the Qth frame image and the Q-1th frame image. In step S123, the variable RW and the sum of the deviation amounts obtained in step S121 are set in the variable RW. In step S125, it is determined whether or not the variable RW is equal to or greater than the strip width default value RWFIX. If the determination result is YES, the process proceeds to step S131, and if the determination result is NO, the process proceeds to step S127.

  In step S127, the variable Q is decremented, and in step S129, it is determined whether or not the variable Q-1 is less than “1”. If the determination result is NO, the process returns to step S121, while if the determination result is YES, the process proceeds to step S145. If the determination result in step S125 is YES, a strip image having a width of RWFIX is created in step S131 using the Q-1th frame image.

  In step S133, it is determined whether or not “0” is set in the variable DRT. If the determination result is YES, the process proceeds to step S139 via step S135. If the determination result is NO, the process proceeds to step S139 via step S137. Proceed to

  In step S135, the image composition process is performed so that the strip image created in step S131 is added to the right side of the Nth frame image. In step S137, the image composition process is performed so as to be added to the left side of the Nth frame image. Execute.

  In step S139, the sum of the variable SZ and the predetermined value RWFIX of the strip width is set in the variable SZ. In step S141, it is determined whether or not the variable SZ is equal to or greater than the predetermined value SZFIX. If the determination result is YES, the process returns to the upper hierarchy routine. If the determination result is NO, the process proceeds to step S143. In step S143, the value indicated by the variable Q-1 is set in the variable P, and then the process returns to step S115.

  In step S145, a strip image having a width of RW is created using the Qth frame image. In step S147, it is determined whether or not “0” is set in the variable DRT. If the determination result is YES, the process proceeds to step S153 through step S149. If the determination result is NO, the process proceeds to step S153 through step S151. Proceed to

  In step S149, the image composition process is performed so that the strip image created in step S145 is added to the right side of the Nth frame image. In step S151, the image composition process is performed so as to be added to the left side of the Nth frame image. Execute.

  In step S153, a strip image having a predetermined width SZFIX-variable RW is created by blackout processing. In step S155, it is determined whether or not “0” is set in the variable DRT. If the determination result is YES, the process proceeds to step S157, and if the determination result is NO, the process proceeds to step S159.

  In step S157, the image composition process is performed so that the strip image created in step S153 is added to the right side of the Nth frame image. In step S159, the image composition process is performed so as to be added to the left side of the Nth frame image. Execute. When the process of step S157 or S159 is completed, the process returns to the upper hierarchy routine.

  As can be understood from the above description, when image data of a plurality of frames respectively corresponding to a plurality of optical images captured on the imaging surface is acquired in parallel with the pan / tilt operation of the imaging surface, the CPU 26 acquires One of the image data of a plurality of frames is designated (S45). The CPU 26 also specifies image data of one frame or two or more frames whose overlapping with the designated image data exceeds the reference from the acquired plural frames of image data, and synthesizes the specified image data with the designated image data. Thus, composite image data having an enlarged angle of view corresponding to the reference is created (S61 to S81, S85 to S95, S99, S101, S111 to S131, S135 to S145, S149, S151). Further, the CPU 26 updates the designation target every time the composite image data creation process is completed (S55).

  As described above, the image data combined with the designated image data corresponds to image data whose overlap with the designated image data exceeds the reference, and the combined image data corresponds to image data having an enlarged angle of view corresponding to the reference. The designated image data is updated every time composite image data is created. As a result, the angle of view of the moving image acquired in parallel with the pan / tilt operation can be enlarged.

  In this embodiment, a moving image file storing a panoramic moving image is newly created. However, when the original moving image file is reproduced without creating a new moving image file, a process for creating a panoramic image for display may be executed in parallel with the reproducing process.

  In this embodiment, a panoramic image is created in accordance with the pan operation. However, a panoramic image may be created according to a tilt operation.

  In this embodiment, the multitask OS and control programs corresponding to a plurality of tasks executed thereby are stored in the flash memory 44 in advance. However, a communication I / F 60 for connecting to an external server is provided in the digital camera 10 in the manner shown in FIG. 20, and some control programs are prepared as internal control programs in the flash memory 44 from the beginning, while other parts are provided. These control programs may be acquired from an external server as an external control program. In this case, the above-described operation is realized by cooperation of the internal control program and the external control program.

  In this embodiment, the process executed by the CPU 26 is divided into a plurality of tasks including the reproduction task shown in FIGS. However, these tasks may be further divided into a plurality of small tasks, and a part of the divided plurality of small tasks may be integrated with other tasks. Further, when the transfer task is divided into a plurality of small tasks, all or part of the transfer task may be acquired from an external server.

  In this embodiment, the digital video camera has been described. However, the present invention can also be applied to a digital still camera, a mobile phone terminal, a smartphone, or the like.

10…
10 ... Digital video camera 16 ... Image sensor 26 ... CPU
32 ... SDRAM
38 ... LCD monitor 42 ... Recording medium

Claims (9)

Acquisition means for acquiring a plurality of images captured on the imaging surface in parallel with the pan / tilt operation of the imaging surface;
Designating means for designating any one of a plurality of images obtained by the obtaining means;
Among the plurality of images acquired by the acquisition unit, one or two or more images whose overlap with the image specified by the specifying unit exceeds a reference and the image specified by the specifying unit are combined and used as the reference An image processing apparatus comprising: a creating unit that creates a composite image having a corresponding enlarged angle of view; and an updating unit that updates a designation target of the designation unit every time processing of the creating unit is completed.   The creation means is a first image detection means for detecting an image that exceeds the reference from one or more images preceding the image designated by the designation means, and 1 delayed from the image designated by the designation means. The image processing apparatus according to claim 1, further comprising: a second image detection unit that detects an image that exceeds the reference from two or more images. Direction detection means for detecting an operation direction of the pan / tilt operation using two or more images among a plurality of images acquired by the acquisition means, and composition of the composite image based on a detection result of the direction detection means The image processing apparatus according to claim 1, further comprising a position adjusting unit that adjusts the position.   The image processing apparatus according to claim 1, further comprising: a file creation unit that creates an image file that stores a plurality of composite images created by the creation unit.   The image processing apparatus according to claim 1, further comprising a size adjusting unit that adjusts a size of the composite image created by the creating unit. In a processor of an image processing apparatus including an acquisition unit that acquires a plurality of images captured on an imaging surface in parallel with a pan / tilt operation of the imaging surface,
A designation step of designating any one of a plurality of images acquired by the acquisition means;
Among the plurality of images acquired by the acquisition unit, one or two or more images whose overlap with the image specified by the specifying step exceeds a reference and the image specified by the specifying step are combined and used as the reference An image processing program for executing a creation step of creating a composite image having a corresponding enlarged angle of view, and an update step of updating a designation target of the designation step every time the process of the creation step is completed. An image processing method executed by an image processing apparatus including an acquisition unit that acquires a plurality of images captured on an imaging surface in parallel with a pan / tilt operation of the imaging surface,
A designation step of designating any one of a plurality of images acquired by the acquisition means;
Among the plurality of images acquired by the acquisition unit, one or two or more images whose overlap with the image specified by the specifying step exceeds a reference and the image specified by the specifying step are combined and used as the reference An image processing method comprising: a creation step of creating a composite image having a corresponding enlarged angle of view; and an update step of updating a designation target of the designation step every time the creation step is completed. Supplied to an image processing apparatus comprising acquisition means for acquiring a plurality of images captured on the imaging surface in parallel with the pan / tilt operation of the imaging surface, and a processor for executing processing according to an internal control program stored in a memory An external control program,
A designation step of designating any one of a plurality of images acquired by the acquisition means;
Among the plurality of images acquired by the acquisition unit, one or two or more images whose overlap with the image specified by the specifying step exceeds a reference and the image specified by the specifying step are combined and used as the reference A creation step for creating a composite image having a corresponding enlarged angle of view, and an update step for updating the designation target of the designation step each time the creation step is completed, in cooperation with the internal control program, in the processor External control program to be executed. Acquisition means for acquiring a plurality of images captured on the imaging surface in parallel with the pan / tilt operation of the imaging surface;
An image processing apparatus comprising: a capturing unit that captures an external control program; and a processor that executes processing according to the external control program captured by the capturing unit and an internal control program stored in a memory,
The external control program is
A designation step of designating any one of a plurality of images acquired by the acquisition means;
Among the plurality of images acquired by the acquisition unit, one or two or more images whose overlap with the image specified by the specifying step exceeds a reference and the image specified by the specifying step are combined and used as the reference A program for creating a composite image having a corresponding enlarged angle of view, and an update step for updating a designation target of the designation step in cooperation with the internal control program every time processing of the creation step is completed An image processing apparatus equivalent to

Images