JP5836109B2 - Image processing apparatus, imaging apparatus including the same, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus capable of generating an image in which a plurality of images are arranged in a tile shape, an imaging apparatus including the image processing apparatus, an image processing method, and an image processing program.

  The digital camera can record a captured image on a recording medium. A technique is known that provides means for easily confirming recorded images by reducing a plurality of recorded images and displaying them simultaneously as index images (see, for example, Patent Document 1). This index image can be stored as an image file as well as displayed. Also, a technique for generating a mosaic image that generates a single image by combining a plurality of images has been disclosed (for example, see Patent Document 2).

  On the other hand, some digital cameras can take a photograph that gives a unique impression that has not existed before by performing special effect image processing different from the conventional one on the photographed image. This special effect image processing is called random tiles, where the captured images are divided and pasted to convert the scene into an image that looks like a combination of multiple images obtained by multiple shootings. There is processing.

  Further, when recording a still image or a moving image with a digital camera, compression is generally performed in order to reduce the size of the image data. At this time, a method of making the color difference component more highly compressed than the luminance component by using human visual characteristics is generally used. For example, in a general JPEG method as a still image compression method, data of all pixels is used for the luminance component, and data sampled in a so-called YC422 format using data for each pixel in the horizontal direction for the color difference component. Compressed. H. In the case of a moving image compressed in the H.264 format or the like, compression is performed after sampling in the YC420 format in which the color difference component is further reduced by half in the vertical direction. This achieves high compression while maintaining natural image quality.

JP-A-5-2612 JP 2000-298722 A

  A common point between index images, mosaic images, and random tile images generated with random tiles is that they are tile-shaped images that are made by pasting a small image onto a background image (for example, a black surface). .

  At this time, when the data obtained by down-sampling the tiled image in the YC422 format or the YC420 format is upsampled to the YC444 format in which each pixel has a luminance component and a color difference component for reproduction, the pixel having no color difference component is reproduced. Data is interpolated using the color difference components of adjacent pixels. In this process, colors that did not exist in the original image may occur. Since a natural image has a high correlation between adjacent pixels, a result very close to the original image can be obtained even by upsampling. However, in the tiled image, a case where the correlation between the pasted image and the background image is very low is assumed. In that case, a color that does not exist in the original image can be seen, and the image becomes uncomfortable especially when enlarged display or printing is performed.

  The present invention has been made in view of the above circumstances, and in particular, includes an image processing apparatus capable of generating an image that does not cause a sense of incongruity even when played back when generating a tiled image. An object is to provide an imaging apparatus, an image processing method, and an image processing program.

In order to achieve the above object, the image processing apparatus according to the first aspect of the present invention pastes partial image data having a linear boundary in one horizontal or vertical direction to background image data. An image composition unit that generates composite image data, a boundary position determination unit that determines a position of the boundary of the partial image data in the background image data when the composite image data is generated, and the composite image data An image sampling unit that performs a sampling process for reducing the number of data of the color difference component of the composite image data than the number of luminance component data of the composite image data, and the boundary position determining unit includes the image sampling unit The boundary position is corrected by enlarging or reducing the partial image data according to the sampling processing method according to the above .

  In order to achieve the above object, an imaging apparatus according to a second aspect of the present invention includes the image processing apparatus according to the first aspect.

In order to achieve the above object, in the image processing method according to the third aspect of the present invention, the image composition unit sets a linear boundary in one horizontal or vertical direction with respect to the background image data. The partial image data is pasted to generate composite image data, and the boundary position determination unit determines the position of the boundary of the partial image data in the background image data when generating the composite image data, and the boundary The position determination unit corrects the position of the boundary by enlarging or reducing the partial image data according to the sampling processing method by the image sampling unit, and the image sampling unit A sampling process is performed to reduce the number of color difference components of the composite image data than the number of luminance components of the composite image data.

In order to achieve the above object, the image processing program according to the fourth aspect of the present invention provides the image composition unit with a linear boundary in one horizontal or vertical direction with respect to the background image data. A procedure of pasting the partial image data to generate composite image data, and a step of causing the boundary position determination unit to determine the position of the boundary of the partial image data in the background image data when generating the composite image data And a step of causing the boundary position determination unit to correct the position of the boundary by enlarging or reducing the partial image data according to a sampling processing method by the image sampling unit, and A sample that reduces the number of color difference components of the composite image data relative to the number of luminance components of the composite image data. To perform the procedure for subjected to processing, to the computer.

  According to the present invention, in particular, an image processing device capable of generating an image that is unlikely to cause discomfort even when played back when generating a tiled image, an imaging device including the same, an image processing method, and image processing A program can be provided.

1 is a block diagram illustrating a configuration of a digital camera as an example of an imaging apparatus including an image processing apparatus according to an embodiment of the present invention. It is a flowchart which shows the main operation | movement of a digital camera. It is the flowchart which showed the detail of the index image generation process. It is a figure which shows the example of the arrangement | positioning pattern in an index image generation process. 5 is a flowchart illustrating details of image processing. It is a flowchart which shows the detail of a mosaic process. It is a figure which shows the example of a division | segmentation in a mosaic process. It is a flowchart which shows the detail of a random tile process. It is a figure which shows the example of the division | segmentation of the flame | frame in a random tile process. It is a figure which shows the example of a production | generation of a random tile image. It is a flowchart shown about an image paste process. It is a flowchart shown about the specific example of correction | amendment of the coordinate of the paste position in the case of an image paste process. It is a figure for demonstrating the downsampling of YC data. It is a figure shown about image reproduction | regeneration when not correcting the coordinate of the paste position of partial image data. It is a figure shown about image reproduction in the case of amending the coordinates of the pasting position of partial image data.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram illustrating a configuration of a digital camera as an example of an imaging apparatus including an image processing apparatus according to an embodiment of the present invention. A digital camera 1 shown in FIG. 1 is an interchangeable lens digital camera. However, it is not always necessary to be an interchangeable lens digital camera, and a lens-integrated digital camera may be used.

  A digital camera 1 shown in FIG. 1 has an interchangeable lens 100 and a camera body 200. The interchangeable lens 100 is configured to be detachable from the camera body 200. When the interchangeable lens 100 is attached to the camera body 200, the interchangeable lens 100 is connected to the camera body 200 so as to be able to communicate. As a result, the interchangeable lens 100 becomes operable according to the control of the camera body 200.

The interchangeable lens 100 includes a lens 102, a diaphragm 104, a driver 106, a microcomputer 108, and a flash memory 110.
The lens 102 is an optical system for condensing a light beam from a subject (not shown) on the image sensor 204 in the camera body 200. The lens 102 may include a plurality of lenses such as a focus lens and a zoom lens.

  The diaphragm 104 is configured to be openable and closable, and adjusts the amount of light incident through the lens 102. The driver 106 has a motor and the like. The driver 106 drives the focus lens and zoom lens in the lens 102 in the optical axis direction and drives the aperture 104 to open and close according to the control of the microcomputer 108.

The microcomputer 108 is communicably connected to the microcomputer 230 in the camera body 200 via the interface (I / F) 112 when the interchangeable lens 100 is attached to the camera body 200. The microcomputer 108 drives the driver 106 in accordance with control from the microcomputer 230. Further, the microcomputer 108 communicates lens information of the interchangeable lens 100 stored in the flash memory 110 to the microcomputer 230 via the I / F 112.
The flash memory 110 stores lens information such as aberration information of the lens 102, a program necessary for executing the operation of the interchangeable lens 100, and the like.

  The camera body 200 includes a mechanical shutter 202, an image sensor 204, an analog processing unit 206, an analog / digital (A / D) conversion unit 208, a bus 210, an SDRAM 212, an AE processing unit 214, and an AF processing unit 216. An image processing unit 218, a display driver 220, a display unit 222, an image sampling unit 223, an image compression / decompression unit 224, a memory interface (I / F) 226, a recording medium 228, a microcomputer 230, , An operation unit 232 and a flash memory 234.

  The mechanical shutter 202 is configured to be movable so that the photoelectric conversion surface of the image sensor 204 is in a light shielding state or an exposed state. The exposure time of the image sensor 204 is adjusted by moving the mechanical shutter 202.

  The image sensor 204 has a photoelectric conversion surface on which a light beam from a subject condensed through the lens 102 is imaged. The photoelectric conversion surface is configured with a plurality of pixels arranged two-dimensionally. A color filter is provided on the light incident side of the photoelectric conversion surface. Such an image sensor 204 converts an image (subject image) corresponding to the light beam formed on the photoelectric conversion surface into an electrical signal (hereinafter referred to as an image signal) corresponding to the light amount and outputs the electrical signal.

  Here, as the imaging device 204, imaging devices having various configurations such as a CCD method and a CMOS method are known. Various color filters such as a Bayer array are also known. In the present embodiment, the configuration of the image sensor 204 is not limited to a specific configuration, and image sensors with various configurations can be used. The image sensor 204 may have an electronic shutter function that electronically controls the exposure time. In the following description, it is assumed that the image sensor 204 has an electronic shutter function.

  The analog processing unit 206 performs analog processing such as CDS (correlated double sampling) processing and AGC (automatic gain control) processing on the image signal obtained by the image sensor 204. The A / D converter 208 converts the image signal analog-processed by the analog processor 206 into a digital signal (hereinafter referred to as image data).

  The bus 210 is a transfer path for transferring various data generated in the camera body 200. The SDRAM 212 is a storage unit for temporarily storing various data generated inside the camera body 200. The SDRAM 212 is also used as a buffer memory at the time of image processing in the image processing unit 218.

  The AE processing unit 214 calculates subject brightness using the image data. Note that the subject luminance is not only calculated from the image data, but may be measured by a dedicated photometric sensor, for example. The AF processing unit 216 extracts high-frequency component signals from the image data, integrates the extracted high-frequency component signals, and acquires a focus evaluation value for AF.

The image processing unit 218 performs various image processing on the image data. The image processing unit 218 in this embodiment includes a basic image processing unit 2181 and a special effect image processing unit 2182.
A basic image processing unit 2181 performs basic image processing necessary for image display and recording on image data. The basic image processing includes, for example, optical black subtraction processing, white balance correction processing, color matrix calculation processing, gamma conversion processing, color correction processing, edge enhancement processing, and noise reduction processing.
The optical black subtraction process is a process of subtracting and removing the dark current component (optical black) of the image data. The white balance correction process is a process of multiplying a gain for correcting the color balance of the image data. The color matrix calculation process is a process of mapping image data onto a specific color space by performing color matrix calculation corresponding to the result of white balance correction on the image data. The gamma conversion process is a process for converting the gradation characteristics of the image data into predetermined characteristics. The color correction process is a process of multiplying C data of the image data converted into the YC format by a saturation correction coefficient and a hue correction coefficient in order to make color reproduction appropriate. In the edge enhancement processing, an edge (contour) component in image data is enhanced by multiplying an edge signal extracted from the image data by using a bandpass filter or the like and multiplying the edge signal by Y with respect to the image data. It is processing. The noise reduction process is a process for removing noise components in the image data.
Further, depending on the color arrangement of the color filters, further synchronization processing may be required as basic image processing. The synchronization processing converts image data in which one pixel corresponds to one color component, such as image data corresponding to a Bayer array, into image data in which one pixel corresponds to a plurality of color components. It is processing.

The special effect image processing unit 2182 applies special effect image processing for giving a special visual effect to the image to the image data. This special effect image processing includes image composition processing for generating a tiled image. Here, the tiled image refers to a composite image configured by pasting a plurality of partial images. This tiled image includes, for example, an index image, a mosaic image, and a random tile image.
The index image generation process is a process of pasting partial images (index images) indicating images recorded on the recording medium 228. In the mosaic image generation process, the mosaic image generation target image is divided into partial image data of a plurality of frames, and the images are pasted so that the partial image data of a certain frame is replaced with the partial image data of another frame. It is processing. The random tile generation process is a process of randomly pasting a plurality of partial image data. Details of these processes will be described later.

  The special effect image processing unit 2182 also has a function as a boundary position determination unit. This function is a function for determining and correcting the pasting position of the partial image data, that is, the boundary position of the partial image data in the background image data.

  The display driver 220 converts the image data obtained by the image processing unit 218 or the image data expanded by the image compression / decompression unit 224 into a video signal and outputs the video signal to the display unit 222. The display unit 222 is, for example, a liquid crystal display (LCD). The display unit 222 displays an image based on the video signal input from the display driver 220.

  In order to reduce the data amount of the image data, the image sampling unit 223 prior to the compression processing in the image compression / decompression unit 224, the color difference component (C data) rather than the number of luminance components (Y data) of the image data. Perform sampling processing (downsampling) to reduce the number of data. In addition, the image sampling unit 223 performs a restoration process (the restoration process for making the number of luminance components (Y data) and the number of color difference components (C data) of the image data decompressed by the decompression process in the image compression / decompression unit 224 equal ( Upsampling). Details of the image sampling unit 223 will be described later.

  At the time of image recording, the image compression / decompression unit 224 performs still image compression processing such as JPEG format or TIFF format, MPEG format, H.264, or the like on image data obtained by image processing in the image processing unit 218. H.264 format moving image compression processing is performed. Further, the image compression / decompression unit 224 performs decompression (decoding) processing on the YC data subjected to the compression processing at the time of image reproduction.

  The memory I / F 226 is an interface for the microcomputer 230 or the like to access the recording medium 228. The recording medium 228 is a memory card that is detachably attached to the camera body 200, for example. The recording medium 228 records image files and the like. The image file is a file in which header information is added to the image data compressed by the image compression / decompression unit 224.

  The microcomputer 230 comprehensively controls the operation of each part of the camera body 200 such as the mechanical shutter 202, the image sensor 204, and the display driver 220. Further, the microcomputer 230 performs AE processing using the subject brightness calculated by the AE processing unit 214 or performs AF processing using the AF evaluation value calculated by the AF processing unit 216. The microcomputer 230 also controls the operation of the interchangeable lens 100 when the interchangeable lens 100 is attached.

  The operation unit 232 is various operation members operated by the user. The operation unit 232 in the present embodiment includes a release button, a moving image button, a menu button, a playback button, an index button, and a power button. Here, some or all of these buttons may be configured as a virtual operation unit operated by a touch panel.

  The release button has a two-stage switch including a first (1st) release switch and a second (2nd) release switch. When the release button is pressed halfway and the first release switch is turned on, the microcomputer 230 executes photographing preparation processing such as AE processing and AF processing. Further, when the release button is fully pressed and the second release switch is turned on, the microcomputer 230 executes a still image recording process.

  The moving image button instructs the microcomputer 230 to execute moving image shooting. When the moving image button is pressed, the microcomputer 230 executes a moving image recording process. When the moving image button is pressed during the moving image recording process, the microcomputer 230 ends the moving image recording process.

  The menu button is an operation unit for instructing display of a menu screen. On the menu screen, the user can change various settings of the camera body 200. In the present embodiment, for example, an image processing mode is set on a menu screen. In this image processing mode, whether to generate a mosaic image or a random time image is set as a process for generating a tiled image.

  The play button is an operation unit for instructing the microcomputer 230 to play back a still image file or a moving image file. The index button is an operation unit for instructing the microcomputer 230 to display an index image. The power button is an operation unit for instructing to turn on or off the power of the camera body 200.

  The flash memory 234 is used for operations of the camera body 200 such as parameters necessary for the operation of the image processing unit 218 such as a white balance gain for white balance correction, a color matrix coefficient for color matrix calculation, and a gamma table for gamma conversion. Various necessary parameters are stored. The flash memory 234 also stores various programs executed by the microcomputer 230.

  The operation of the digital camera shown in FIG. 1 will be described below. FIG. 2 is a flowchart showing the main operation of the digital camera shown in FIG. The operation of FIG. 2 is performed when the power of the digital camera shown in FIG. 1 is turned on, for example.

  After powering on the digital camera, the microcomputer 230 performs an initialization process (step S101). In the initialization processing, the microcomputer 230 performs processing such as turning off the recording flag set in the register of the microcomputer 230. The recording flag is a flag indicating whether or not a moving image is being recorded. While the recording flag is Off, it indicates that the moving image is not being recorded. On the other hand, while the recording flag is On, it indicates that moving image recording is in progress.

Next, the microcomputer 230 determines whether or not the reproduction button of the operation unit 232 has been pressed by the user (step S102). If it is determined in step S102 that the playback button has been pressed, the microcomputer 230 executes playback processing (step S103).
In the reproduction process, the microcomputer 230 waits for the user to select an image file (still image file or moving image file). When the image file is selected, the microcomputer 230 decodes the selected image file by the image compression / decompression unit 224. The microcomputer 230 inputs the image data decoded from the selected image file to the image sampling unit 223. The image sampling unit 223 upsamples the image data and inputs the upsampled image data to the display driver 220. The display driver 220 converts the input image data into a video signal and causes the display unit 222 to display an image corresponding to the video signal. Thereafter, when the user gives an instruction to end reproduction, for example, when the reproduction button is pressed again, the microcomputer 230 returns the process to step S102.

  If it is determined in step S102 that the playback button has not been pressed, the microcomputer 230 determines whether to generate an index image (step S104). For example, when the index button of the operation unit 232 is pressed by the user, the microcomputer 230 determines to generate an index image.

  If it is determined in step S104 that an index image is to be generated, the microcomputer 230 performs an index image generation process using the image processing unit 218 (step S105). Details of the index image generation processing will be described later.

  If it is determined in step S104 that no index image is to be generated, the microcomputer 230 determines whether to perform camera settings (step S106). For example, when the menu button of the operation unit 232 is pressed by the user, the microcomputer 230 determines to set the camera.

If it is determined in step S106 that camera settings are to be made, the microcomputer 230 controls the display driver 220 to display a menu screen on the display unit 222, and then executes camera setting processing (step S107).
In the camera setting process, the microcomputer 230 waits for a camera setting change instruction from the user. When an instruction to change any camera setting is given, the microcomputer 230 changes the camera setting according to the instruction. In this camera setting process, for example, the setting of the image recording mode and the setting of the image quality at the time of still image shooting or moving image shooting are changed. In the camera setting process, whether or not to execute special effect image processing, and various modes such as a mosaic mode and a random tile mode can be set as a special effect image processing mode when executing the special effect image processing. Is possible. Here, the mosaic mode is a special effect image processing mode for generating a mosaic image. The random tile mode is a special effect image processing mode for generating a random tile image. Note that there may be modes other than the mosaic mode and the random tile mode as the special effect image processing mode.

  If it is determined in step S106 that the camera setting is not performed, the microcomputer 230 determines whether or not the moving image button of the operation unit 232 is pressed by the user (step S108). If the moving image button is pressed in the determination in step S108, the microcomputer 230 inverts the recording flag (step S109). That is, the microcomputer 230 turns On when the recording flag is Off and turns Off when it is On. Thereafter, the microcomputer 230 determines whether the moving image is currently being recorded, that is, whether the recording flag is On (step S110).

  If it is determined in step S110 that the recording flag is On, the microcomputer 230 generates a moving image file and records the generated moving image file on the recording medium 228 (step S111). The moving image file has a header information recording unit and a moving image recording unit. In the header information section, header information for a moving image file is recorded. The header information includes a file name, shooting information at the time of moving image shooting (ISO sensitivity, aperture value, shooting date, etc.), pointer information for specifying the position of each of a plurality of image data constituting moving image data, and the like. Record. Furthermore, partial image data (thumbnail image data of the first frame) indicating moving image data obtained by moving image shooting is also recorded. The moving image recording unit records moving image data (YC data for moving image recording) in a compressed state.

If the moving image button is not pressed in the determination in step S108, or if the recording flag is not On in the determination in step S110, after the moving image file is generated in step S111, the microcomputer 230 is currently recording the moving image. It is determined whether or not the recording flag is On (step S112). If it is determined in step S112 that the recording flag is OFF, the microcomputer 230 causes the release button of the operation unit 232 to be half-pressed by the user, and the state of the release button changes from the OFF state to the ON state of the 1st release switch. It is determined whether or not (step S113). If it is determined in step S113 that the state of the release button has transitioned to the ON state of the first release switch, the microcomputer 230 performs AE processing and AF processing (step S114).
In the AE process, the microcomputer 230 causes the AE processing unit 214 to calculate the subject brightness. Thereafter, the microcomputer 230 determines the ISO sensitivity, the aperture value, and the shutter speed at the time of executing the still image shooting according to the subject brightness calculated by the AE processing unit 214. In the AE process, the ISO sensitivity, the aperture value, and the shutter speed may be determined so that the luminance of a specific part in the image data is appropriate.
In the AF process, the microcomputer 230 causes the AF processing unit 216 to acquire a focus evaluation value. Then, the microcomputer 230 instructs the microcomputer 108 to drive the focus lens of the lens 102 little by little while evaluating the contrast in the image data based on the focus evaluation value acquired by the AF processing unit 216. Thereafter, the microcomputer 230 instructs the microcomputer 108 to stop driving the focus lens when the contrast reaches the maximum. Such AF processing is so-called contrast AF processing. A phase difference AF process may be used as the AF process. In the AF process, the focus lens may be driven so as to focus on a specific part in the image data.

  Here, the AE process and the AF process in the example of FIG. 2 are executed at the timing when the state of the release button transitions to the On state of the 1st release switch. That is, when the state of the release button does not transition to the On state of the 1st release switch, for example, when the state of the release button remains in the Off state, when the 1st release switch remains in the On state, For example, when the On state is maintained, the AE process and the AF process are not executed. Of course, the AE process and the AF process may be executed.

  After step S114, the microcomputer 230 determines whether the power of the digital camera is turned off (step S115). If it is determined in step S115 that the power of the digital camera is not turned off, the microcomputer 230 executes the processing after step S102. On the other hand, if it is determined in step S115 that the power of the digital camera has been turned off, the microcomputer 230 ends the processing in FIG.

  If it is determined in step S113 that the state of the release button has not changed to the ON state of the first release switch, the microcomputer 230 causes the release button of the operation unit 232 to be fully depressed by the user. Is determined whether or not the 2nd release switch is in the ON state (step S116).

  In step S116, when the state of the release button is the ON state of the 2nd release switch, the microcomputer 230 executes a still image recording process using the mechanical shutter 202 (step S117). For this purpose, the microcomputer 230 sets the gain control amount (amplification factor) in the analog processing unit 206 according to the ISO sensitivity determined in step S114, and transmits the aperture value determined in step S114 to the microcomputer 108. . Thereafter, the microcomputer 230 controls the exposure amount of the image sensor 204 by operating the mechanical shutter 202 according to the shutter speed determined in step S114 in synchronization with the driving of the diaphragm 104 under the control of the microcomputer 108. Image data is stored in the SDRAM 212 by such still image recording processing.

  After executing the still image recording process, the microcomputer 230 performs image processing on the image data stored in the SDRAM 212 by the still image recording process by using the image processing unit 218, thereby still image data (for still image recording). YC data) is generated (step S118). At this time, if it is set in advance to perform special effect image processing, both basic image processing by the basic image processing unit 2181 and special effect image processing by the special effect image processing unit 2182 are performed on the image data. . On the other hand, when it is not preset to perform special effect image processing, only basic image processing by the basic image processing unit 2181 is performed on the image data.

  After the image processing, the microcomputer 230 performs processing for recording the still image data stored in the SDRAM 212 as a result of the image processing as a still image file in the set still image recording format (step S119). At this time, for example, when the set still image recording format is JPEG format, the microcomputer 230 downsamples still image data stored in the SDRAM 212 as a result of the still image recording processing in the image sampling unit 223. The downsampled image data (YC data) is input to the image compression / decompression unit 224 and instructs the image compression / decompression unit 224 to execute still image compression processing. In response to this instruction, the image compression / decompression unit 224 performs still image compression processing corresponding to a preset recording mode, and stores the compressed still image data in the SDRAM 212. Thereafter, the microcomputer 230 reads the still image data compressed by the image compression / decompression unit 224 from the SDRAM 212, adds predetermined header information to the read still image data, creates a still image file, and creates the created still image file. Is recorded on the recording medium 228. The still image file has a header information recording unit and a still image recording unit. In the header information section, header information for a still image file is recorded. As the header information, a file name, shooting information at the time of shooting a still image (ISO sensitivity, aperture value, shooting date, etc.) and the like are recorded. Further, partial image data (thumbnail image data) indicating still image data obtained by still image shooting is also recorded. The still image recording unit records still image data in a compressed state. In the above-described example, the image processing unit 218 converts the image data into YC format image data before recording. On the other hand, for example, when the set still image recording format is the TIFF format, the image data is converted again into the RGB format without down-sampling, and the image data is recorded.

  If it is determined in step S116 that the state of the release button is not the ON state of the 2nd release switch, the microcomputer 230 executes AE processing (step S120). This AE process is a process for moving image shooting or live view display. After the AE process, the microcomputer 230 executes a shooting process for moving image shooting or live view display using an electronic shutter (step S121). In this photographing process, the microcomputer 230 controls the exposure amount of the image sensor 204 by operating the electronic shutter function of the image sensor 204 in accordance with the shutter speed determined by the AE process. After the photographing process, the microcomputer 230 performs image processing on the image data stored in the SDRAM 212 as a result of the photographing process using the image processing unit 218 (step S122). After the image processing, the microcomputer 230 instructs the display driver 220 to reproduce the image data stored in the SDRAM 212 as a result of the image processing. Upon receiving this instruction, the display driver 220 reads the image data from the SDRAM 212, converts the read image data into a video signal, and outputs it to the display unit 222. The display unit 222 reproduces an image based on this video signal (step S123). With such live view display, the user can check the composition using the display unit 222.

  Further, after the live view display, the microcomputer 230 determines whether or not the moving image is currently being recorded, that is, whether or not the recording flag is On (step S124). If it is determined in step S124 that the recording flag is Off, the microcomputer 230 performs the determination in step S115. If it is determined in step S124 that the recording flag is On, the microcomputer 230 executes a moving image recording process (step S125). Thereafter, the microcomputer 230 performs the determination in step S115. In the moving image recording process, after the image data generated in step S122 is down-sampled by the image sampling unit 223, the image data (YC data) down-sampled according to the set moving image recording format is subjected to image compression / decompression. The image compression / decompression unit 224 is instructed to input to the unit 224 and execute the moving image compression process. Then, the moving image data obtained by the moving image compression process is added to the moving image recording unit of the moving image file.

  FIG. 3 is a flowchart showing details of the index image generation process. In the index image generation process, the image processing unit 218 counts the number of image files (still image files and moving image files) currently recorded on the recording medium 228, and acquires the number of recordings (step S201).

Subsequently, the image processing unit 218 determines an arrangement pattern of partial image data (step S202).
FIG. 4 shows an example of an arrangement pattern. FIG. 4A shows an example in which 16-frame partial images are simultaneously arranged. FIG. 4B shows an example in which partial images of 64 frames are arranged at the same time. The numbers given in FIGS. 4A and 4B are the pasting positions of partial images created by resizing or the like using the image data recorded on the recording medium 228 (the value of p described later). Corresponding). Here, in the example of FIGS. 4A and 4B, an interval is provided between the partial images, but this interval is not necessarily provided. The arrangement pattern shown in FIG. 4 is an example, and the arrangement pattern shown in FIG. 4 is not necessarily required. For example, the number of frames of partial images arranged simultaneously as an index image does not necessarily need to be 16 frames or 64 frames.

  The microcomputer 230 determines an arrangement pattern according to the number of recordings, for example. For example, the microcomputer 230 selects the arrangement pattern shown in FIG. 4A when the number of recorded sheets is 16 or less, and selects the arrangement pattern shown in FIG. 4B when the number of recorded sheets is 17 or more. In addition, the user may be able to set the arrangement pattern on the menu screen.

  After determining the arrangement pattern, the image processing unit 218 requests the microcomputer 230 to secure a buffer area (step S203). In response to this, the microcomputer 230 secures a buffer area for storing index image data in the SDRAM 212. The buffer is secured by storing, for example, background image data used for pasting an image in the buffer area. As the background image data, it is possible to use arbitrary image data such as image data of a single color (for example, black one color) or image data of a predetermined pattern.

  Subsequently, the image processing unit 218 sets the parameter i to the initial value 1 (step S204). Further, the image processing unit 218 sets the parameter p to the initial value 1 (step S205). i is a parameter for specifying the order of the partial image data. On the other hand, p is a parameter for specifying the pasting position of the partial image.

  Subsequently, the image processing unit 218 determines a partial image pasting position in the index image data (step S206). The pasting position is a position on the background image data at the boundary of each partial image data constituting the index image data. In step S206, the image processing unit 218 calculates upper left coordinates and lower right coordinates (may be lower left coordinates and upper right coordinates) of the p-th position in the index image data. For example, since p = 1 at the first time, the upper left coordinates and upper right coordinates of the first position shown in FIG. 4A or 4B are calculated. When the arrangement position is determined in advance, the upper left coordinates and upper right coordinates of the determined p-th position may be acquired. In addition, the upper left and upper right coordinates of the p-th position are calculated from the size of the index image, the number of partial image data to be pasted along the horizontal and vertical directions of the index image, and the interval between the partial images. You may do it.

  After determining the p-th pasting position, the image processing unit 218 reads image data for an index image from the i-th image file recorded on the recording medium 228 (step S207). The order of image files is, for example, the order of recording time.

  After reading the image data for the index image from the i-th image file, the image processing unit 218 performs an image pasting process using the special effect image processing unit 2182 (step S208). Details of the image pasting process will be described later.

  After the image pasting process, the image processing unit 218 adds 1 to p (step S209). Thereafter, the image processing unit 218 determines whether p is equal to or greater than the number of frames (step S210). When the arrangement pattern is the pattern of FIG. 4A, it is determined whether p is 16 or more. When the arrangement pattern is the pattern of FIG. 4B, it is determined whether p is 64 or more. If it is determined in step S210 that p is less than the number of frames, it means that the pasting position remains. In this case, the image processing unit 218 adds 1 to i (step S214). Thereafter, the image processing unit 218 determines whether i is equal to or smaller than the number of recordings acquired in step S201 (step S215). In step S215, if i is not equal to or smaller than the number of recording sheets, it means that partial image data to be pasted still remains. In this case, the image processing unit 218 returns the process to step S206 to determine the pasting position of the next partial image data.

  If it is determined in step S210 that p is equal to or greater than the number of frames, the image processing unit 218 outputs the index image data generated through the image pasting process. In response to this, the microcomputer 230 down-samples the index image data generated by the image processing unit 218 by the image sampling unit 223, further compresses the index image data by the image compression / decompression unit 224, and then stores it on the recording medium 228 as a still image file. Recording is performed (step S211). Subsequently, the image processing unit 218 requests the microcomputer 230 to clear the buffer area (step S212). In response to this, the microcomputer 230 clears the data stored in the buffer area secured in the SDRAM 212. The buffer is cleared, for example, by replacing the data in the buffer area with background image data used for image pasting described later.

  Thereafter, the image processing unit 218 returns p to 1 (step S213). Then, the image processing unit 218 performs processing subsequent to step S214. When image files of the number of frames or more are recorded on the recording medium 228, the second or more index image data is generated by the processing after step S214.

  If it is determined in step S215 that i exceeds the number of recording sheets, the image processing unit 218 determines whether p exceeds 1 (step S216). If it is determined in step S216 that p is greater than 1, the image processing unit 218 outputs the index image data generated through the image pasting process so far. In response to this, the microcomputer 230 down-samples the index image data generated by the image processing unit 218 by the image sampling unit 223, further compresses the index image data by the image compression / decompression unit 224, and then stores it on the recording medium 228 as a still image file. Recording is performed (step S217). The case where p exceeds 1 in step S216 indicates that p has not been returned to 1 in step S213. In this case, pasting of the partial images for the number of frames has not been completed, and no index image data has been recorded in step S211. Accordingly, in step S217, the index image data in a state where a partial image smaller than the number of frames is pasted is recorded.

  When it is determined in step S216 that p is 1 or less, or after step S217, the image processing unit 218 requests the microcomputer 230 to release the buffer area (step S218). In response to this, the microcomputer 230 releases the buffer area reserved in the SDRAM 212, and ends the processing of FIG.

  FIG. 5 is a flowchart showing details of the image processing. In FIG. 5, the image processing unit 218 performs OB subtraction processing by the basic image processing unit 2181 (step S <b> 301). In the OB subtraction process, the basic image processing unit 2181 removes a dark current component in the image data by subtracting an optical black (OB) value from the input image data. After the OB subtraction, the image processing unit 218 performs WB correction processing by the basic image processing unit 2181 (step S302). In the WB correction process, the basic image processing unit 2181 corrects the color balance of the image data by multiplying each color component of the input image data by a WB (white balance) gain. After the WB correction process, the image processing unit 218 performs a synchronization process using the basic image processing unit 2181 (step S303). In the synchronization processing, the basic image processing unit 2181 synchronizes the input image data using interpolation processing. Thereby, image data in which one pixel has one color component of RGB is converted into image data in which one pixel has three color components of RGB.

  After the synchronization processing, the image processing unit 218 performs color matrix calculation by the basic image processing unit 2181 (step S304). In the color matrix calculation, the basic image processing unit 2181 converts the color of the image data by multiplying each pixel of the input image data by a color matrix coefficient. After the color matrix calculation, the image processing unit 218 performs gamma conversion processing and color correction processing using the basic image processing unit 2181 (step S305). At this time, the basic image processing unit 2181 performs gamma conversion processing on the image data using a gamma table set by the microcomputer 230. Further, the basic image processing unit 2181 performs a predetermined matrix operation to convert the image data into the YC format. Although details will be described later, in the image processing at the time of recording a still image, for example, the data is converted into YC422 format data. The YC422 format data is data in which Y: Cb: Cr is 4: 2: 2. In the image processing at the time of moving image recording, for example, the data is converted into YC420 format data. The YC420 format data is data in which Y: Cb: Cr is 4: 1: 1. In the color correction process, the basic image processing unit 2181 multiplies C (Cb, Cr) data in the image data converted into the YC format by the saturation correction coefficient and the hue correction coefficient, thereby correcting the color of the image data. I do.

  After the color correction processing, the image processing unit 218 performs edge enhancement processing using the basic image processing unit 2181 (step S306). In the edge enhancement processing, the basic image processing unit 2181 performs bandpass filter processing on the Y data in the color-corrected YC data to extract edge components in the image data, and the extracted edge components according to the edge enhancement amount. Multiply the coefficient. Then, the basic image processing unit 2181 emphasizes the edge component of the image data by adding the edge component multiplied by the coefficient to the original Y data. After the edge enhancement processing, the image processing unit 218 performs noise reduction (NR) processing using the basic image processing unit 2181 (step S307). In the noise reduction process, the basic image processing unit 2181 performs a coring process or the like on the Y data subjected to the edge enhancement process to reduce noise components in the image data. You may reduce a noise component with respect to C data. When the recording format is TIFF format, the data after the noise reduction processing is converted into RGB format again by performing a predetermined matrix calculation.

  After the noise reduction process, the image processing unit 218 determines whether the mosaic mode is set as the special effect image processing mode by the microcomputer 230 (step S308). As described above, the setting for setting the special effect image processing mode to the mosaic mode is performed in the camera setting.

  When it is determined in step S308 that the mosaic mode is set as the special effect image processing mode, the image processing unit 218 performs mosaic processing by using the special effect image processing unit 2182 (step S309). Thereafter, the image processing unit 218 ends the process of FIG. Here, details of the mosaic processing will be described later.

  When it is determined in step S308 that the mosaic mode is not set as the special effect image processing mode, the image processing unit 218 determines whether or not the random tile mode is set as the special effect image processing mode (step S308). S310). As described above, the setting for setting the special effect image processing mode to the Moranda tile mode is performed in the camera setting.

  If it is determined in step S310 that the random tile mode is set as the special effect image processing mode, the image processing unit 218 performs random tile processing by the special effect image processing unit 2182 (step S311). Thereafter, the image processing unit 218 ends the process of FIG. Here, details of the random tile processing will be described later.

  Further, when it is determined in step S310 that the random tile mode is not set as the special effect image processing mode, the image processing unit 218 also ends the process of FIG. Note that, when there are special effect image processing modes other than the mosaic mode and the random tile mode, the same determination as the determinations shown in steps S308 and S310 is performed for each special effect image processing mode.

  Here, the order of the processing in FIG. 5 is an example and can be changed as appropriate. For example, the synchronization process may be performed before the WB correction process.

  FIG. 6 is a flowchart showing details of the mosaic process. In the mosaic processing, the image processing unit 218 divides the image data after the noise reduction processing into partial image data of a plurality of frames each having one or more pixels (step S401). FIG. 7 shows an example of division. FIG. 7 shows an example in which image data is divided into vertical 300 frames and horizontal 400 frames. Here, the number of divisions is not limited to the example shown in FIG. In the example of FIG. 7, an interval is provided between the frames. This interval may not be provided.

  Subsequently, the image processing unit 218 calculates the average color of each frame (step S402). In the case of the YC format, the average color is a value obtained by dividing the result of adding the color difference data constituting each frame for each component (Cb, Cr) by the number of pixels constituting each frame. In the case of the RGB format, the average color is a value obtained by dividing the result of adding the pixel data constituting each frame for each component (R, G, B) by the number of pixels constituting each frame.

  After calculating the average color, the image processing unit 218 requests the microcomputer 230 to secure a buffer area (step S403). In response to this, the microcomputer 230 secures a buffer area in the SDRAM 212. As a method for securing the buffer area, the method described in the description of the index image generation process can be applied.

  Subsequently, the image processing unit 218 sets the parameter i to the initial value 1 (step S404). i is a parameter for specifying the order of frames. This i identifies the pasting position. The pasting position is a position on the background image data at the boundary of each partial image data constituting the mosaic image data.

  After setting i, the image processing unit 218 selects partial image data of a frame to be replaced with the i-th frame (step S405). In the mosaic process, as shown in FIG. 7, mosaic image data is generated by replacing the partial image data of the i-th frame with the partial image data of another frame. The frame to be replaced is, for example, a frame having an average color closest to the average color of the i-th frame. By performing such replacement, the frame before replacement is replaced with another frame having a similar color on average. Thereby, a mosaic image is generated.

  Further, instead of replacing a frame in the same image, a frame having a similar average color may be searched for and replaced from another image data recorded on the recording medium 228.

  After selecting the partial image data of the frame to be replaced, the image processing unit 218 performs image pasting processing using the special effect image processing unit 2182 (step S406). Details of the image pasting process will be described later.

  After the image pasting process, the image processing unit 218 adds 1 to i (step S407). Thereafter, the image processing unit 218 determines whether i is equal to or less than the number of frames (vertical frame number × horizontal frame number, 120,000 frames in FIG. 7) (step S408). If i is equal to or less than the number of frames in step S408, it means that partial image data of the replaced frame remains. In this case, the image processing unit 218 returns the process to step S405 and selects the partial image data of the frame to be replaced with the next partial image data.

  If it is determined in step S408 that i exceeds the number of frames, the image processing unit 218 stores the mosaic image data generated through the image pasting process and the image data after the noise reduction process. Copying is performed (overwriting the image data in the buffer area) in the buffer area (step S409). Subsequently, the image processing unit 218 requests the microcomputer 230 to release the buffer area (step S410). In response to this, the microcomputer 230 releases the buffer area reserved in the SDRAM 212 and ends the processing of FIG.

  FIG. 8 is a flowchart showing details of the random tile process. In the random tile process, the image processing unit 218 divides the image data after the noise reduction process into partial image data of a plurality of frames each having one or more pixels (step S501). FIG. 9 shows an example of division. FIG. 9 shows an example in which image data is divided into 6 vertical frames and 8 horizontal frames. Here, the number of divisions is not limited to the example shown in FIG. In the example of FIG. 9, no interval is provided between the frames. An interval may be provided.

  Subsequently, the image processing unit 218 determines the copy order of the partial image data (step S502). The copy order is generated by, for example, randomly replacing each element in an array in which numbers from 1 to the number of frames are stored. When the image data is moving image data, the arrangement order may be determined so that the difference in data of each pixel between adjacent frames is less than or equal to a predetermined amount.

  After determining the copy order, the image processing unit 218 requests the microcomputer 230 to secure a buffer area (step S503). In response to this, the microcomputer 230 secures a buffer area in the SDRAM 212. As a method for securing the buffer area, the method described in the description of the index image generation process can be applied.

  Subsequently, the image processing unit 218 sets the parameter i to the initial value 1 (step S504). i is a parameter indicating the index of the array.

  After setting i, the image processing unit 218 calculates the pasting position of the partial image data of the frame having the same number as the i-th element in the array (step S505). The pasting position is a position on the background image data at the boundary between the partial image data constituting the random tile image data. In calculating the pasting position, the upper left coordinate and lower right coordinate of the frame having the same number as the i-th element in the array are detected, and each coordinate is randomly random by a certain amount so that the aspect ratio of the partial image data does not change. Move with. For example, as shown in FIG. 9, each coordinate is shifted by a size within 10% of the original size of the partial image data. By setting the pasting position in this way, a random tile image as shown in FIG. 10A is generated.

  After calculating the pasting position, the image processing unit 218 performs an image pasting process using the special effect image processing unit 2182 (step S506). Details of the image pasting process will be described later.

  After the image pasting process, the image processing unit 218 adds 1 to i (step S507). Thereafter, the image processing unit 218 determines whether i is equal to or less than the number of frames (vertical frame number × horizontal frame number, 48 frames in the case of FIG. 9) (step S508). If i is equal to or less than the number of frames in step S508, it means that partial image data of the frame to be pasted remains. In this case, the image processing unit 218 returns the process to step S505 to calculate the pasting position of the next partial image data.

  If it is determined in step S508 that i exceeds the number of frames, the image processing unit 218 stores the random tile image data generated through the image pasting process in the image data after the noise reduction process. Copying is performed (overwriting the image data in the buffer area) in the buffer area that has been made (step S509). Subsequently, the image processing unit 218 requests the microcomputer 230 to release the buffer area (step S510). In response to this, the microcomputer 230 releases the buffer area secured in the SDRAM 212, and ends the processing of FIG.

  Here, in the example of FIG. 10A, the shape of the partial image data to be pasted is a rectangle, but the shape of the partial image data to be pasted is not necessarily a rectangle. For example, partial image data having a shape in which four corners are chamfered as shown in FIG. 10B may be pasted, or partial image data having a substantially semicircular shape as shown in FIG. 10C. You may make it stick. As described above, the partial image data only needs to have a linear boundary in the horizontal direction or the vertical direction. The partial image data shown in FIG. 10B and the partial image data shown in FIG. 10C are generated by masking the data to be chamfered in the divided partial image data with the background image data. it can.

  Next, an image pasting process as an image processing method of the present embodiment will be described with reference to FIG. The image pasting process is common to the index image generation process, the mosaic process, and the random tile process. In the image pasting process, composite image data (index image data, mosaic image data, or random tile image data) generated from image data down-sampled in the YC format is appropriately up-sampled during playback. Then, the pasting is performed after correcting the pasting position or size of the partial image data.

  In the image pasting process, the special effect image processing unit 2182 reads partial image data to be copied to the background image data as copy source image data (step S601). In the case of index image generation processing, this partial image data is partial image data generated by resizing processing or the like from image data read from the i-th image file. In the case of mosaic processing, it is partial image data of a frame that replaces the i-th frame. Further, in the case of random tile processing, the partial image data of the frame having the same number as the i-th element in the array in which the numbers from 1 to the number of frames are randomly stored.

  After reading the copy source image data, the special effect image processing unit 2182 determines whether the format for down-sampling the copy destination image data is the YC422 format or the YC420 format (step S602). The downsampling format of the copy destination image data is notified by the microcomputer 230, for example.

  If it is determined in step S602 that the downsampling format is the YC422 format or the YC420 format, the special effect image processing unit 2182 corrects the pasting position or size of the partial image data to correct the pasting position X coordinate (horizontal The direction coordinates are corrected (step S603). This correction will be described later.

  When it is determined in step S602 that the downsampling format is not the YC422 format or the YC420 format, or after step S603, the special effect image processing unit 2182 determines whether the format at the time of downsampling the copy destination image data is the YC420 format. Is determined (step S604).

  When it is determined in step S604 that the downsampling format is the YC420 format, the special effect image processing unit 2182 corrects the pasting position or size of the partial image data to correct the pasting position Y coordinate (vertical coordinate). Is corrected (step S605). This correction will be described later.

  When it is determined in step S604 that the downsampling format is not the YC420 format or after step S605, the special effect image processing unit 2182 sets the partial image data to the coordinates of the pasting position in the background image data copied to the buffer area. Copy (synthesize) (step S606). Thereafter, the special effect image processing unit 2182 ends the process of FIG. As described in the random tile generation process, when it is necessary to enlarge or reduce the partial image data, the partial image data is resized and then copied. When the partial image data has a shape other than rectangular, unnecessary pixel data is masked before copying.

The correction of the coordinates of the pasting position will be described with reference to FIG. The numbers shown in FIG. 12 indicate the coordinates of the pixel, and the coordinates (1, 1) correspond to the pixel position at the upper left corner, which is the origin coordinate. Furthermore, the black dots in FIG. 12 indicate the pixels of the partial image data before the pasting position is corrected. Further, the hatched points in FIG. 12 indicate pixels expanded for correcting the pasting position.
First, when the coordinates (1, 1) are defined as the pixel position at the upper left corner, the X coordinate of the partial image data pasting position is an odd number (1 + 2i (i is an integer of 0 or more)). ) And the coordinates of the paste position at the right end are corrected to be an even number (2j (j is an integer of 1 or more)). For example, in the example of FIG. 12, the coordinates of the pasting position of the leftmost pixel of the partial image data before correction are 4, and the coordinates of the pasting position of the rightmost pixel are 14. Therefore, the partial image data is enlarged by one pixel in the left direction so that the coordinates of the pasting position of the leftmost pixel are odd. As a result, the coordinates of the pasting position of the leftmost pixel are 3 (1 + 2 × 1). In the example of FIG. 12, the partial image data may be reduced by one pixel in the right direction so that the coordinate of the leftmost pixel is 5. If the coordinates of the pasting position of the leftmost pixel of the partial image data before correction are an even number and the coordinates of the pasting position of the rightmost pixel are an odd number, the partial image data is not enlarged or reduced. You can simply shift the coordinates of the pasting position.

  On the other hand, as for the Y coordinate of the pasting position of the partial image data, the uppermost pasting position coordinate is an odd number (1 + 2i (i is an integer of 0 or more)), and the lowermost pasting position coordinate is an even number (2j (j is It is corrected to be an integer of 1 or more)). For example, in the example of FIG. 12, the coordinates of the pasting position of the pixel at the upper end of the partial image data before correction are 4, and the coordinates of the pasting position of the pixel at the lower end are 14. Therefore, the partial image data is enlarged by one pixel in the upward direction so that the coordinates of the attachment position of the uppermost pixel are odd. Thereby, the coordinate of the pasting position of the pixel at the upper end becomes 3. In the example of FIG. 12, the partial image data may be reduced by one pixel in the downward direction so that the coordinate of the uppermost pixel is 5. In addition, when the coordinates of the pasting position of the top pixel of the partial image data before correction are an even number and the coordinates of the pasting position of the bottom pixel are an odd number, the partial image data is not enlarged or reduced. You can simply shift the coordinates of the pasting position.

  When the coordinates (1, 1) are defined as the pixel position at the upper left corner, it is assumed that pairs are formed at intervals of two pixels in the horizontal direction from the leftmost pixel. At this time, if the X coordinate of the leftmost pixel of the partial image data is an odd number and the X coordinate of the rightmost pixel is an even number, it is possible to create a set within the range of the same partial image data. Similarly, it is assumed that pairs are formed at intervals of two pixels in the vertical direction from the upper end pixel. At this time, if the Y coordinate of the uppermost pixel of the partial image data is an odd number and the Y coordinate of the lowermost pixel is an even number, a set can be created within the range of the same partial image data.

Hereinafter, the effect of this embodiment will be described.
In order to facilitate understanding of the explanation of the effect, first, the downsampling format will be explained. When YC data is generated by performing matrix calculation on image data, YC data (YC444 format) in which each pixel has Y, Cb, and Cr information as shown in FIG. 13A is obtained. In the case of the YC444 format, as shown in FIG. 13A, the position of the Y data completely matches the positions of the Cb data and the Cr data.

  If image data in the YC444 format as shown in FIG. 13A is recorded on the recording medium 228 as it is, the capacity tends to increase. For this reason, normally, when recording image data in YC format, sampling processing (downsampling) is performed to reduce the number of C data with respect to the number of Y data. Here, the reason why the C data is reduced with respect to the Y data is that the characteristics of the human eye have characteristics that are high in sensitivity to luminance and low in sensitivity to color difference.

  FIG. 13B and FIG. 13C are in the YC422 format. In this embodiment, the YC422 format is used as the downsampling format when recording a still image. The YC422 format is a format that halves Cb data and Cr data with respect to Y data. In the YC422 format downsampling, a pair of 2 pixel units is created from the left end of the Cb data and Cr data, and one Cb data and one Cr data are generated from each pair.

  In FIG. 13B, the Cb data and Cr data of the left coordinate of each set are left, and the Cb data and Cr data of the right coordinate of each set are discarded. In this case, it can be considered that the position of the Y data matches the position of the Cb and Cr data. Here, in the following description, the format of FIG. 13B is referred to as a YC422 (match) format.

  In FIG. 13C, each set of Cb data and Cr data is averaged to be one. In this case, it can be considered that Cb and Cr data are located at the center of Y data adjacent in the horizontal direction. Here, in the following description, the format of FIG. 13C is referred to as a YC422 (center) format.

  When reproducing such image data in the YC422 format, it is necessary to restore the image data in the YC422 format to image data in the YC444 format. When restoring the YC422 (coincidence) format, Cb data and Cr data of the left coordinates that are not discarded among the two pixels that were paired at the time of downsampling are copied to the right coordinates. On the other hand, when restoring YC422 (center), the average value is copied to the two pixels that were paired during downsampling.

  FIG.13 (d) and FIG.13 (e) are YC420 formats. In this embodiment, the YC420 format is used as the downsampling format when recording a moving image. The YC420 format is a format in which Cb data and Cr data are 1/4 with respect to Y data. In YC420 format downsampling, a set of 4 pixels in total, 2 pixels in the vertical direction and 2 pixels in the horizontal direction, is created from the upper left corner of the Cb data and Cr data, and one Cb data and one Cr data are generated from each set To do.

  In FIG. 13D, the Cb data and Cr data of the upper left coordinates of each set remain, and the remaining Cb data and Cr data are discarded. In this case, it can be considered that the position of the Y data matches the position of the Cb and Cr data. In the following description, the format of FIG. 13D is referred to as a YC420 (match) format.

  Further, FIG. 13E shows two pixels adjacent in the vertical direction while discarding the right-side coordinate Cb data and Cr data of the two pixels adjacent in the horizontal direction in the same manner as YC422 (match). The average is one. In this case, it can be considered that the positions of Y data, Cb data, and Cr data coincide with each other in the horizontal direction, and that Cb and Cr data are located at the center of the Y data in the vertical direction. Here, in the following description, the format of FIG. 13E is referred to as a YC420 (moving image) format.

  When reproducing such image data in the YC420 format, it is necessary to restore the image data in the YC420 format to image data in the YC444 format. When restoring the YC420 (coincidence) format, the Cb data and Cr data of the upper left coordinates that have not been discarded among the four pixels that were paired at the time of downsampling are copied to the other three coordinates. On the other hand, when restoring YC420 (moving image), the average value is copied to the four pixels that were paired during downsampling.

  FIG. 14 is a diagram showing image reproduction when the coordinates of the pasting position of the partial image data are not corrected. Here, the example of FIG. 14 shows an example of reproducing image data that has been down-sampled in the YC422 (match) format.

  FIG. 14A shows image data in YC444 format before downsampling. As described above, at the time of YC422 (format) downsampling, the pixels shown in FIG. 14A are divided into sets of two-pixel intervals in the horizontal direction. At this time, when the pasting position is not corrected, there is a possibility that a pair of Cb data and Cr data having greatly different values of two pixels in the horizontal direction is generated. In the example of FIG. 14A, the groups in the third column and the fourth column correspond to such groups. Such a set is a set having no correlation between adjacent pixels such as a boundary portion between the background image data and the partial image data. When the origin coordinates are (1, 1), the left end of the partial image data. Such a group occurs when the coordinates of the pasting position are even numbers or when the pasting position at the right end is an odd number.

  When the image data shown in FIG. 14A is down-sampled in the YC422 (coincidence) format, the image data shown in FIG. 14B is obtained. As shown in FIG. 14B, the Y data is the same data as FIG. On the other hand, as for Cb data and Cr data, left coordinate data remains in the set of two pixel intervals in the horizontal direction, and right coordinate data is discarded.

  For reproduction, when the image data shown in FIG. 14B is restored to the YC444 format, the image data shown in FIG. 14C is obtained. Here, the Cb data and Cr data in the fourth column in FIG. 14C are a copy of the data in the third column, and the data in the third column is the fourth column in FIG. It is very different from the data. For this reason, the Cb data and Cr data in the fourth column in FIG. 14C are also significantly different from the Cb data and Cr data in the fourth column in FIG. The YC data in the fourth column in FIG. 14A shows black when converted to RGB. On the other hand, the YC data in the fourth column in FIG. 14C shows dark red when converted to RGB. When such image data is reproduced, a red display is made at a pixel that should originally be displayed in black, and the user feels uncomfortable.

  FIG. 15 is a diagram illustrating image reproduction when the coordinates of the pasting position are corrected. The example of FIG. 15 also shows a case where downsampling is performed in the YC422 (match) format.

  FIG. 15A shows image data in YC444 format before downsampling. In the present embodiment, when the origin coordinates are (1, 1), the coordinates of the leftmost pasting position of the partial image data are set to odd numbers, and the coordinates of the rightmost pasting position of the partial image data are set to even numbers. The X coordinate of the pasting position is corrected. By correcting the pasting position in this way, it is possible to create a set of two pixel intervals in the horizontal direction within the same partial image data. If within the same partial image data, it is considered that the correlation between adjacent pixels is high. Therefore, it is possible to reduce the possibility that a pair having a significantly different value of two horizontal pixels in Cb data and Cr data will be generated.

  When the image data shown in FIG. 15A is down-sampled in the YC422 (coincidence) format, the image data shown in FIG. 15B is obtained. As shown in FIG. 15B, the Y data is the same data as FIG. On the other hand, as for the Cb data and Cr data, only the left coordinate data remains in the set of two pixel intervals in the horizontal direction, and the right coordinate data is discarded.

  When the image data shown in FIG. 15B is restored to the YC444 format for reproduction, the image data shown in FIG. 15C is obtained. Here, the Cb data and Cr data in the fourth column in FIG. 15C are a copy of the data in the third column, and the data in the third column is the fourth column in FIG. 15A. The data is highly correlated with the data. When such image data is reproduced, the pixels that should be displayed in black are correctly displayed in black, and the user does not feel uncomfortable.

  Here, FIGS. 14 and 15 exemplify the YC422 (coincidence) format, but the description given with reference to FIGS. 14 and 15 is also applied when downsampling is performed in other formats. The For example, in the case of the YC420 (match) format, at the time of restoration, the data of the upper left pixel of the four pixels forming a set is copied to the other three pixels. Therefore, if there is a pixel having a low correlation with respect to the upper left pixel, the difference between the C data of the low correlation pixel before and after the restoration becomes large. When such an image is displayed, the user feels uncomfortable. In the case of the YC422 (center) or YC420 (moving image) format, the average value of the data of a pair of pixels is copied to each pair of pixels at the time of restoration. In this case, if the correlation of the four pixels constituting the set is low, the difference between the C data of all the pixels constituting the set before and after the restoration may increase. On the other hand, in this embodiment, it is possible to increase the correlation of the pixels that form a pair, so that it is possible to perform reproduction without feeling uncomfortable for the user regardless of the downsampling format.

  As described above, according to the present embodiment, composite image data configured by combining (pasting) a plurality of partial image data such as index image data, mosaic image data, and random tile image data is generated. At this time, the coordinates of the pasting position of the partial image data or the size of the partial image data is corrected. Thereby, downsampling and upsampling can be performed with pixels in the same partial image data. Therefore, it is possible to reduce the difference in color difference data between downsampling and upsampling, and to reduce the uncomfortable feeling during image reproduction.

[Modification]
Hereinafter, modifications of the present invention will be described.
First, in the above example, YC data is taken as an example, but the method of this embodiment can be applied even to data in another color space. For example, in the HD standard, RGB image data is converted into luminance (Y) and color differences (Pb, Pr). Also in this case, since the same downsampling as YC data is performed, the technique of this embodiment is effective.

  In the above-described example, the coordinates of the pasting position are corrected so that the coordinates of the end closer to the origin coordinates are odd numbers and the coordinates of the end far from the origin coordinates are even numbers. This is because the origin coordinates are odd numbers. If the origin coordinates are even (for example, (0, 0)), the coordinates of the pasting position are set so that the coordinates of the end closer to the origin coordinates are even and the coordinates of the end far from the origin coordinates are odd. Correct the coordinates.

  The above-described example describes an example in which image data is down-sampled by dividing it into sets of two-pixel intervals in the horizontal direction or the vertical direction. In practice, the technique of the present embodiment down-samples image data by dividing the image data into groups of m pixels in the vertical direction (m is an integer of 2 or more) or n pixels in the horizontal direction (n is an integer of 2 or more). It is applicable to the case where In this case, the X coordinate of the pasting position of the partial image data is 1 + n × i (i is an integer of 0 or more) when the origin coordinate is (1, 1). Correction is made so that the coordinates of the paste position on the right end are n × j (j is an integer of 1 or more). Further, the Y coordinate of the pasting position of the partial image data is 1 + m × i (i is an integer of 0 or more) when the origin coordinate is (1, 1), and the lower end Correction is performed so that the coordinates of the pasting position are m × j (j is an integer of 1 or more).

  In the above-described embodiment, the image sampling unit 223 is provided as an independent block. On the other hand, the image sampling unit 223 may be provided in the image processing unit 218 or in the display driver 220.

  In the above-described embodiment, the X coordinate (horizontal direction) and Y coordinate (vertical direction) of the pasting position are corrected. This is because the arrangement direction of the pixels is a direction along the horizontal direction and the vertical direction. In the case of using the image sensor 204 or the display unit 222 in which the pixel arrangement direction is oblique, the correction direction of the attachment position is also an oblique direction along the pixel arrangement direction.

  Furthermore, each processing method performed by the image processing apparatus in the above-described embodiment, that is, the processing shown in each flowchart can be stored as a program that can be executed by the microcomputer 230. In addition, memory cards (ROM cards, RAM cards, etc.), magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROM, DVD, etc.), storage media of external storage devices such as semiconductor memories, etc. are distributed. be able to. The microcomputer 230 reads the program stored in the storage medium of the external storage device, and the operation is controlled by the read program, so that the above-described processing can be executed.

  Although the present invention has been described above based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications and applications are naturally possible within the scope of the gist of the present invention. Further, in the description of each operation flowchart described above, the operation is described using “first”, “next”, and the like for convenience, but this does not mean that it is essential to perform the operations in this order. Absent.

  Further, the above-described embodiments include various stages of the invention, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some configuration requirements are deleted from all the configuration requirements shown in the embodiment, the above-described problem can be solved, and this configuration requirement is deleted when the above-described effects can be obtained. The configuration can also be extracted as an invention.

  DESCRIPTION OF SYMBOLS 1 ... Digital camera, 100 ... Interchangeable lens, 102 ... Lens, 104 ... Aperture, 106 ... Driver, 108 ... Microcomputer, 110 ... Flash memory, 112 ... Interface (I / F), 200 ... Camera body, 202 ... Mechanical shutter , 204... Image sensor, 206. Analog processing unit, 208. Analog / digital (A / D) conversion unit, 210. Bus, 212. SDRAM, 214. AE processing unit, 216 ... AF processing unit, 218. , 220 ... Display driver, 222 ... Display unit, 223 ... Image sampling unit, 224 ... Image compression / decompression unit, 226 ... Memory interface (I / F), 228 ... Recording medium, 230 ... Microcomputer, 232 ... Operation unit, 234 ... Flash memory, 2181 ... Basic image processing unit, 2182 ... Special effect picture Image processing unit

Claims (12)

An image composition unit for generating composite image data by pasting partial image data having a linear boundary in one direction in the horizontal direction or the vertical direction with respect to the background image data;
A boundary position determination unit that determines a position of the boundary of the partial image data in the background image data when generating the composite image data;
An image sampling unit that performs a sampling process on the composite image data to reduce the number of color difference components of the composite image data from the number of luminance components of the composite image data;
Comprising
The image processing apparatus according to claim 1, wherein the boundary position determination unit corrects the position of the boundary by enlarging or reducing the partial image data in accordance with a sampling process performed by the image sampling unit.   The boundary position determination unit performs the sampling process on the composite image data that has been subjected to the sampling process for reducing the number of color difference components of the composite image data than the number of luminance components of the composite image data. When restoration processing is performed to make the number of luminance component data and the number of color difference components of the composite image data the same, between the image data before the sampling processing and the image data after the restoration processing The image processing apparatus according to claim 1, wherein the position of the boundary is corrected so that a difference in color difference components is reduced.   In the boundary position determination unit, the sampling processing method uses a horizontal color difference component of the composite image data as 1 / n of the number of horizontal luminance components of the composite image data (n is an integer of 2 or more). When the boundary is a linear boundary along the vertical direction, the position of the boundary is corrected so that a set of n pixel intervals is formed in the horizontal direction of the partial image data. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   When the left end origin in the background image data is 1, the boundary position determination unit sets the position of the left end of the partial image data as coordinates of 1 + n × i (i is an integer of 0 or more). The image processing apparatus according to claim 3, wherein the correction is performed as follows.   The boundary position determination unit sets the position of the boundary at the right end in the partial image data to coordinates of n × j (j is an integer of 1 or more) when the left end origin in the background image data is 1. The image processing apparatus according to claim 3, wherein the correction is performed as follows.   In the boundary position determination unit, the sampling processing method may be configured such that the color difference component in the vertical direction of the composite image data is 1 / m (m is an integer of 2 or more) of the number of luminance components in the vertical direction of the composite image data. When the boundary is a linear boundary along the horizontal direction, the position of the boundary is corrected so that a set of m pixel intervals is formed in the vertical direction of the partial image data. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The boundary position determination unit sets the position of the boundary at the upper end in the partial image data as coordinates of 1 + m × i (i is an integer equal to or greater than 0) when the upper end origin in the background image data is 1. The image processing apparatus according to claim 6, wherein the correction is performed as follows.   The boundary position determination unit sets the position of the boundary at the lower end in the partial image data as coordinates of m × j (j is an integer equal to or greater than 1) when the upper end origin in the background image data is 1. The image processing apparatus according to claim 6, wherein the correction is performed as follows.   The image processing apparatus according to claim 1, wherein the image synthesis unit synthesizes a plurality of partial image data with the background image data.   An imaging apparatus comprising the image processing apparatus according to claim 1. The image composition unit generates the composite image data by pasting the partial image data having a linear boundary in one direction in the horizontal direction or the vertical direction with respect to the background image data,
The boundary position determination unit determines the position of the boundary of the partial image data in the background image data when generating the composite image data;
The boundary position determination unit corrects the position of the boundary by enlarging or reducing the partial image data according to a sampling processing method by the image sampling unit,
The image sampling unit performs a sampling process on the composite image data so that the number of color difference components of the composite image data is less than the number of luminance components of the composite image data.
An image processing method. A procedure for pasting partial image data having a linear boundary in one direction in the horizontal direction or the vertical direction with respect to the background image data in the image composition unit to generate composite image data;
A procedure for causing the boundary position determination unit to determine the position of the boundary of the partial image data in the background image data when generating the composite image data;
A procedure for causing the boundary position determination unit to correct the position of the boundary by enlarging or reducing the partial image data according to a sampling processing method by the image sampling unit;
A procedure for causing the image sampling unit to perform a sampling process for reducing the number of color difference components of the composite image data relative to the number of luminance components of the composite image data with respect to the composite image data;
An image processing program for causing a computer to execute.

Images