JP2020150517A - Image processing apparatus, image processing method, computer program and storage medium - Google Patents

Abstract

To provide an image processing apparatus which can properly discriminate and process an image according to the presence or absence and a type of combining process.SOLUTION: An image processing apparatus according to an embodiment includes: combining means that combines a plurality of photographed images to each other to generate a composite image; record control means that records the photographed image or the composite image as a recorded image in a recording portion; and determining means that determines a photographing mode or a method of combination which is processed in the combining means. The record control means records data of a type which is determined on the basis of a determination result based on the determination, in association with the recorded image that has been photographed or combined by the photographing mode or the combining method which has been determined by the determining means.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image processing apparatus or the like capable of synthesizing a plurality of images or reproducing a composite image.

In an image pickup device such as a digital camera or a mobile terminal with a camera, it is possible to add shooting conditions and the like to the attribute information of the image data obtained by shooting and record it as one image file. Patent Document 1 discloses a technique of recording the number of times of multiplex as metadata in a file header when recording composite image data obtained by multiple-exposure imaging in which a plurality of images are combined at the time of photographing. In Patent Document 1, it is also determined that the image data is an image taken by multiple exposure by a plurality of times.

Japanese Unexamined Patent Publication No. 2012-024978

However, with the above-mentioned existing technique, it is only possible to identify whether or not the image is taken in a shooting mode involving one composition called multiple exposure shooting. No consideration was given to images generated by shooting modes involving various compositing or various compositing processing sequences, and general-purpose incidental information for identifying attribute information of the images.
The present invention has been made in view of such a problem, and provides an image processing apparatus capable of appropriately discriminating and processing an image according to the presence or absence and type of composition processing.

In order to solve the above problems, the image processing apparatus of the present invention
A compositing means that synthesizes a plurality of captured images to generate a composite image,
A recording control means for recording the captured image or the composite image as a recorded image in the recording unit, and
It has a shooting mode or a determination means for determining a synthesis method performed by the synthesis means.
The recording control means is characterized in that a recorded image photographed or synthesized by a shooting mode or a synthesis method determined by the determination means is associated with type data determined based on the determination result of the determination and recorded. To do.

According to the present invention, it is possible to obtain an image processing apparatus capable of appropriately discriminating and processing an image according to the presence or absence and type of composition processing.

It is a block block diagram of the image processing apparatus in this Example. It is a figure which shows the example of the data structure of an image data file. It is a figure which shows the example of the data structure of the synthetic information part. It is the whole flowchart of the image processing apparatus in an Example. It is a flowchart of the shooting mode in an Example. It is a flowchart of the reproduction mode in an Example. It is a figure which shows the image list display example in an Example.

Hereinafter, examples of the present invention will be described in detail with reference to the drawings. As the image processing device according to the embodiment of the present invention, an example of an image pickup device will be described. However, the image processing device includes a general-purpose image processing device (hereinafter referred to as a PC) such as a personal computer, a digital camera, a digital movie camera, and the like. Further, it can be applied to any device that can display a list of image files, such as a mobile phone, a smartphone, a so-called tablet terminal, and a car navigation device, and the image processing device of the present invention includes those product forms.

FIG. 1 is a block diagram showing a functional configuration of an image pickup apparatus 100 as an example of the image processing apparatus of this embodiment. The control unit 101 includes a CPU (Central Processing Unit) as a computer, and functions as a control means for controlling the entire image pickup device.
The control unit 101 reads a computer program for operating each component included in the image pickup apparatus 100 from a ROM (read only memory) 102 as a storage medium, and expands the computer program into a RAM (random access memory) 103. Execute. The ROM 102 is a rewritable non-volatile memory, and stores parameters and the like necessary for the operation of each component included in the image pickup apparatus 100 in addition to the operation program executed by the CPU. The RAM 103 is a rewritable volatile memory, and is used as a temporary storage area for data output in the operation of each component included in the image pickup apparatus 100.

The image pickup optical system 104 includes a lens, a diaphragm, a drive unit for driving the optical members thereof, and the like, and the light from the subject is imaged on the image pickup unit 105. The image pickup unit 105 includes an image pickup device such as a CCD (Charge Coupled Device) image sensor and a CMOS (Complementary Metal-Node Semiconductor) image sensor. The image pickup element functions as an image pickup means for performing photoelectric conversion on the optical image formed by the image pickup optical system 104 to form an analog image signal and generate a captured image. The analog image signal is A / D converted by the A / D (analog / digital) conversion unit of the imaging unit 105 to become digital image data and stored in the RAM 103. In this embodiment, the image processing device may have the image pickup means in the image processing device, but the image processing device may not have the image pickup means.

The image processing unit 107 applies various image processing such as white balance adjustment, color interpolation, reduction / enlargement, and filtering to the image data stored in the RAM 103. The recording medium 108 is a memory card or the like that can be attached to and detached from the image pickup apparatus 100, and functions as a recording unit for recording image data stored in the RAM 103. Here, the control unit 101 also functions as a recording control means for recording a captured image or the composite image as a recorded image on the recording medium 108. The image data processed by the image processing unit 107, the image data after A / D conversion by the imaging unit 105, and the like are recorded on the recording medium 108 as recorded image data. Further, the image processing unit 107 synthesizes a plurality of image data obtained by the photographing and output from the imaging unit 105 by a method based on a photographing mode set in advance at the time of photographing by the imaging unit 105.

For example, the control unit 101 gives an instruction to the image processing unit 107 according to a signal indicating a shooting mode set by the user by the operation unit 111 or a shooting mode determined based on a shooting scene determined from the image by the control unit 101 at the time of imaging. put out. As a result, the compositing process is performed according to the shooting mode. Further, the control unit 101 determines the type data indicating the type of the image related to the composition by determining the shooting mode at this time, and determines the image data of the composite image after the composition process is performed by the image processing unit 107. The type data is associated and recorded on the recording medium 108. For example, in the present embodiment, an image file in which at least a part of the header information is configured in accordance with a general-purpose standard (Exif standard) is generated. Then, by recording the type data in the data area conforming to the standard of the header information, the image data also recorded in the image data unit in the image file is associated with the type data.

The image processing unit 107 performs a compositing process on a plurality of image data input based on the instruction of the control unit 101. At this time, at least a part of the data to be synthesized may be the data read from the ROM 102 or the recording medium 108. Further, the image processing unit 107 synthesizes a plurality of image data recorded in the ROM 102 or the recording medium 108 at the time of reproduction according to the method of synthesis processing set by the user by the operation unit 111. Each shooting mode and the corresponding compositing process and various compositing processes during playback will be described later.

The display unit 109 includes a display device such as an LCD (Liquid Crystal Display), and displays a through image of the captured image acquired by the imaging unit 105. Further, the display unit 109 displays data such as characters and images, and a so-called graphical user interface such as a menu.
The distance map acquisition unit 110 acquires information related to the distance distribution of the subject as a distance map by, for example, a TOF (Time of Flight) method or a stereo camera. Data such as image shift amount distribution and defocus amount distribution between images with parallax related to the distance map or the captured image represent the distribution of depth information in the depth direction of the subject. In the following, data representing the distribution of depth information will be referred to as distance map data. The acquired distance map data is stored in the RAM 103 as digital data. Many methods are well known for generating map data, and any of them may be used.

The operation unit 111 includes buttons, dials, touch panels, slide bars, and the like of various physical operation members, receives operations by the user, and outputs an instruction signal to the control unit 101 and the like.
The communication unit 106 is a wireless LAN interface or the like for communicating with an external device, but may be of any type as long as it can communicate with an external device such as a wired LAN interface or a USB interface. Image files can be transmitted and received to and from an external device (for example, a PC, a smartphone, etc.) via the communication unit 106.
(Data structure of image file)

FIG. 2 shows the data structure of the image file generated and managed by the image pickup apparatus 100. In this embodiment, it is assumed that a file format according to the Exif standard is adopted. The image data file 201 is generated, for example, at the time of imaging by the imaging unit 105, acquired via the communication unit 106, or stored in the recording medium 108 or the ROM 102. When the image pickup apparatus 100 processes the image data file 201, it may be temporarily copied to the RAM 103.

The image data file 201 has a marker SOI (202) indicating the start of an image at the beginning, and then has an application marker APP1 (203) (APP1 area) as a header area.
Within the application marker APP1 (203), the size of APP1 (204), the identification code (205) of APP1, DateTime (206) indicating the date and time when the image was created, DateTimeOriginal (207) indicating the date and time when the image data was generated, and the image. It is composed of a Relation Information (208), shooting information, etc. (209) and a thumbnail image (210) showing the relationship between the two.

The DateTime (206) and DateTimeOriginal (207) store the shooting date / time information stored in the RAM 103. In addition, Relation Information (208) describes, for example, predetermined information indicating that there is a relationship between a plurality of captured images to be combined. This predetermined information may be the image number of the related image recorded first, or may be information indicating a specific shooting mode.
The shooting information and the like (209) include, for example, the shooting date and time, shooting parameters (exposure time, aperture value, ISO sensitivity, focal length, presence / absence of auxiliary light source, etc.), and the composite information unit 2091 according to this embodiment. The synthesis information unit 2091 holds the synthesis-related information generated and processed in this embodiment.

The recorded image data is obtained from the image data unit 215 in which the quantization table DQT (212), the Huffman table DHT (212), the frame start marker SOF (213), the scan start marker (214), and the compressed data are stored. It is configured and terminated with a marker EOI (216) indicating the end of the image data.
Since the file structure shown in this embodiment is adapted to the structure of the Exif standard, it can be compatible with the Exif standard. However, the recording structure of various information such as the synthetic information of this embodiment is not necessarily limited to the structure of the Exif standard. For example, it may be recorded in a file different from the image data file 201, or may be recorded in a special database included in the image pickup apparatus 100.

FIG. 3 shows an example of the information held in the synthetic information unit 2091. In this embodiment, based on the composite image and the information related to the composite image on the image processing unit 107 in the subsequent stage, the application of the external device, the network server, etc., the optimization and restoration of the composite processing, AI, etc. are used. Record various information related to this synthesis for learning. Moreover, by storing at least a part of this information in the general-purpose Exif area, compatibility with various Exif-compliant devices can be made, and it can be referred to by a general-purpose application.

The composite information unit 2091 is provided with areas for describing the image identification information 301, the composite type information 302, the source image number information 303 of the composite image, the composite image information 304, and the source image information 305 of the composite image.
The image identification information 301 is type data mainly indicating the type of the image related to the composition, and any one of the classification information such as "non-composite image", "general composite image", and "composite image acquired at the time of shooting" is stored. To. The "composite image acquired at the time of shooting" is an image synthesized from only a series of a plurality of source images (provisional recorded images) acquired from the image sensor at the time of shooting (the use of all of them is not essential). In addition, it is a classification attached to a composite image that can be regarded as having constant shooting conditions (aperture value, subject distance, light source, lens focal length, etc.) and subject range (shooting field of view) when all source images are acquired. ..

"3" is assigned as the first type data to the "composite image acquired at the time of shooting". Here, the shooting condition that can be regarded as constant in the "composite image acquired at the time of shooting" is that the "general composite image" is a classification attached to the composite image synthesized from a plurality of shot images (source images), and the method of compositing is described. Not limited. "2" is given to the "general composite image" as the second type data.
The "non-composite image" is a classification given to an image that cannot be regarded as a composite image classified into a "general composite image" and a "composite image acquired at the time of shooting", for example, an image that is considered not to be composited. "1" is given as the type data to the "non-composite image". A value other than any of the above "1" to "3" may be prepared and used as the image identification information 301. For example, when it is unclear which classification is assigned, "0" may be added as the type data.
(Each synthesis mode and its classification)

Next, an example of assigning the image identification information 301 to the image obtained by the one accompanied by the compositing process in various shooting modes and the composite image obtained by the various compositing processes during reproduction will be described.
In this embodiment, as described above, when the shooting conditions and the angle of view of the plurality of images used for compositing at the time of shooting are the same, "3" is generated as the first type data. Further, when one of the shooting conditions and the angle of view of the plurality of images used for compositing is different, "2" is generated as the second type data. In the case of non-synthesis, "1" is generated as the third type data.

For example, a mode in which HDR (High Dynamic Range) synthesis is performed at the time of shooting as one of the types of synthesis of a multi-shot system (a shooting mode involving multiple shootings at different timings or an imaging optical system is hereinafter referred to as a multi-shot system). There is. In this HDR composition mode at the time of shooting, there are some source images such as ISO and shutter speed whose shooting conditions cannot be regarded as constant. Therefore, "2", which is the second type data, is added as the image identification information. Here, HDR composition at the time of shooting is, for example, taking a plurality of images having different exposures (ISO, shutter speed, etc.), developing each image with a different gamma, and synthesizing the development results. The HDR composition mode at the time of shooting is a mode in which an appropriate image is extracted and combined according to the brightness value of the image from a plurality of images having different exposure conditions.

Further, even when HDR processing is performed by an image processing device or a PC application during playback as one of the types of multi-shot compositing (HDR compositing mode during playback), the second type of data is "2" as image identification information. Is given. This is because the image was not composited at the time of shooting. This is also because there are some source images such as ISO and shutter speed whose shooting conditions cannot be regarded as constant. Here, the HDR processing performed by an image processing device, a PC application, or the like at the time of reproduction is, for example, developing an image selected by a user with gamma according to an exposure difference and synthesizing the development results.

Further, as one of the types of multi-shot synthesis, for example, in the case of a composite image obtained in the HDR painting-like shooting mode in which HDR painting-like shooting is performed, "2" is added as image identification information in the same manner as described above. The HDR painting-like shooting is a mode in which a composite image having a painting-like atmosphere is generated by performing gradation processing from a plurality of images having different exposures (ISO, shutter speed, etc.).
Further, as one of the types of composition, "2" is added as image identification information even in the case of a composite image obtained in a shooting mode in which a plurality of images are taken to generate a composite image in order to achieve a multi-shot noise reduction function. This is because the shooting conditions can be regarded as constant, but the function is premised on handheld shooting, and the angle of view of each shot image may differ due to camera shake or the like, and the angle of view cannot be regarded as constant. The multi-shot reduction function is a shooting mode in which multiple images taken under the same shooting conditions are aligned based on the detected shake amount and then combined to obtain a composite image with reduced noise. It is used when it is dark and there is a lot of noise.

Further, "2" is also added as image identification information in the case of a composite image obtained in the handheld night view shooting mode on the premise that the night view is shot a plurality of times without a strobe in handheld shooting. This is also because the shooting conditions are constant, but this is a function premised on handheld shooting, and the angle of view of each shot image may differ due to camera shake, and the angle of view cannot be regarded as constant.
Further, as one of the types of composition, "2" is added as the image identification information when the night view is continuously shot with a strobe and without a strobe by handheld shooting. This is a function premised on handheld shooting, and the angle of view cannot be regarded as constant. This is also because there are captured images in which the conditions related to the "light source" change among the imaging conditions.

Next, the case of so-called creative filter type shooting will be described. The creative filter type shooting is a shooting mode in which a shot image is recorded by applying a special image processing effect.
Images for so-called rough monochrome shooting, soft focus shooting, diorama style shooting, toy camera shooting, oil painting style shooting, watercolor style shooting, fisheye style shooting, beautiful skin shooting, background blur shooting, etc. "1" is added as the identification information. Further, "1" is added as the image identification information to the image obtained in the editing mode (application) in which the image processing equivalent to the image processing performed on the captured image in these modes is applied to the image at the time of reproduction. The reason for assigning "1" is that although some image processing performed in these shooting modes or editing modes (applications) involves compositing images, a plurality of shot images (source images) are used. This is because it is not a synthetic process.

Rough monochrome photography is photography in which one image is photographed, random noise is superimposed on the photographed image, and monochrome processing is performed to produce an effect similar to that of a monochrome film. The soft focus shooting is a shooting in which, for example, one image is shot, and the shot image is subjected to LPF (Low Pass Filter) and the original shot image is combined at a constant ratio. What is diorama-style shooting? For example, one image is taken, and the shot image with LPF applied and the original shot image are combined in a part of the image to blur a part of the image to create a real subject. This mode generates an image that gives an image effect as if the image was taken as a diorama.

The toy camera shooting is a shooting in which, for example, one image is taken and an image effect is added as if the image was taken by the toy camera. Image processing performed by toy camera shooting, for example, reproduces a sweet focus effect by synthesizing an LPF-applied image and the original image, and bright finishes, biased color tones, and peripheral dimming due to brightness and color conversion. It is a process to reproduce the effect such as. The oil painting style photography is, for example, taking one image, performing image processing on the photographed image, and performing image processing to make the photographed image look like an oil painting by combining with the original image.

In the watercolor-style photography, for example, one image is taken, the photographed image is subjected to image processing, and the photographed image is combined with the original image to be subjected to image processing to make the image look like a watercolor painting. Fisheye-style photography is, for example, taking one image and combining (interlacing) pixels so that each area has a fisheye lens-like distortion, giving the effect of shooting with a fisheye lens. Is. The skin-beautifying shooting is a shooting mode in which the NR (Noise Reduction) of the skin-colored portion is strengthened to reduce the sense of resolution and smooth the texture of the skin. The background blur shooting is a shooting in which, for example, two images are taken by changing the aperture, the main subject and the background are separated from the difference between the two images, and the background is blurred. In this case, the composite source image is one image, and a plurality of images are taken to determine the blurred area. In some cases, background blur shooting is realized by opening the aperture.
Date imprint photography is the superimposition of date text data on an image.

Further, in the case of multiple exposure photography (multiple composition mode) as one of the types of composition, "2", which is the second type data, is added as the image identification information. This is because the shooting conditions and the angle of view can be freely set by the user and cannot be regarded as constant. This is because the recorded image can be selected as the first image in the case of continuous shooting priority. The multiple exposure shooting includes, for example, an addition mode, an addition averaging mode, a comparative bright mode, a comparative dark mode, and the like. The addition mode is a mode in which a plurality of images are taken and RAW images are simply added. The addition averaging mode is a mode in which a plurality of images are taken and RAW images are simply added and averaged. This is for improving the S / N of, for example, a very dark image. The comparative brightness mode is to perform a comparative brightness composition process in which the one whose pixel value at the same coordinate is not low is adopted when a plurality of images are taken and a RAW image is combined.

The comparative dark mode is a process of performing a comparative dark compositing process in which the one in which the pixel values at the same coordinates are not high is adopted when a plurality of images are taken and the RAW images are combined.
Further, in the case of panoramic photography, depth composition photography, or starry sky portrait photography as one of the types of composition, "2" is added as image identification information. This is because the source images having different angles of view are combined. In the panoramic shooting, the user pans the camera to shoot at a plurality of angles of view and combines them. By this composition, a wide-angle image is generated by connecting a plurality of images captured at a plurality of different angles of view. Wide-angle images obtained by compositing in panoramic photography include images in all directions (360 degrees).
Depth composite shooting is to take a plurality of images by shaking the focus, synthesize the areas in which the plurality of images are in focus based on the distance map acquired by the distance map acquisition unit 110, etc., and have a depth higher than each image. It is a shooting that produces a deep image of. The starry sky portrait photography is a shooting in which, for example, after strobe photography in which the strobe is fired, the strobe does not emit light and continuously shoots, for example, two images, and for example, three images are combined.

As one of the types of composition, "3" is added as image identification information in starry sky trajectory photography. This is because the angle of view (range of the subject, shooting field of view) and shooting conditions can be regarded as constant. The starry sky trajectory shooting is a shooting mode in which a plurality of starry skies are shot and comparatively brightly combined.
Further, consider the case where the image pickup unit 105 is, for example, a so-called laminated sensor in which the image pickup element and the signal processing unit have an integrated multilayer structure. At this time, even if there is image composition performed in the stacking sensor, the output from the stacking sensor may be one image data whose data format is almost the same as that of the image in which normal composition is not performed. "1" is added as the image identification information.

However, if the control unit 101 can instruct the image pickup unit 105, which is a laminated sensor, to perform the synthesis process to the signal processing unit in the image pickup unit 105, the image pickup unit 105 outputs in response to the synthesis instruction. "2" is added as image identification information to the image data.
In addition, as one of the types of compositing, there is a shooting mode called super-resolution shooting, in which the image sensor or the imaging optical system is moved minutely to perform a plurality of shot images, and the resolution is increased by synthesizing those images. .. In the case of this super-resolution shooting mode, since the shooting field of view cannot be regarded as constant at the point where the images are taken with the angle of view shifted, "2" is added as the image identification information.
The composition type information 302 is information indicating the type of composition processing mode, composition processing, and composition condition under which the composition image is synthesized. The composition type information 302 is at least information corresponding to the present image data from the type data corresponding to a plurality of types for composition such as the multiple exposure mode, additive synthesis, comparative bright composition, HDR composition, and panorama composition as described above. One is stored.

The source image number information 303 is information about the number of images acquired to generate the image data of the image data file 201, and is a composite image indicating the number of captured images (source images) and the number of source images used for the composite image. Information on the number and image number used is included. The number of captured images indicates the total number of images captured to generate the present image data. This includes, for example, the number of image data and the like that are not used in the composite image due to poor image quality due to camera shake or the like. The number of composite images indicates the number of images actually stored in the image data unit 215 of the image data file 201 and used to generate the image data. The image number used is information in which the image numbers related to each of the image data synthesized at the time of generating the present image data, which is counted in the number of composite images, are arranged.

In the composite image information 304, information on statistical values related to the shooting conditions of the images used for synthesis is recorded. In FIG. 3, the maximum value of the exposure time in the image used for the composition is recorded as the statistical value of the exposure time. Further, as statistical values of aperture value, ISO sensitivity, and focal length, an average value of aperture values of a plurality of images used for synthesis, an average value of ISO sensitivity, and an average value of focal length are recorded.
The statistical value related to each shooting condition recorded in the composite image information 304 is not limited to one type, and is an arbitrary statistical value depending on the purpose of data collection or application such as maximum value, minimum value, total value, average value, and dispersion value. May be recorded in multiple types. Further, in this embodiment, only the statistical values of the shooting parameters for the images used for compositing are stored, but the present invention is not limited to this, and the images taken to generate the image data and not used for compositing are also stored. One or more types of information on the above statistical values including the information may be stored.

The source image information 305 is information regarding the shooting parameters of each image acquired to generate the image data of the image data file 201. Shooting parameters such as exposure time, aperture value, ISO sensitivity, and focal length are recorded in order for each shot image. By storing such information, it is possible to determine how many images taken by shooting a certain scene and how many images were taken with what shooting parameters were obtained and generated. Information can be collected.
(Overall operation of digital camera)

Hereinafter, the overall operation of the image pickup apparatus 100 of the present embodiment will be described with reference to a flowchart for explaining the overall operation of the image pickup apparatus 100 of the present embodiment of FIG. The flowchart shown below including FIG. 4 is realized by the control unit 101 reading the program stored in the ROM 102 and performing arithmetic processing and control.

(Overall flow)
In step S401, the power switch included in the operation unit 111 is operated by the user to switch the power on, so that the control unit 101 initializes the flags, control variables, and the like. In addition, shooting mode information is acquired, and judgment and setting are performed. At this time, when the shooting mode accompanied by the compositing process is set, the setting information of the number of images N used for compositing is also acquired, and the determination and the setting are performed. In step S402, the control unit 101 determines the setting position of the mode changeover switch included in the operation unit 111, proceeds to step S403 when the shooting mode is set, and steps S404 when a mode other than the shooting mode is set. Proceed to. Details of the shooting mode processing in step S403 will be described later.

In step S404, the control unit 101 determines whether the mode changeover switch is set to the reproduction mode, proceeds to step S405 if it is set to the reproduction mode, and proceeds to step S406 if a mode other than the reproduction mode is set. move on. Details of the reproduction mode processing in step S405 will be described later. In step S406, the control unit 101 executes processing according to the other selected modes, and proceeds to step S407 after the execution. Other modes include a communication mode for transmitting and receiving files stored in the recording medium 108.

In step S407, the control unit 101 determines the setting position of the power switch. If the power switch is set to power on, the process returns to step S402, and if the power switch is set to power off, the process proceeds to step S408. In step S408, the control unit 101 performs a predetermined end process. The termination process includes the following processes. That is, the control unit 101 changes the display of the display unit 109 to the finished state, closes the lens barrier, and protects the image pickup unit 105. Further, the control unit 101 records parameters including flags and control variables, setting values, and setting modes in a non-volatile memory or the like, and then shuts off the power supply to a portion that does not require power supply.
(Shooting / recording processing)
FIG. 5 shows a flowchart of the photographing / recording process of the imaging device 100 in this embodiment.

(Shooting mode processing)
FIG. 5 is a flowchart showing details of the shooting mode processing in step S403 of FIG.
When the shooting mode is started, in step S501, the control unit 101 sequentially outputs the images captured by the display unit 109, and performs a through display in which the images are sequentially displayed on the display unit 109.

In step S502, the control unit 101 determines whether or not the shutter switch SW1 has been pressed by the user. If it is pressed, the process proceeds to step S503, and if it is not pressed, the shutter switch SW1 is waited for to be pressed. In step S503, the control unit 101 performs a shooting preparation operation. That is, distance measurement is performed to focus the photographing lens 103 on the subject, and photometry is performed to determine the aperture value and the shutter time. Based on the photometric result, the control unit 101 also sets the flash emission amount and the like if necessary.

In step S504, the control unit 101 determines whether or not the shutter switch SW2 has been pressed by the user, and if it is pressed, the process proceeds to step S506. If it is not pressed, the process proceeds to step S505, and the control unit 101 determines whether the shutter switch SW1 is continuously pressed. When the shutter switch SW1 is released, the process returns to step S502. If Yes in step S505, the process returns to step S504.
In step S506, the control unit 101 acquires the date and time when the shooting starts from the RTC (Real Time Clock) in the control unit 101 and stores it in the RAM 103. Subsequently, in step S507, the synthesis processing flag i is set to 0.
Subsequently, in step S508, the control unit 101 opens the shutter in the optical system 104 having an aperture function according to the aperture value according to the photometric data stored in the RAM 103 or the ROM 102, and starts exposing the image sensor of the image pickup unit 105.

In step S509, the control unit 101 outputs image data from the image pickup unit 105, and then uses the image processing unit 107 to perform image processing such as compression / coding processing. After that, in step S510, the control unit 101 determines whether or not the set shooting mode is a shooting mode involving compositing processing. That is, in step S401, the control unit 101 functions as a determination means for determining the photographing mode or the synthesis method performed by the synthesis means.
If the shooting mode does not include a compositing process, the process proceeds to step S511, and the image identification information 301 to be recorded in the header is determined as the type data “1” indicating that it is a normal image (non-composite image) and stored.

At this time, the control unit 101 functions as a type data generation means for generating type data according to the type of composition associated with the setting of the shooting mode, for example. As described above, the control unit 101 determines the type data (“1” as the image identification information 301) when the image processing unit 107 does not synthesize the data. Then, in step S512, the image data is recorded on the recording medium 108 together with the image data. At this time, the control unit 101 functions as a recording control means for recording the type data corresponding to the type of synthesis in the APP1 (203), which is the header area of the Exif standard, in association with the recorded image. After step S512, the process returns to step S501.

If it is determined in step S510 that the synthesis processing mode is set, the process proceeds to step S513, and synthesis is performed in the preset synthesis processing mode. At this time, the control unit 101 functions as a compositing means for composing a plurality of captured images by a plurality of types of methods to generate a composite image. Next, in step S514, it is determined whether or not the number of captured images used for compositing has reached the preset number N of images, and if not, the process returns to step S508 to perform a new exposure operation. When the number of images reaches N, the file name and shooting date and time are acquired in step S515, and in step S516, a header for recording these data and type data according to the type of composition in APP1 (203) as a header area. Generate information.

In step S516, the control unit 101 determines the type data to be recorded in association with the image data according to the set shooting mode. As described above, when the control unit 101 is synthesized by the synthesis means, the control unit 101 has at least two types of data (“2” or “3” as the image identification information 301) depending on the type of synthesis. Generate one of them. In addition, the image processing unit 107 determines an appropriate classification from at least one of three types of data, including the case where the composition is performed and the case where the composition is not performed. Next, in step S517, the control unit 101 displays an image on the display unit 109 and records on the recording medium 108. At this time, the control unit 101 functions as a recording means for recording the type data in the APP1 (203), which is the header area of the Exif standard, in association with the recorded image.

At this time, when the type data is a predetermined type data (composite identification information is 3), at least "the number of captured images used for compositing" and "shooting conditions of the captured images used for compositing" are set. For example, the exposure time and the like are recorded in the same region APP1 (203). After step S517, the process returns to step S501.
In the above embodiment, the composition is executed every time the exposure is completed, but it goes without saying that the composition process may be performed after N times of imaging.
(Playback mode processing)
FIG. 6 is a flowchart showing details of the reproduction mode process of step 405 of FIG.
The playback mode is started by the playback instruction via the mode changeover switch by the user. Alternatively, this flow is started when the power of the device is turned on and started while the playback mode is set.

First, in step S601, the control unit 101 acquires initial image information from the recording medium 108. This initial image is the latest image or the image displayed in the previous playback. By acquiring the initial image information before calculating the total number of images and creating the search list, which will be described later, the image of the initial image can be displayed as soon as the playback mode is entered, and during processing such as calculating the total number of images. , You can prevent the problem that nothing is displayed.
In step S602, the control unit 101 determines whether or not the initial image information in step S601 has been acquired correctly. This judgment is made because there are cases where there is no image or there are cases where image information cannot be obtained due to media defects. If the initial image information can be obtained, it is determined that there is at least one image. If there is an image, the process proceeds to step S603, and if there is no image, the process proceeds to step S609.

In step S603, the control unit 101 reads the initial image from the recording medium 108 based on the initial image information acquired in step S601. In step S604, file analysis processing is performed, and the control unit 101 analyzes and acquires attribute information such as shooting information and composite information of the read initial image.
That is, at this time, the control unit 101 sets the type data according to the type of composition recorded in association with the recorded image, the information indicating the composition type, the number of captured images used for the composition, and the captured image used for the composition. Type for acquiring shooting conditions It functions as a data acquisition means. Also, check if the image data is corrupted.

In step S605, the control unit 101 displays the read initial image, and selectively displays attribute information such as shooting information and composite information together with the image as needed. This selective display will be described later in FIG. Further, when it is analyzed that the data is invalid such that a part of the file of the initial image is broken according to the file analysis result in step S604, the control unit 101 also displays an error.
In step S606, the control unit 101 starts calculating the total number of images recorded on the recording medium 108 to acquire the total number of images. This process works asynchronously with the image display process, and it is possible to proceed to the next image display process without waiting for the completion. By processing in this way, even when many images are recorded on the recording medium 108 and it takes time to calculate the total number of images, the images can be viewed.

In step S607, the control unit 101 analyzes all the images in the recording medium 108 and starts creating a search list as the search result. The search list is a list in which attribute information added to each image is acquired in advance and managed collectively. By creating this search list, for example, it is possible to collectively set images having a certain attribute as playback targets or deletion targets. Similar to the calculation of the total number of images, the search list creation process is performed asynchronously with the image display process, and the next image display process is possible. In step S608, the control unit 101 waits for the user to input playback. Proceed to the state.
If there is no image in step S602, the process proceeds to step S609, and the control unit 101 displays a message such as "there is no image" via the display unit 109 to indicate that there is no image. The process proceeds to step S608.

(Screen display)
Next, a method of displaying image data in this embodiment will be described. FIG. 7A shows an example of a screen displayed on the display unit 109 when the image display application is operated in the present device.
FIG. 7A shows that the image data Image1 to Image12 stored in the RAM 103 are displayed in a list. This image is obtained from, for example, a digital camera or a mobile phone. It may also be downloaded from a server on the network. It is also possible to use thumbnails of image data without using the image data body. When the display switching button 401 is selected (touched) in the state of FIG. 7 (A), the screen transitions to the screen of FIG. 7 (B).

As shown in FIG. 7B, for the image data to which the composition-related information is added, an icon in the shape of a “pin” (hereinafter, pin icon) is superimposed and displayed on the thumbnail of the image. That is, the pin icon is displayed for the image file having the composition-related information. In the example of this screen, image identification information 301 ("1" to "3", etc.), composition type information 302, etc. are described as composition-related information in Image 1, 3, 4, 6, 7, 8, 10, and 11. It shows that there is.

Of course, the pin icon does not necessarily have to be completely superimposed on the thumbnail, and it is sufficient to distinguish between image files that have / do not have composition-related information, so a part of the pin icon may be superimposed or superimposed on the thumbnail in the image data. It does not have to be displayed in the vicinity. On this screen, a pin icon is superimposed and displayed on the image data in which the composition-related information is described. By checking this screen, the user can easily check which image data the composition-related information is added to.

When the display switching button is further clicked in the state of FIG. 7B, the screen transitions to the screen of FIG. 7C. As shown in the screen of FIG. 7C, the image identification information 301 and the composition type are changed by changing to the icons and characters corresponding to the image identification information 301 and the composition type information 302 instead of the pin icons up to that point. Information 302 and the like can be easily identified.
For example, for Images 1, 4, 10 and 11, "P" indicating panoramic composition or an icon indicating panorama is superimposed and displayed. This is to indicate that the composition type information 302 is panoramic composition, that is, the composition using panoramic composition is the composite image data.
For Images 6 and 8, "HDR" indicating HDR composition or an icon indicating HDR is superimposed and displayed. This is to show that the composition type information 302 is the image data obtained by "HDR composition".

Image7 is superposed with "additive" indicating additive synthesis or an icon indicating additive synthesis. This indicates that the composition type information 302 is "additive synthesis", that is, image data obtained by additive synthesis. Image3 remains a pin icon. This indicates image data in which the composition type information 302 is not described. Further, when the number of captured images used for compositing and the shooting conditions of the captured images used for compositing are recorded on the recording medium in association with the recorded images, at least one of them should be selectively displayed. You may be able to do it. In particular, when the type data is, for example, "3", the number of captured images used for compositing and the shooting conditions of the captured images used for compositing are recorded on the recording medium in association with the recorded images.

As described above, on the screen of FIG. 7C, characters, icons, and the like corresponding to the composition type information 302 are superimposed and displayed on the image data. As a result, the user can easily confirm which image data was combined by which composition type, the number of captured images, the shooting conditions of the captured images used for the composition, and the like. When the display switching button is clicked on the display screen of FIG. 7 (C), the screen changes to the screen of FIG. 7 (A). That is, each time the display switching button is clicked, the above three types of display screens are switched in order.

In the above embodiment, it is assumed that each time the display switching button 401 is operated, the display form loops in the order of FIG. 7 (A) → FIG. 7 (B) → FIG. 7 (C) → FIG. 7 (A). .. However, one of the display types may be specified directly from the display menu, and the display may be performed accordingly.
In addition, shooting conditions for captured images such as image identification information 301, composite type information 302, source image number information 303 which is the number of captured images used for composition, composite image information 304, and source image information of composite images. Etc. may be displayed at least one of them. By looking at them, the photographer's intentions and ingenuity can be grasped.
(Image processing during playback)

In addition, an operation when the image processing unit 107 performs image processing at the time of reproducing an image will be described. For example, it is assumed that the control unit 101 reads image data from the ROM 102 or the recording medium 108 and displays it on the display device of the display unit 109. In that state, in response to an image processing instruction from the user via the operation unit 111, the image processing unit 107 is made to input the image data corresponding to the image displayed on the display unit 109 and perform the image processing. Here, the control unit 101 rewrites the header information of the image data that is the target of the image processing according to the image processing performed by the image processing unit 107. Specifically, for the image data in which "3" is stored in the image identification information 301 of the image file, the image data to be processed such as gradation conversion processing, color conversion processing, defect correction processing, and filter processing. Consider the case where image processing performed based on the pixel value is performed.

In that case, the control unit 101 also stores "3" indicating the composite image acquired at the time of shooting in the image identification information 301 in the image file that stores the processed image data. On the other hand, the image data in which "3" is stored in the image identification information 301 of the image file is subjected to some kind of composition processing with other source images, text data, frames, icons, CG, and the like. Consider the case where image processing accompanied by compositing processing is performed. In that case, the control unit 101 stores "2" indicating a general composite image in the image identification information 301 in the image file that stores the processed image data. By operating the control unit 101 in this way, the reliability of the information of the composite image “3” acquired at the time of shooting can be ensured in the system. Further, the control unit 101 stores the processed image data for the image data in which "2" is stored in the image identification information 301 of the image file, regardless of which of the above image processing is performed. The image identification information 301 also stores "2" indicating a general composite image in the file.

Further, the image data in which "1" is stored in the image identification information 301 of the image file is subjected to image processing corresponding to the processing classified as "2" or "1" in each of the above shooting modes. Consider the case. In that case, the control unit 101 determines the image identification information 301 corresponding to the processed image data according to the above classification. That is, the control unit 101 determines whether or not the image processing performed on the reproduced image is composition and the method of composition processing. Then, based on the determination result, the image identification information 301 to be recorded in the image file that stores the processed image data is determined, and recorded in the composite information unit 2091 of the image file including the processed image data. When image processing performed based on the pixel value of the image data to be processed such as other gradation conversion processing, color conversion processing, defect correction processing, and filter processing is performed, the processed image data is stored. The image file also stores "1" indicating a non-composite image in the image identification information 301.

As described above, in the present embodiment, by recording the information stored in the composite information unit 2091 together with the image data, the user or the computer appropriately determines the image according to the presence or absence and type of the composite processing. You will be able to process it. For example, it is possible to collect image files including HDR-synthesized images, analyze the information recorded in the composite information unit 2091 by AI or the like, and learn various shooting conditions. Then, it can be used as reference information for learning the exposure difference from the scene and composition, and the exposure time and ISO of the material image can also be utilized for that purpose.

In addition, for starry sky trajectory photography, if the tag indicating the total exposure period including the exposure time of each captured image and the non-exposure period between the captures of each captured image attached to the composite image is taken for this amount of time. It becomes possible to infer that the trajectory will be about this level. As a result, it is not necessary to set the shooting time unnecessarily long.
Also, when using it for camera learning, composition failed (when the camera judges that it failed at the time of shooting, when the cloud side judges it as a failure, or when the user judges it as a failure and gives a bad rating etc.) Record the information in a file. Then, it can be used for the next shooting conditions.

For example, if the cause of the failure is considered to be camera shake, it can be regarded as a user who is prone to camera shake, and improvements can be made such as setting the exposure time automatically determined by the control unit 101 for the next and subsequent shootings to a shorter time. become.
In addition, the information in the composite information unit is referred to when performing image processing on the composite JPEG image used for determining the image processing parameters of the composite image with an application or the like.

In an application that holds NR parameters for each ISO sensitivity and applies the parameters according to the captured ISO sensitivity, the logic for determining the parameters can be changed in the case of a composite image.
For example, in the case of a non-composite image of ISO3200, the parameter is set to ISO3200, in the case of a composite image of ISO3200X2, the parameter is set to ISO2500, in the case of a composite image of ISO3200X4, the parameter is set to ISO1600, and so on. The parameters can be switched according to the ISO sensitivity of the source image.
Further, by embedding the composite information unit 2091 in the data of the shooting conditions in the APP1 of the Exif structure, high compatibility with the existing Exif-compliant device and software can be obtained.

(Other Examples)
The present invention is not limited to the above examples, and various modifications can be made based on the gist of the present invention, and these are not excluded from the scope of the present invention.
Further, a computer program that realizes a part or all of the control in this embodiment to realize the functions of the above-described embodiment may be supplied to an image processing device or the like via a network or various storage media. Then, a computer (or CPU, MPU, etc.) in the image processing device or the like may read and execute the program. In that case, the program and the storage medium storing the program constitute the present invention.

100 Imaging device 105 Imaging unit 110 Distance map acquisition unit

Claims (15)

A compositing means that synthesizes a plurality of captured images to generate a composite image,
A recording control means for recording the captured image or the composite image as a recorded image in the recording unit, and
It has a shooting mode or a determination means for determining a synthesis method performed by the synthesis means.
The recording control means is characterized in that a recorded image captured or synthesized by a shooting mode or a synthesis method determined by the determination means is associated with type data determined based on the determination result of the determination and recorded. Image processing device. The first aspect of claim 1, wherein the determination means determines a shooting mode corresponding to the plurality of shot images based on a preset shooting mode when the plurality of shot images are shot. Image processing device. When the type data generated by the type data generation means is a predetermined type data, at least the number of captured images used for compositing and the shooting conditions of the captured images used for compositing are recorded in association with the recorded images. The image processing apparatus according to claim 1 or 2. The image according to claim 1, wherein the type data generation means generates at least one of two types of type data according to the type of synthesis when the composition is performed by the synthesis means. Processing equipment. The type data generation means generates the first type data when the shooting conditions and the angle of view of the plurality of images used for compositing at the time of shooting are the same, and the shooting conditions for the plurality of images used for compositing. The image processing apparatus according to claim 4, wherein when one of the angles of view is different from that of the image processing device, the second type of data is generated. The method according to any one of claims 1 to 5, wherein the type data generating means generates a third type data when a plurality of captured images are not combined by the combining means. Image processing device. The image processing apparatus according to any one of claims 1 to 6, wherein the recording means records the type data in a header area of an image data file. The image processing apparatus according to claim 7, wherein the recording means records the type data in the APP1 region of an Exif standard image data file. The image processing apparatus according to any one of claims 1 to 8, further comprising an imaging means for generating the captured image. Further, any one of claims 1 to 9, further comprising a display means capable of selectively displaying at least one of the number of captured images used for compositing and the photographing conditions of the captured images used for compositing. The image processing apparatus according to the section. A reproduction means for reproducing a recorded image from a recording medium on which a captured image or a composite image is recorded, and
Type data acquisition indicating the classification related to the composition recorded in association with the recorded image, information indicating the composition type, the number of captured images used for the composition, and the imaging conditions of the captured image used for the composition. Means and
A display capable of selectively displaying at least one of the type data acquired by the type data acquisition means, information indicating the composition type, the number of captured images used for composition, and the imaging conditions of the captured image used for composition. An image processing device comprising means and means. A compositing step for compositing multiple captured images by multiple methods to generate a composite image, and
A recording step of recording the captured image or the composite image as a recorded image,
It has a type data generation step that generates type data according to the type of synthesis by the synthesis step.
The recording step records the type data generated by the type data generation step in association with the recorded image, and at least synthesizes the type data generated by the type data generation step when the type data is a predetermined type data. An image processing method characterized in that the number of captured images used in the above and the imaging conditions of the captured images used for compositing are recorded in association with the recorded images. A playback step of reproducing a recorded image from a recording medium on which a captured image or a composite image is recorded, and
Type data acquisition according to the type of composition recorded in association with the recorded image, information indicating the composition type, the number of captured images used for composition, and the imaging conditions of the captured image used for composition Steps and
A display that can selectively display at least one of the type data acquired by the type data acquisition step, information indicating the composition type, the number of captured images used for composition, and the photographing conditions of the captured image used for composition. An image processing method characterized by having steps and. A computer program for operating a computer as each means of the image processing apparatus according to any one of claims 1 to 11. A computer-readable storage medium that stores the computer program according to claim 14.

Images