JP5621524B2 - Image processing apparatus and method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus and method, and a program, and more particularly, to a technique capable of realizing image processing that can represent a more natural and realistic landscape, which is image processing that matches the purpose of a user's imaging action.

2. Description of the Related Art Conventionally, there is a field of display technology in which a display device displays an image for viewing, and a car navigation device displays an image for information confirmation such as a map. In the field of such display technology, effect display according to time change and weather change is performed (see Patent Documents 1 to 4).
On the other hand, there is a field of imaging technology in which a user captures a captured image using a camera. Also in the field of such imaging technology, with the advent of digital cameras, it is possible to display captured images, and more recently, it is also possible to display captured images that have been subjected to image processing.

Japanese Patent Laid-Open No. 05-199491 Japanese Patent Application Laid-Open No. 07-210123 Japanese Patent Laid-Open No. 08-16586 JP 2002-286461 A

However, in the field of imaging technology, in general, an imaging action in which a user takes an image using a digital camera is more important than an action in which the user confirms a captured image obtained as a result of the imaging action. It becomes. Here, the imaging action is an action accompanied by a user's subjective purpose, and is an active action of creating an image that achieves the purpose. For this reason, it is required that image data that can be said to be created by the user be subjected to image processing that matches the purpose of an active action called an imaging action.
On the other hand, in the field of display technology, an image to be displayed is often an image created by another person other than the user. The act of appreciating or confirming an image created by another person is the main act of the user. In other words, the field of display technology has been developed as a main act is a passive act in which a user views or confirms a unilaterally presented image rather than an active act such as an imaging act. It is.
Therefore, even if the technology in the field of display technology, for example, the technology described in Patent Documents 1 to 4 is applied to the field of imaging technology as it is, image processing that matches the purpose of the user's imaging action is performed on the data of the captured image. It is difficult.

For example, consider a case where a user captures a landscape image seen from a predetermined location as a captured image using a digital camera. However, it is assumed that the purpose of the user's imaging action is to obtain a landscape image that is visible from the predetermined location and is in a desired state.
Here, even if it is the same predetermined place, if the season is different, the state of the landscape will be different.
Then, in order to achieve the purpose of the imaging action, the user goes to a predetermined place, waits until the time when the landscape is assumed to be in a desired state, and captures an image of the landscape.
However, the captured image obtained in this case is not necessarily an image in which a desired landscape is reflected, that is, an image in which the purpose of the imaging action is achieved.

This is because the state of the landscape varies depending on various conditions such as weather, and the probability that these various conditions are the same across time and place is very low.
For example, when the purpose of the imaging action is to obtain an image of a snowed landscape, the user usually predicts the date and time when snow falls by weather forecast or the like, and captures an image of the landscape at the predicted date and time.
However, the weather at the predicted date and time is not necessarily snowfall. If the predicted date and time is clear, for example, the user cannot obtain an image of a snowed landscape, that is, an image in which the purpose of the imaging action is achieved.

In this way, for a user, an image that is in a desired state (snow-covered state, etc.) of a landscape simply by performing an imaging operation (such as pressing a release button) using a digital camera, that is, the user It is difficult to obtain an image in which the purpose of the imaging action is achieved.
Therefore, it is required that the digital camera performs image processing on the captured image data and converts the captured image data into image data that achieves the purpose of the imaging action.

However, simply applying the techniques described in Patent Documents 1 to 4 makes it difficult to meet such requirements.
That is, the techniques described in Patent Documents 1 to 4 are the first technique for automatically selecting and displaying a picture or image to be displayed in accordance with a change in time or a change in weather, or a display of color, brightness, or the like. This is a second technique in which the adjustment data is preset and the entire image is corrected according to the current season, date, weather, and the like.
In the former first technique, it is necessary to prepare in advance a pattern or image to be selected and displayed. However, it is not realistic to prepare a landscape image that can be seen from a predetermined place. In the latter second technique, not only the color of the landscape but also the color and brightness of the entire image are uniformly corrected, so that only correction that is unnatural and lacks reality is possible.

Conventionally, in 3D CG (Computer Graphics) and architectural landscape simulation software, the landscape of buildings viewed from various viewpoints, the surrounding landscape such as trees, and the shade due to seasonal changes, etc. There is something that can be simulated.
However, it is clear that the images obtained by these simulation software are virtual and artificial images compared to captured images (photo images) obtained as a result of imaging the real world, which is unnatural and realistic. It was lacking.
Furthermore, it is assumed that these simulation softwares perform simulation drawing based on things different from the real world, such as design data and a three-dimensional shape model. For this reason, it is very difficult to apply these simulation software as they are for processing a captured image obtained as a result of imaging a real world landscape or the like.

  In summary, there is a demand for processing a captured image so as to be a more natural and realistic image that matches the purpose of the user's imaging action. However, the conventional techniques including Patent Documents 1 to 4 cannot sufficiently meet the demand. For this reason, the realization of the technique which can fully respond to the said request is calculated | required.

  The present invention has been made in view of such a situation, and realizes image processing that matches the purpose of the user's imaging action and can express a more natural and realistic landscape state. With the goal.

An image processing apparatus according to an aspect of the present invention includes an original image acquisition unit that acquires an original image to be processed, and a subject on which a predetermined deposit in the original image acquired by the original image acquisition unit can accumulate and the object specifying means for specifying the part of the position and shape, so as to correspond to the position and shape of the object portion which is the predetermined deposit identified may be deposited by the object specifying means, the predetermined deposition on said original image characterized in that it comprises a synthesizing means for generating a composite image by combining by placing the object image representing the object.

  According to another aspect of the present invention, an image processing method and a program corresponding to the above-described image processing apparatus according to one aspect of the present invention are provided.

  ADVANTAGE OF THE INVENTION According to this invention, it is an image process corresponding to the objective of a user's imaging action, Comprising: The image process which can express the state of a more natural and realistic landscape can be implement | achieved.

1 is a block diagram illustrating a hardware configuration of an imaging apparatus as a first embodiment of an image processing apparatus of the present invention. It is a functional block diagram which shows the functional structural example which implement | achieves the execution function of the image synthesis process which concerns on 1st Embodiment of this invention among the functional structures of the imaging device of FIG. It is a figure which shows an example of an original image. It is a figure which shows an example of the synthesized image obtained from the original image of FIG. 3 is a flowchart illustrating an example of a flow of image composition processing according to the first embodiment of the present invention executed by the imaging apparatus of FIG. 1. 6 is a flowchart for explaining an example of a detailed flow of a target subject specifying process included in the image composition process of FIG. 5. FIG. 7 is a flowchart for explaining an example of a detailed flow of a target subject specifying process included in the image composition process of FIG. 5, which is different from the example of FIG. 6. 6 is a flowchart for explaining an example of a detailed flow of a snow image generation process included in the image composition process of FIG. 5. FIG. 9 is a flowchart illustrating an example of a detailed flow of a snow image generation process included in the image composition process of FIG. 5, and an example different from the example of FIG. 8. It is a functional block diagram which shows the functional structural example which implement | achieves the execution function of the image synthesis process which concerns on 2nd Embodiment of this invention among the functional structures of the imaging device of FIG. It is a flowchart which shows an example of the flow of the image compositing process which concerns on 2nd Embodiment of this invention which the imaging device of FIG. 1 performs. It is a flowchart for demonstrating an example of the detailed flow of the growth image generation process which concerns on 2nd Embodiment of this invention which the imaging device of FIG. 1 performs.

  Hereinafter, as an embodiment of an image processing apparatus of the present invention, a first embodiment and a second embodiment will be described individually in that order based on the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing a hardware configuration of an imaging apparatus 1 as a first embodiment of the image processing apparatus of the present invention. The imaging device 1 can be configured by a digital camera, for example.

  The imaging apparatus 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a timer unit 14, a bus 15, an input / output interface 16, and an imaging unit 17. An operation unit 18, a display unit 19, a storage unit 20, a communication unit 21, a GPS (Global Positioning System) unit 22, and a drive 23.

The CPU 11 executes various processes according to a program recorded in the ROM 12 or according to a program loaded from the storage unit 20 to the RAM 13.
The RAM 13 also appropriately stores data necessary for the CPU 11 to execute various processes.

  For example, in the present embodiment, programs for realizing the functions of the original image acquisition unit 41 to display control unit 45 shown in FIG. 2 to be described later are stored in the ROM 12 and the storage unit 20. Therefore, when the CPU 11 executes processing according to these programs, the functions of the original image acquisition unit 41 to display control unit 45 in FIG. 2 can be realized.

  The timekeeping unit 14 performs a timekeeping operation and notifies the CPU 11 of the current time and the like.

  The CPU 11, ROM 12, RAM 13, and timer unit 14 are connected to each other via a bus 15. An input / output interface 16 is also connected to the bus 15. An imaging unit 17, an operation unit 18, a display unit 19, a storage unit 20, a communication unit 21, a GPS unit 22, and a drive 23 are connected to the input / output interface 16.

  Although not shown, the imaging unit 17 includes an optical lens unit and an image sensor.

The optical lens unit is configured with a lens that collects light, such as a focus lens and a zoom lens, in order to capture an image of the subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

The image sensor includes a photoelectric conversion element, an AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) a subject image at regular time intervals to accumulate image signals, and sequentially supplies the accumulated image signals to the AFE as analog signals.
The AFE executes various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. A digital signal is generated by various signal processing and output as an output signal of the imaging unit 17.
Hereinafter, the output signal of the imaging unit 17 is referred to as “captured image data”. Therefore, captured image data is output from the imaging unit 17 and is appropriately supplied to the CPU 11 and the like.

The operation unit 18 includes various buttons and the like, and accepts user instruction operations.
The display unit 19 is composed of a liquid crystal display or the like and displays various images.
The storage unit 20 is configured by a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores captured image data output from the imaging unit 17. The storage unit 20 also stores various data necessary for various image processing, for example, image data, various flag values, threshold values, and the like.
The communication unit 21 controls communication with other devices (not shown) via a network including the Internet.

  The GPS unit 22 measures the current position of the imaging device 1 by calculating the distance to a plurality of GPS satellites.

  A removable medium 31 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 23. The program read from the removable medium 31 by the drive 23 is installed in the storage unit 20 as necessary. The removable medium 31 can also store various data such as image data stored in the storage unit 20 in the same manner as the storage unit 20.

FIG. 2 is a functional block diagram illustrating an example of a functional configuration that realizes an execution function of the image composition processing according to the first embodiment of the present invention, among the functional configurations of the imaging apparatus 1 in FIG. 1.
Here, the image composition processing according to the first embodiment includes original image data representing a general landscape, and an image in which a certain amount of snow is applied to a target subject described later (hereinafter referred to as a “snow image”). A series of processing until combining the data.

  The CPU 11 includes an original image acquisition unit 41, a target subject specifying unit 42, a snow image generation unit 43, a synthesis unit 44, and a display control unit 45 in order to execute such an image synthesis process. .

In the present embodiment, each of the original image acquisition unit 41 to the display control unit 45 is configured as a combination of hardware called the CPU 11 and a program (software) stored in the ROM 12 or the like in the configuration shown in FIG. Yes.
However, this is merely an example, and it is naturally possible to transfer at least part of the functions of the original image acquisition unit 41 to the display control unit 45 to other components than the CPU 11.

In the present embodiment, a snow information storage unit 51 is provided as an area of the storage unit 20.
Note that the snow information storage unit 51 is provided as an area of the storage unit 20 is merely an example. For example, the snow information storage unit 51 may be provided as an area of the removable media 31. Further, the snow information storage unit 51 is not particularly required to be provided in the imaging device 1, and may be provided in another device connected via the communication unit 21, for example.

  In the present embodiment, the original image acquisition unit 41 acquires captured image data output from the imaging unit 17 as original image data, and supplies the acquired data to the target subject specifying unit 42 and the synthesis unit 44.

In addition, in this embodiment, the data of the captured image output from the imaging unit 17 is adopted as the data of the original image. However, the data is not particularly limited to the example of the present embodiment, and the snow target area described later can be specified. If it is image data, the supply source is not ask | required.
For example, the original image acquisition unit 41 can also acquire image data read from the removable medium 31 or image data supplied from the Internet or the like via the communication unit 21 as original image data.

In the present embodiment, the original image data includes position information indicating a region where the original image was captured (hereinafter referred to as “imaging region”) and a time when the original image was captured (hereinafter, “ It is assumed that time information indicating "imaging time" is added as at least meta information.
Here, the position information is obtained by, for example, the GPS unit 22 (FIG. 1), and the time information is obtained by, for example, the time measuring unit 14 (FIG. 1).

Based on the original image data supplied from the original image acquisition unit 41, the target subject specifying unit 42 includes one or more subjects (hereinafter referred to as “ A process for specifying the position and shape of the “subject” is executed.
Such processing for specifying the position and shape of the target subject is hereinafter referred to as “target subject specification processing”.
Here, the method for specifying the position and shape of the target subject is not particularly limited. In the present specification, two specific examples of such various methods will be described later with reference to FIGS. 6 and 7.

FIG. 3 shows an example of the original image.
In the original image 61 of FIG. 3, a target subject 71a indicating a road, a target subject 71b indicating a house, and the like are specified. Of course, other subjects such as homes may also be identified as subject subjects because of the possibility of snow accumulation, but here, for convenience of explanation, subject subjects 71a and 71b whose reference numerals are shown in FIG. Hereinafter, the description will be given focusing only on the above.

  The target subject specifying unit 42 notifies the snow image generating unit 43 of the position and shape of each of the one or more target subjects.

The snow image generation unit 43 includes the one or more target subjects on which a certain amount of snow has been accumulated based on the positions and shapes of the one or more target subjects notified from the target subject specifying unit 42, and the other portions are transparent. The generated image data is generated as snow image data.
Here, a portion above the target subject and where a certain amount of snow is accumulated is hereinafter referred to as a “snow accumulation region”. Therefore, in the present embodiment, the snow image includes both the target subject and the snow accumulation area. However, it is sufficient if the snow image includes a snow accumulation region, and the target subject may be omitted.
In the present embodiment, in the snow image, pixels other than the target subject and the snow accumulation region, that is, transparent pixels, are not given a predetermined pixel value in the data and are flagged as “no pixel value”. To be expressed.
Here, the method for generating the snow image is not particularly limited. In this specification, two specific examples of such various methods will be described later with reference to FIGS. 8 and 9.

In the present embodiment, the snow amount of the snow region, that is, the reference for the vertical length (height) in the image of the snow region can be freely set by the user. Called “amount”. That is, the user can instruct the set snow amount by operating the operation unit 18.
Although the method for instructing the set snow amount is not particularly limited, in the present embodiment, two types of setting modes, a snow amount adjustment mode and a seasonal change mode, are provided, and the user can select any type of setting mode. It is assumed that the method of instructing the set snow amount using is used.
The snow accumulation mode is a mode in which the user directly designates the adjustment of the set snow amount by operating the operation unit 18 in units of centimeters, for example. In the snow cover amount adjustment mode, the user can specify a minus amount to reduce the set snow cover amount for a composite image that has already snowed (the composite image will be described later). Can also be reduced.
On the other hand, in the seasonal change mode, a list of regions and seasons is displayed on the display unit 19, and the user operates the operation unit 18 to select any region and any season from the list. A mode in which the amount of snowfall in each year in the region and season is indicated as the set amount of snowfall. In order to realize the seasonal change mode, information in which a predetermined amount of snow is associated with a pattern specified by a plurality of types of parameters including at least a region and a season (hereinafter referred to as “snow cover information”). ) Is recorded in the snow cover information storage unit 51.

  The snow image generation unit 43 supplies the snow image data generated in this way to the synthesis unit 44.

The combining unit 44 generates combined image data by combining the original image data supplied from the original image acquiring unit 41 and the snow image data generated by the snow image generating unit 43.
Specifically, for example, the snow image is superimposed on the original image in a state where the transparency is set to 0% (non-transparent) for the target subject and the snow covered region, and the transparency is set to 100% (transparent) for the other subjects. As described above, composite image data is generated.

FIG. 4 shows an example of a composite image obtained from the original image of FIG.
The composite image 62 is an image in which a snow cover area is added on each target subject, for example, a snow cover area 72a is added on the target subject 71a, and the target image 71b is displayed on the target image 71b. This is an image with the snow region 72b added.

Here, the transparency of the target subject and the snowy area need not be 0% (non-transparent), and may be translucent, for example, 30%. Thereby, since the color of the original image can be reflected in the composite image, for example, a more natural expression can be realized so that it can be understood that the original landscape exists behind the snowy area.
Further, the transparency of the snowy area does not need to be uniformly the same ratio, and approaches 0% (non-transparent) as it goes downward, in other words, approaches 100% (transparent) as it goes upward. Good. As a result, the density of snowfall increases as it goes down, making it harder to see the original landscape, and the density of snowfall decreases as it goes up, making it easier to see the original landscape. Is also possible.

The combining unit 44 supplies the composite image data to the display control unit 45.
Then, the display control unit 45 causes the display unit 19 to display a composite image represented by the data supplied from the synthesis unit 44, that is, an image in which the target subject has been snowed with respect to the original image.

  Further, when there is a save designation operation on the operation unit 18 by the user, the synthesis unit 44 saves the data of the synthesized image in, for example, the removable medium 31.

  Next, an image composition process executed by the imaging apparatus 1 having the functional configuration shown in FIG. 2 described above will be described.

For example, after the captured image is captured, the image composition processing is a confirmation screen (so-called preview screen) before confirming that the captured image data is recorded in the storage memory (removable medium 31 or the like in FIG. 1). When the user performs an instruction operation on the operation unit 18 to change the captured image to an image in a snowy state during the display, the operation is started.
Then, after the captured image after the state of snow has been changed by this image composition processing is displayed again on the confirmation screen and the user performs an instruction operation to confirm recording in the storage memory, the imaging device 1 Data of the captured image after the snow condition is changed is recorded in the storage memory. Alternatively, the imaging device 1 determines the permanent storage of the captured image data temporarily stored in the storage memory.
In this case, when the user does not perform an instruction operation to confirm recording in the storage memory, the captured image data recorded in the storage memory or the storage unit 20 is automatically deleted. After the storage or non-storage of the captured image data is confirmed, the state of the imaging device 1 shifts to a standby state for the next shooting.
The user performs the change instruction operation as many times as necessary before performing the instruction operation for confirming the recording in the storage memory (before shifting to the standby state for the next shooting). By executing the image composition process, it is possible to confirm the change in the snow cover state each time on the confirmation screen.
On the other hand, when the user performs an instruction operation to confirm recording in the storage memory without performing an instruction operation to change the snow cover state, the snow composition state is changed without executing the image composition process. The data of the captured image not recorded is recorded in the storage memory.
Note that a recorded image recording instruction operation (shutter button full press operation or the like) can also be used as an instruction operation for changing to an image in a snowy state.
In this way, on the confirmation screen immediately after shooting, the snow cover state is changed to an arbitrary state in accordance with the user's instruction operation, and the change result can be checked, so that the combination of composition and snow cover state is desired. Until the condition is reached, it is possible to re-shoot as many times as possible while changing both the composition and the snow cover. In addition, since only the captured image data in which the combination of the composition and the snow cover state is in a desired state can be recorded in the storage memory, it is possible to save the memory capacity without storing unnecessary captured image data. It becomes possible.
That is, if the user changes only the snow cover state, the user can edit the image using a personal computer at home after returning home, for example. However, it is essential for the user to change and check the snow cover state on the spot (immediately after photographing) in order to obtain a desired combination of composition and snow cover state. In order to make such a change and confirmation, an image composition process is executed.
As described above, it is desirable to perform image composition processing before confirming recording in the storage memory immediately after shooting (before shifting to the next shooting standby state), but after recording in the storage memory. It is also possible to read the captured image at an arbitrary timing and execute the image composition processing.
Further, for example, a CPU with higher processing capability may be used to display the image data sequentially obtained for live view display while performing image synthesis processing in real time.
In addition, the snow accumulation mode / seasonal change mode and season change mode are set and stored in advance before shooting, and when the shooting instruction is issued, the user does not perform any other instruction operation. An image synthesis process may be automatically executed under conditions to automatically record a captured image after the snow cover state is changed.

  FIG. 5 is a flowchart showing an example of the flow of such image composition processing.

  In step S1, the original image acquisition unit 41 acquires original image data.

In step S2, the target subject specifying unit 42 specifies the shape and position of each of the one or more target subjects from the original image data acquired in step S1 by executing the target subject specifying process.
Details of the target subject specifying process will be described later with reference to specific examples of FIGS. 6 and 7.

In step S <b> 3, the snow image generation unit 43 accepts a setting operation for setting the set snow amount setting mode.
That is, as described above, there are two types of setting modes, the snow accumulation mode and the seasonal change mode, as the set snow accumulation mode. For this reason, in this embodiment, the user can select a desired mode among these two types of setting modes by operating the operation unit 18.
When the snow image generation unit 43 receives such a selection operation on the operation unit 18 by the user, the snow image generation unit 43 advances the process to step S4.

  In step S4, the snow image generating unit 43 determines whether or not the selected setting mode is the seasonal change mode by interpreting the content of the selection operation received in the process of step S3.

When the snow accumulation mode is selected as the set snow accumulation setting mode, it is determined as NO in step S4, and the process proceeds to step S5.
In step S <b> 5, the snow image generation unit 43 determines the snow amount (including the increase / decrease amount) directly designated by the user by operating the operation unit 18 as the set snow amount.
Thereby, a process progresses to step S9. However, the processing after step S9 will be described later.

  On the other hand, when the seasonal change mode is selected as the set snow amount setting mode, it is determined as YES in step S4, and the following processes in steps S6 to S8 are executed.

  In step S <b> 6, the snow image generation unit 43 controls the display control unit 45 (not shown in FIG. 2) to display a list of regions and seasons that can be selected by the user on the display unit 19.

  The user who sees the list displayed on the display unit 19 operates the operation unit 18 to select a desired region and a desired season.

In step S <b> 7, the snow image generation unit 43 recognizes the region and season patterns selected by the user.
When the user does not select an area, in the present embodiment, the snow image generation unit 43 regards that the imaging area of the original image is selected, and meta information (position) attached to the data of the original image The imaging area is recognized from the information.
In addition, when the user does not select a season, in the present embodiment, the snow image generation unit 43 regards that the season when the original image was captured is selected, and is attached to the data of the original image. Recognize the season from meta information (time information).

  In step S8, the snow image generation unit 43 reads the snow cover information corresponding to the pattern recognized in the process of step S7 from the snow cover information storage unit 51, and determines a set snow cover amount based on the snow cover information.

  In this way, when the set snow amount is determined in the process of step S5 or S8, the process proceeds to step S9.

In step S9, the snow image generation unit 43 generates snow image data. That is, a snow cover area indicating snow accumulated based on the set snow amount determined in the process of step S5 or S8 is added to the upper part of the target object whose position and shape have been specified by the target subject specifying process of step S2. The image data is generated as snow image data.
Such processing in step S9 is hereinafter referred to as “snow image generation processing” in accordance with the description of the step in FIG. Details of the snow image generation processing will be described later with reference to specific examples of FIGS. 8 and 9.

  In step S10, the synthesis unit 44 synthesizes the composite image data by combining the original image data acquired in step S1 with the snow image data generated in step S9. Generate.

In step S11, the display control unit 45 causes the display unit 19 to display a composite image expressed by the data generated in the process of step S10.
Thereby, the image composition process is completed.

Next, details of the target subject specifying process in step S2 in the image compositing process of FIG. 5 will be described using the specific examples of FIGS.
FIG. 6 is a flowchart for explaining an example of a detailed flow of the target subject specifying process.

  In the target subject specifying process of the example of FIG. 6, the target subject specifying unit 42 specifies the position and shape of the target subject by creating a saliency map.

The saliency map refers to the following image data. In other words, the receptive field cells in the retina of the human eye and the visual cortex of the brain have the property of recognizing the contour component and the straight line component of the image reflected on the retina. The saliency map is image data generated from the original image data by the simulation based on such properties and used to extract continuous contour components from the original image. .
The resolution (size) of the saliency map is the same as that of the original image, and each pixel value of the saliency map is the sum of the pixel values of a plurality of types of feature amount maps or values weighted thereto.
Therefore, in the target subject specifying process of the example of FIG. 6, first, a plurality of types of feature amount maps are generated by repeating the following loop process of steps S21 to S23.

In step S21, the target subject specifying unit 42 sets a predetermined type of feature as a processing target.
In step S <b> 22, the target subject specifying unit 42 generates a type of feature amount map corresponding to the processing target.

Here, the feature amount map is data of an image having the same resolution (size) as the original image, and the feature amount in each pixel of the original image with respect to a predetermined type of feature is represented by a corresponding pixel. This is image data having values.
That is, the feature map type is specified in accordance with the feature type selected in step S21.
What is mainly adopted as the types of features that can be selected in the process of step S21 is a luminance or color difference (that is, a contour component) between adjacent pixels and a linear component in the vertical and horizontal diagonal directions. All of these types of features were adopted based on the nature of human receptive field cells that specifically senses luminance and color differences and vertical and horizontal diagonal lines.
Emphasizing luminance and color contour components is equivalent to enhancing high-frequency components of the spatial frequency of an image. Therefore, when the luminance or color contour component is selected as a feature in the process of step S21, the target subject specifying unit 42 uses a spatial frequency low-pass filter (such as a Gaussian filter) as the process of step S22. The difference between the data obtained as a result of multiplying the data and the original image data is calculated. As a result, image data in which a predetermined high-frequency component is emphasized, that is, a feature amount map for luminance and color contour components is generated.
In addition, when a linear component in a predetermined direction in the vertical and horizontal diagonal directions is selected as a feature in the process of step S21, the target subject specifying unit 42 performs a filter (step S22) that emphasizes the contour component in the predetermined direction. The difference between the data obtained by applying the Gabor filter or the like to the original image data and the original image data is calculated. Thereby, image data in which a linear component in a predetermined direction is emphasized, that is, a feature amount map for the linear component in the predetermined direction is generated.

In this manner, when the type of feature amount map corresponding to the processing target is generated in the processing of steps S21 and S22, the processing proceeds to step S23.
In step S23, the target subject specifying unit 42 determines whether or not all types of features have been set as processing targets.
Here, any of “all types of features” can be arbitrarily determined by the designer or user of the imaging apparatus 1. That is, the design manufacturer or user of the imaging apparatus 1 can determine an arbitrary number of features of any type.
In this manner, when there is a plurality of feature types determined by the design producer or user of the imaging device 1 that have not yet been set as the processing target, it is determined as NO in step S23, and the processing is performed. Is returned to step S21, and the subsequent processing is repeated.
That is, each of a plurality of feature types determined by the design producer or user of the imaging device 1 is sequentially set as a processing target, and the loop processing of steps S21 to S23 is repeatedly executed each time, thereby performing the processing. A type of feature amount map corresponding to the object is generated.
When a plurality of types of feature quantity maps such as a feature quantity map for high-frequency components of luminance and color and a feature quantity map for longitudinal and horizontal diagonal line components are generated, it is determined as YES in step S23, and processing is performed. Advances to step S24.

  In step S <b> 24, the target subject specifying unit 42 assigns a certain weight to each of the pixel values constituting the data of the plurality of types of feature amount maps as necessary, and takes the sum of the weights to obtain the saliency. Generate a map.

As a specific method for generating a saliency map, for example, Treisman's feature integration theory or a saliency map method by Itti and Koch et al. Can be adopted.
As for the feature integration theory of Treisman, “AMTreisman And G. Gelade,“ A Feature-Integration Theory of Attention ”, Cognitive Psychology, Vol. 12, No. 1, pp97-136, 1980” may be referred to.
For the saliency map by Itti and Koch et al., See “L. Itti, C. Koch, and E. Niebur,“ A Model of Saliency-Based Visual Attention for Rapid Scene Analysis ”, IEEE Transactions on Pattern Analysis and Machine Intelligence. , Vol. 20, No. 11, November 1998 ”.

  In step S25, the target subject specifying unit 42 specifies the point-of-interest area based on the saliency map. That is, the target subject specifying unit 42 specifies an aggregate (region) of pixels having a pixel value equal to or greater than a predetermined threshold among the pixel values constituting the saliency map as the attention point region.

In step S26, the target subject specifying unit 42 specifies the position and shape of the target subject based on the attention point area. For example, the target subject specifying unit 42 specifies the position and shape of the target subject by combining outlines above the attention point region.
Thereby, the target subject specifying process ends. That is, the process of step S2 in FIG. 5 ends, and the process proceeds to step S3.

The example of the target subject specifying process in step S2 of the image composition process in FIG. 5 has been described above with reference to the flowchart in FIG. Next, another example of the target subject specifying process will be described with reference to the flowchart of FIG.
FIG. 7 is an example of a detailed flow of the target subject specifying process, and is a flowchart for explaining an example different from the example of FIG.

  In step S41, the target subject specifying unit 42 specifies the three-dimensional shape of the target subject from the data of the two-dimensional original image.

  In step S42, the target subject specifying unit 42 specifies a plane portion in the specified three-dimensional shape.

In step S43, the target subject specifying unit 42 specifies the position and shape of the target subject based on the specified plane portion.
Thereby, the target subject specifying process ends. That is, the process of step S2 in FIG. 5 ends, and the process proceeds to step S3.

Heretofore, two examples of the target subject specifying process in step S2 of the image composition process in FIG. 5 have been described with reference to the respective flowcharts in FIGS.
Next, two examples of the snow image generation process in step S9 of the image composition process will be described with reference to the flowcharts of FIGS.

  FIG. 8 is a flowchart for explaining an example of a detailed flow of the snow image generation process.

In step S61, the snow image generation unit 43 executes a process of randomly arranging unit snow images on the reference line (hereinafter referred to as “unit snow image arrangement process” in accordance with the description of FIG. 8).
Here, the reference line of the first unit snow image arrangement process is a portion (mainly at the top) where there is a possibility of snow accumulation in the target subject whose position and shape are specified by the target subject specifying process in step S2 of FIG. The outline of the shape is adopted.
The unit snow image is a snow color image of n × m pixels. n is an arbitrary integer value indicating the number of pixels in the horizontal direction. m is an integer value indicating the number of pixels in the vertical direction, and is an arbitrary integer value (including n = m) independent of n. n and m may be set within a range that looks like natural snow when unit snow images are continuously arranged.
If all the unit snow images are uniformly the same color, the portions where the unit snow images are continuously arranged are uniformly the same color and do not look like natural snow. Therefore, the color of the unit snow image may be changed randomly within a range that does not become unnatural as the color at the time of snow accumulation. Alternatively, the color of each n × m pixel constituting the unit snow image may be randomly changed within a range that does not become unnatural as the color at the time of snow accumulation.
When there are a plurality of target subjects, the unit snow image arrangement process is executed for each of the plurality of target subjects.

  In step S62, the snow image generation unit 43 updates the contour line of the shape of the upper part of the unit snow image at the time when the processing of the previous step S61 is completed as a new reference line.

  In step S63, the snow image generation unit 43 recognizes the amount of snow in the image based on the height from the initial reference line (above the target subject) to the reference line updated in the immediately preceding step S62. Then, it is determined whether or not the snow cover amount exceeds the set snow cover amount determined in the process of step S5 or S8 in FIG.

If the amount of snow in the image has not reached the set amount of snow, it is determined NO in step S63, the process returns to step S61, and the subsequent processes are repeated.
That is, until the amount of snow in the image reaches the set amount of snow, the loop processing of steps S61 to S63 is repeatedly executed, and the unit snow image arrangement processing is executed each time.
As a result, the unit snow images are gradually stacked on top of the target subject. In this case, a region where such unit snow images are stacked is a snow accumulation region. That is, by repeatedly executing the unit snow image arrangement process, the height of the snow area in the image increases so that the amount of snow in the image increases.
Thereafter, when the amount of snow in the image exceeds the set amount of snow, it is determined as YES in step S63, and the snow image generation process ends.
When there are a plurality of target subjects, the processes of steps S61 to S63 are performed for each of the plurality of target subjects, and a plurality of snow areas are formed above the plurality of target subjects. In this case, if, for example, the maximum amount (highest) out of the amount of snow (height) in the image of the snow region exceeds the set amount of snow, it is determined to be YES in step S63, and the snow image generation process ends. .

When the snow image generation process is completed in this way, one or more objects having the same resolution (same size) as the original image, each having a fixed amount (fixed height) of snow accumulation area added to the upper part, respectively. Image data including a subject and transparent otherwise is generated as snow image data.
Thereafter, the process proceeds from step S9 to step S10 in FIG. 5, and such snow image data is combined with the original image data to generate combined image data.

The example of the snow image generation process in step S9 of the image composition process in FIG. 5 has been described above with reference to the flowchart in FIG. Next, another example of the snow image generation process will be described with reference to the flowchart of FIG.
FIG. 9 is an example of a detailed flow of the snow image generation process, and is a flowchart for explaining an example different from the example of FIG.

  In step S81, the snow image generation unit 43 sets the shape of the snow target model in a predetermined snow simulation in accordance with the position and shape of the target subject specified by the target subject specifying process in step S2 of FIG.

  In step S82, the snow image generating unit 43 sets various conditions for the snowfall simulation, such as setting the set snow amount determined in the process of step S5 or S8 in FIG. 5 as the amount of snow in the snowfall simulation.

In step S83, the snow image generation unit 43 generates snow image data by performing a snowfall simulation on the snow target model set in step S81 in accordance with the conditions set in step S82. .
Thereby, the snow image generation process ends.

When the snow image generation process is completed in this way, one or more objects having the same resolution (same size) as the original image, each having a fixed amount (fixed height) of snow accumulation area added to the upper part, respectively. Image data including a subject and transparent otherwise is generated as snow image data.
Thereafter, the process proceeds from step S9 to step S10 in FIG. 5, and such snow image data is combined with the original image data to generate combined image data.

  Note that the snowfall simulation technique applied to the snow image generation process in the example of FIG. 9 is not particularly limited. For example, a technique using a filter called a Monte Carlo filter or a particle filter can be applied. Therefore, the outline of the method will be described below.

The snowfall simulation to which the method using the particle filter is applied is executed as follows.
That is, when snow particles randomly generated from a predetermined position in the sky are given random fluctuations at regular intervals, the motion state of each snow particle is calculated. When such a calculation is repeatedly executed for each target subject until each snow particle reaches, the number of snow particles reaching each target subject is obtained.
When the calculation until the number of snow particles is obtained is repeatedly executed, a three-dimensional shape of snow (three-dimensional shape of the snow region) that accumulates on each of the target subjects is formed.
As a result, the snow amount of each of the target subjects is naturally random, but when the average value (or the maximum value) reaches the set snow amount determined in step S5 or S8 of FIG. The three-dimensional shape of the region is determined.

Further, the snow image generation unit 43 simulates how the three-dimensional shape of the snow area determined as described above looks with respect to the angle of view of the original image, thereby generating a two-dimensional snow area. Generate snow image data.
Here, the method of obtaining a two-dimensional image from the three-dimensional shape of the snow region is not particularly limited, and for example, a ray tracing method widely used as a CG (computer graphics) method is employed. be able to.
The ray tracing method simulates how each pixel of a two-dimensional screen looks by tracing the light propagation path, assuming that light is emitted from the eye viewing the image. A method of generating two-dimensional image data.

As described above, the imaging apparatus 1 according to the first embodiment of the present invention includes the original image acquisition unit 41, the target subject specifying unit 42, the operation unit 18, the snow image generation unit 43, and the synthesis unit 44. It is equipped with.
The original image acquisition unit 41 acquires data of an original image including a subject.
The target subject specifying unit 42 specifies the shape of the target subject with respect to the original image data, using a subject that can be synthesized so that at least a part of snow as an example of a predetermined object touches or overlaps.
The operation unit 18 receives the set snow accumulation amount as a reference for the amount of snow synthesis specified by the user.
The snow image generation unit 43 converts the image data arranged at a position where the snow of the snow amount based on the set snow amount (the above-described snow region) touches or overlaps at least a part of the shape of the target subject. Generated as image data.
The combining unit 44 generates combined image data by combining the original image data and the snow image data.

  Therefore, the imaging apparatus 1 according to the first embodiment of the present invention can generate composite image data as if snow was piled up on the captured image. Such a composite image is, for example, an image simulating snow accumulation in different regions and seasons.

Here, a case where the user captures an image of a landscape seen from a predetermined place (for example, A-ku, Tokyo, Japan) as the captured image using the imaging device 1 is considered. However, it is assumed that the purpose of the user's imaging action is to obtain an image of a landscape at the time of snowing that can be seen from the predetermined place (for example, Tokyo A Ward, Japan).
In this case, even if it is not actually snowing at the time of shooting by the user, that is, even if an image of a landscape that has not snowed is obtained as an original image, an image of a landscape in which a desired amount of snow has accumulated as a composite image. It can be easily obtained.
Furthermore, since a certain amount of snow is not uniformly accumulated, the set snow amount specified by the user is used as a reference, and the snow amount is determined for each subject based on the reference, so that more natural and realistic snow can be obtained. Image processing that can express the state of a certain snowfall can be realized.

The above-described effects that can be achieved by the imaging apparatus 1 according to the first embodiment of the present invention will be specifically described as follows (1) and (2).
(1) Since an image of a desired landscape that matches the user's intention to draw, such as a landscape with a desired amount of snow in a predetermined place, can be easily obtained as a simulated composite image, for the user, It can reduce failure and make it easier to cover when it fails.
(2) There are shootings that are difficult to re-shoot, such as shooting in places where it is difficult to re-shoot such as overseas travel destinations, and shooting of subjects that are difficult to reproduce or repeat the same state. Even in such a case, the user can obtain a composite image close to a desired landscape by using a captured image obtained by one shooting as an original image, and further, data of the composite image. Can be stored data. As a result, re-shooting is unnecessary.
Unlike the case of adopting image editing software used exclusively for personal computers, it is not necessary for the user to individually specify the location of snow, the amount of snow, and the state of snow (color, etc.). By simply designating the season, it is possible to automatically determine the location of snow and the state (color, etc.) of snow and easily synthesize snow images.

Heretofore, the imaging apparatus 1 as the first embodiment of the image processing apparatus of the present invention has been described.
Here, in the first embodiment, the predetermined object to be combined with the original image is snow, but is not particularly limited to this.
As the predetermined object to be combined with the original image, for example, any deposit that accumulates on the subject over time can be employed. In other words, an example of the deposit is the above-described snow. Accordingly, deposits other than snow, such as puddles and dust, can be employed as the predetermined object to be combined with the original image.
Furthermore, as the predetermined object to be combined with the original image, for example, a growing thing that grows with the passage of time, such as a flower blooming in a tree or a flower growing in a garden, can be adopted.
Therefore, hereinafter, an embodiment in which a “growth” is adopted as a predetermined object to be combined with the original image will be described as a second embodiment.

[Second Embodiment]
The imaging apparatus 1 according to the second embodiment of the present invention can have basically the same hardware configuration as the imaging apparatus 1 according to the first embodiment.
Therefore, FIG. 1 is also a block diagram illustrating a hardware configuration of the imaging apparatus 1 according to the second embodiment. The imaging device 1 according to the second embodiment can be configured by a digital camera, for example, as in the first embodiment.

FIG. 10 is a functional block diagram illustrating an example of a functional configuration that implements an execution function of an image composition process according to the second embodiment, among the functional configurations of the imaging apparatus 1 of FIG.
Here, the image composition processing according to the second embodiment refers to an original image data representing a general landscape and an image including a growth that has achieved a certain amount of growth (hereinafter referred to as a “growth image”). A series of processing until combining the data.

  The CPU 11 includes an original image acquisition unit 41, a target subject identification unit 42, a synthesis unit 44, a display control unit 45, and a growth image generation unit 71 in order to execute such an image synthesis process. Yes.

In the present embodiment, each of the original image acquisition unit 41, the target subject specifying unit 42, the synthesis unit 44, the display control unit 45, and the growth image generation unit 71 includes a hardware called a CPU 11 in the configuration illustrated in FIG. 1. The program is configured as a combination with a program (software) stored in the ROM 12 or the like.
However, this is merely an example, and at least some of the functions of the original image acquisition unit 41, the target subject specifying unit 42, the synthesis unit 44, the display control unit 45, and the growth image generation unit 71 are other than the CPU 11. Of course, it is also possible to transfer to a component.

In the present embodiment, a growth information storage unit 81 is provided as an area of the storage unit 20.
Note that the growth information storage unit 81 is provided as an area of the storage unit 20 is merely an example, and may be provided as an area of the removable medium 31, for example. The growth information storage unit 81 is not particularly required to be provided in the imaging device 1, and may be provided in another device connected via the communication unit 21, for example.

  The functional blocks of the original image acquisition unit 41, the target subject specifying unit 42, the synthesis unit 44, and the display control unit 45 have basically the same functional configuration as the functional blocks with the same reference numerals in FIG. Therefore, these descriptions are omitted here as appropriate.

Based on the original image data acquired by the original image acquisition unit 41, the target subject specifying unit 42 selects one or more subjects that contain or may come into contact with growth, for example, a tree with flowers such as cherry blossoms. As the target subject, the position and shape of the target subject are specified.
The target subject specifying unit 42 notifies the growth image generating unit 71 of the position and shape of each of the one or more target subjects.

Based on the position and shape of the one or more target subjects notified from the target subject specifying unit 42, the growth image generation unit 71 adds the one or more target subjects to which the growth material that has achieved a certain amount of growth has been added. Data of an image that is included and is otherwise transparent is generated as data of a growth image.
Here, in the present embodiment, the growth image includes both the target subject and the growth object added thereto. For example, if the target subject is a tree, the growth is a branch of the tree, and the amount of leaves that grow on the branch and the amount of flowers that bloom on the branch change depending on the degree of growth of the branch. However, it suffices if the growth image contains the growth material, and the target subject may be omitted.
In the present embodiment, in the growth image, pixels other than the target object and the growth material, that is, transparent pixels, are not given a predetermined pixel value in the data, and have a flag “no pixel value”. It shall be expressed by being stood.
Here, the method for generating the growth image is not particularly limited. In this specification, one specific example of such various methods will be described later with reference to FIG.

In the present embodiment, the reference of the growth degree of the grown product can be freely set by the user, and is hereinafter referred to as “set growth degree”. That is, the user can instruct the set growth degree by operating the operation unit 18.
Although the method of instructing the set growth degree is not particularly limited, in the present embodiment, two types of setting modes, a growth degree adjustment mode and a seasonal change mode, are provided, and the user uses any type of setting mode. It is assumed that the method of instructing the set growth degree is adopted.
The set degree adjustment mode is a mode in which the user directly indicates the adjustment amount of the set growth degree by operating the operation unit 18. In the growth degree addition / subtraction mode, the user restores growth by setting a minus amount to reduce the set growth degree with respect to an already grown composite image (the composite image will be described later) (past The composite image can be obtained.
On the other hand, in the seasonal change mode, a list of regions and seasons is displayed on the display unit 19, and the user operates the operation unit 18 to select any region and any season from the list. This is a mode in which the annual growth rate in the region and season is indicated as the set growth rate. In order to realize the seasonal change mode, information in which a predetermined growth degree is associated with a pattern specified by a plurality of types of parameters including at least a region and a season (hereinafter referred to as “growth information”). Is recorded in the growth information storage unit 81.

  The growth image generation unit 71 supplies the growth image data generated in this way to the synthesis unit 44.

The combining unit 44 generates combined image data by combining the original image data supplied from the original image acquiring unit 41 and the grown image data generated by the grown image generating unit 71.
Specifically, for example, the growth image is placed on the original image in a state where the transparency is set to 0% (non-transparent) for the target subject and the growth, and the transparency is set to 100% (transparency) for the other subjects. Composite image data is generated so as to be superimposed.

Here, the transparency of the target subject and the growth object need not be 0% (non-transparent), and may be semi-transparent, such as 30%. Thereby, since the color of the original image can be reflected in the composite image, for example, a more natural expression can be realized so that the appearance of the original scenery in the back of the growth can be understood in a relaxed manner.
Further, the transparency of the growth need not be uniformly the same ratio, and it becomes closer to 0% (non-transparent) as it goes downward, in other words, it approaches 100% (transparent) as it goes upward. Good. As a result, the density of the growth (flower blooms, leaves with thick leaves, etc.) increases as it goes down, and it becomes difficult to see the original landscape. More natural expressions, such as making it easier to see the original landscape, are also possible.

  Next, an image composition process according to the second embodiment executed by the imaging apparatus 1 having the functional configuration of FIG. 2 described above will be described.

The image compositing process according to the second embodiment is, for example, for confirmation before confirming that the captured image data is recorded in the storage memory (such as the removable medium 31 in FIG. 1) after the captured image is captured. When the user performs an instruction operation for changing the growth degree of the growth included in the captured image on the operation unit 18 while the screen (so-called preview screen) is displayed, the operation is started.
Then, the captured image after the growth degree is changed by this image composition processing is displayed again on the confirmation screen, and after the user performs an instruction operation to confirm recording in the storage memory, the imaging device 1 Data of the captured image after the degree is changed is recorded in the storage memory. Alternatively, the imaging device 1 determines the permanent storage of the captured image data temporarily stored in the storage memory.
In this case, when the user does not perform an instruction operation to confirm recording in the storage memory, the captured image data recorded in the storage memory or the storage unit 20 is automatically deleted. After the storage or non-storage of the captured image data is confirmed, the state of the imaging device 1 shifts to a standby state for the next shooting.
The user performs the change instruction operation as many times as necessary before performing the instruction operation for confirming the recording in the storage memory (before shifting to the standby state for the next shooting). By executing the image compositing process, it is possible to confirm the change in the growth degree of the grown product on the confirmation screen each time.
On the other hand, when the user performs an instruction operation to confirm recording in the storage memory without performing an instruction operation to change the growth degree, the growth degree of the growth is changed without executing the image composition process. Data of the captured image that has not been recorded is recorded in the storage memory.
It should be noted that a captured image recording instruction operation (shutter button full pressing operation, etc.) can also be used as an instruction operation for changing the growth degree of the growth.
In this way, on the confirmation screen immediately after shooting, the growth degree of the growth is changed to an arbitrary degree according to the user's instruction operation, and the change result can be confirmed, so the combination of composition and growth degree is It is possible to re-shoot as many times as necessary while changing both the composition and the degree of growth until the desired state is achieved. In addition, since only the captured image data in which the combination of composition and the degree of growth is in a desired state can be recorded in the storage memory, it is possible to save memory capacity without storing useless captured image data. It becomes possible.
That is, if the user changes only the growth degree of the grown product, for example, the user can edit the image using a personal computer at home after returning home. However, it is essential for the user to change and confirm the growth degree on the spot (immediately after shooting) in order to obtain a desired combination of composition and growth degree. In order to make such a change and confirmation, an image composition process is executed.
As described above, it is desirable to perform image composition processing before confirming recording in the storage memory immediately after shooting (before shifting to the next shooting standby state), but after recording in the storage memory. It is also possible to read the captured image at an arbitrary timing and execute the image composition processing.
Further, for example, a CPU with higher processing capability may be used to display the image data sequentially obtained for live view display while performing image synthesis processing in real time.
In addition, the growth degree adjustment mode and the seasonal change mode are set and stored in advance before shooting, and when the shooting instruction is issued, the user does not perform any other instruction operation. An image synthesis process may be automatically executed under conditions, and a captured image after the growth degree is changed may be automatically recorded.

  FIG. 11 is a flowchart showing an example of the flow of image composition processing according to the second embodiment.

  In step S101, the original image acquisition unit 41 acquires original image data.

In step S102, the target subject specifying unit 42 executes the target subject specifying process to specify the shape and position of each of the one or more target subjects from the original image data acquired in the process of step S1.
It should be noted that various types of processing can be employed as the target subject specifying processing, including the examples of FIGS. 6 and 7 described above.
Here, for convenience of explanation, the following description will be given on the assumption that the cherry tree is the target subject and the shape and position of the cherry tree are specified.

In step S <b> 103, the growth image generating unit 71 accepts a setting mode selection operation for the set growth degree.
That is, as described above, there are two types of setting modes, the growth degree adjustment mode and the seasonal change mode, as the set growth degree setting modes. For this reason, in this embodiment, the user can select a desired mode among these two types of setting modes by operating the operation unit 18.
When the growth image generation unit 71 accepts such a selection operation performed on the operation unit 18 by the user, the process proceeds to step S104.

  In step S104, the growth image generation unit 71 determines whether or not the selected setting mode is the seasonal change mode by interpreting the content of the selection operation received in the process of step S3.

When the growth degree addition / subtraction mode is selected as the setting mode of the set growth degree, it is determined as NO in Step S104, and the process proceeds to Step S105.
In step S105, the growth image generation unit 71 determines the growth degree (including the flowering degree of cherry blossoms and the like) directly designated by the user by operating the operation unit 18 as the set growth degree.
Accordingly, the process proceeds to step S109. However, the processing after step S109 will be described later.

  On the other hand, when the seasonal change mode is selected as the setting mode of the set growth degree, it is determined as YES in step S104, and the following processes of steps S106 to S108 are executed.

  In step S <b> 106, the growth image generating unit 71 controls the display control unit 45 (not shown in FIG. 10) to display a list of regions and seasons that can be selected by the user on the display unit 19.

  The user who sees the list displayed on the display unit 19 operates the operation unit 18 to select a desired region and a desired season.

In step S107, the growth image generation unit 71 recognizes the region and season patterns selected by the user.
When the user does not select an area, in this embodiment, the growth image generation unit 71 regards that the imaging area of the original image has been selected, and meta information ( The imaging area is recognized from the position information.
When the user does not select a season, in this embodiment, the growth image generation unit 71 regards that the season when the original image was captured is selected, and is attached to the data of the original image. The season is recognized from the meta information (time information).

  In step S108, the growth image generation unit 71 reads growth information corresponding to the pattern recognized in the process of step S107 from the growth information storage unit 81, and determines a set growth degree based on the growth information.

  In this way, when the set growth degree is determined in the process of step S105 or S108, the process proceeds to step S109.

In step S109, the growth image generation unit 71 generates growth image data. That is, a growth that has grown in accordance with the set growth degree determined in the process of step S105 or S108 is placed in or inside the target object whose position and shape have been specified by the target object specifying process of step S102. The image data is generated as growth image data. Specifically, for example, in the above-described example, the cherry tree is the target subject, so the growth becomes a branch of the cherry tree, and as a result, each branch where the cherry blossoms bloom according to the degree of growth (flowering degree). Data of an image including a tree having “” is obtained as growth image data.
Such processing in step S109 is hereinafter referred to as “growth image generation processing” in accordance with the description of the same step in FIG. Details of the growth image generation processing will be described later with reference to a specific example of FIG.

  In step S110, the synthesizing unit 44 synthesizes the original image data acquired in step S101 and the growth image data generated in the growth image generation process in step S109, thereby generating a synthesized image. Generate data.

In step S111, the display control unit 45 causes the display unit 19 to display a composite image expressed by the data generated in the process of step S110.
Thereby, the image composition process is completed.

Heretofore, the image composition processing according to the second embodiment has been described with reference to the flowchart of FIG.
Next, a detailed example of the growth image generation processing in step S109 among such image synthesis processing will be described with reference to the flowchart of FIG.

  FIG. 12 is a flowchart for explaining an example of a detailed flow of the growth image generation processing according to the second embodiment.

In step S121, the growth image generating unit 71 executes a process of randomly arranging unit element images on the reference line (hereinafter referred to as “unit element image arrangement process” in accordance with the description of FIG. 12).
Here, the reference line of the first unit element image arrangement process is a contour line of a growth object arranged so as to be a part of or in contact with the target subject whose position and shape are specified by the target subject specifying process in step S102 of FIG. Is adopted.
The unit element image is an image of n × m pixels showing at least a part of the growth element of the growth, for example, if the growth is a branch of a cherry tree, a cherry color (pink color) showing at least a part of the cherry blossom. Etc.). n is an arbitrary integer value indicating the number of pixels in the horizontal direction. m is an integer value indicating the number of pixels in the vertical direction, and is an arbitrary integer value (including n = m) independent of n. n and m may be set within a range in which a growth product appears to grow naturally when unit element images are continuously arranged.
If all the unit element images are uniformly the same color, the portion where the unit element images are continuously arranged is uniformly the same color, and does not look like a natural growth product. For example, if the growth is a branch of a cherry tree, the same pink area is simply added to the branch, and the cherry blossoms do not appear to be naturally blooming.
Therefore, the color of the unit element image may be randomly changed within a range that does not become unnatural as a color at the time of growth, for example, a color at the time of blooming cherry blossoms. Alternatively, the color of each n × m pixel constituting the unit element image may be randomly changed within a range that does not become unnatural as a color at the time of growth, for example, a color at the time of blooming cherry blossoms.
When there are a plurality of growths, the unit element image arrangement process is executed for each of the plurality of growths. For example, when the target subject is a cherry tree, there are a plurality of branches that are grown, so unit element image placement processing is executed for each of the plurality of branches, and as a result, a cherry blossom (unit The element image) is arranged so as to bloom.

  In step S122, the growth image generating unit 71 updates the contour line of the unit element image at the time when the processing of the previous step S121 is completed as a new reference line.

  In step S123, the growth image generating unit 71 determines the growth of the growth in the image based on the height from the initial reference line (above the target subject) to the reference line updated in the immediately preceding step S122. The degree of growth (such as the degree of cherry blossom bloom) is recognized, and it is determined whether or not the degree of growth exceeds the set degree of growth determined in step S105 or S108 of FIG.

If the growth degree of the growth in the image has not reached the set growth degree, it is determined as NO in step S123, the process returns to step S121, and the subsequent processes are repeated.
That is, the loop processing of steps S121 to S123 is repeatedly executed until the growth degree (such as the flowering degree of cherry blossoms) in the image reaches the set growth degree, and the unit element image arrangement process is executed each time. Is done.
As a result, the unit element images are gradually stacked on the growth object arranged to be in contact with a part of the subject. For example, when the target subject is a cherry tree, the flowering progresses as the cherry blossoms (unit element images) are gradually stacked on each of a plurality of branches that are grown. I will be forgotten.
Thereafter, when the growth degree of the growth in the image exceeds the set growth degree (when the cherry blossom flowering degree exceeds the set flowering degree), it is determined as YES in step S123, and the growth image generation process ends. .
When there are a plurality of growths, the processes in steps S121 to S123 are performed on each of the plurality of growths, and the size of the plurality of growths in the image increases. In this case, for example, when the growth degree corresponding to the maximum size among the sizes in the image of the growth exceeds the set growth degree, it is determined as YES in Step S123, and the growth image generation process is ended.

When the growth image generation process is completed in this manner, an image having the same resolution (same size) as the original image and grown at a growth amount based on the set growth degree is located at or inside the set subject. The arranged image data is generated as growth image data.
For example, when the target subject is a cherry tree and the growth is each of a plurality of branches, the image includes a cherry tree having each of a plurality of branches flowered based on a set growth degree (set flowering degree). Are generated as growth image data.
Thereafter, the process proceeds from step S109 to step S110 in FIG. 11, and such growth image data is combined with the original image data to generate combined image data.

As described above, the imaging apparatus 1 according to the second embodiment of the present invention includes the original image acquisition unit 41, the target subject specifying unit 42, the operation unit 18, the growth image generation unit 71, and the synthesis unit 44. And.
The original image acquisition unit 41 acquires data of an original image including a subject.
The target subject specifying unit 42 is a subject (for example, a cherry tree or the like) that can be synthesized so that at least a part of a growth (for example, a cherry tree branch) that is an example of a predetermined object touches or overlaps the original image data. ) As a target subject, the shape of the target subject is specified.
The operation unit 18 receives the set growth degree as a reference for the growth amount (synthesis amount) of the growth specified by the user.
The growth image generation unit 71 generates, as growth image data, data of an image in which a growth grown with a growth amount based on the set growth degree is arranged inside or in contact with the set subject.
The combining unit 44 generates combined image data by combining the original image data and the growth image data.

  Therefore, the imaging apparatus 1 according to the second embodiment of the present invention can generate composite image data in a state in which a growth has grown on the captured image. Such a composite image is, for example, an image imitating the growth state of a growth product in a different region or season.

Here, a case where the user captures an image of a landscape seen from a predetermined place (for example, A-ku, Tokyo, Japan) as the captured image using the imaging device 1 is considered. However, the purpose of the user's imaging action is to obtain an image of a cherry tree landscape at the time of flowering that can be seen from the predetermined place (for example, A-ku, Tokyo, Japan).
In this case, even if the cherry blossoms are not actually blooming at the time of shooting by the user, that is, even if an image of the landscape of the cherry blossom trees that has not blossomed is obtained as the original image, it is flowered as a composite image at a desired flowering degree It is possible to easily obtain an image of the cherry tree landscape.
Further, a region indicating a cherry blossom of a certain area is not attached to each branch, but the amount of flowering (the area of the region indicating a cherry blossom) is determined based on the set flowering degree specified by the user. ) Is determined for each branch (growth), it is possible to realize image processing that can express a more natural and realistic appearance of cherry blossoms.

The effects that can be achieved by the imaging apparatus 1 of the second embodiment of the present invention will be specifically described as follows (1) and (2), for example.
(1) Since it is possible to easily obtain an image of a desired landscape suitable for the user's drawing intention, such as a landscape of a cherry tree blossomed at a desired flowering degree in a predetermined place, for a user, Can reduce shooting failures or facilitate the cover at the time of failure.
(2) There are shootings that are difficult to re-shoot, such as shooting in places where it is difficult to re-shoot such as overseas travel destinations, and shooting of subjects that are difficult to reproduce or repeat the same state. Even in such a case, the user can obtain a composite image close to a desired landscape by using a captured image obtained by one shooting as an original image, and further, data of the composite image. Can be stored data. As a result, re-shooting is unnecessary.

  In addition, this invention is not limited to the above-mentioned embodiment, The deformation | transformation in the range which can achieve the objective of this invention, improvement, etc. are included in this invention.

For example, in the above-described embodiment, the synthesis target to be synthesized with the original image is a deposit represented by snow in the first embodiment and a growth in the second embodiment, but is not particularly limited thereto. .
In other words, the image processing apparatus to which the present invention is applied only needs to have an original image acquisition function, a subject identification function, a reception function, an object image generation function, and a composition function. The form is not particularly limited.
Here, the original image acquisition function is a function for acquiring data of an original image including a subject.
The subject specifying function is a function for specifying the shape of a target subject for a subject that can be synthesized so that at least a part of a predetermined object touches or overlaps the original image data.
The accepting function is a function for accepting a reference for the amount of synthesis of a predetermined object designated by the user.
The object image generation function refers to data of an image in which a predetermined amount of a composite object based on an accepted reference is in contact with or overlaps at least a part of the shape of a subject as object image data. It is a function to generate.
The composition function is a function for generating composite image data by compositing original image data and object image data.
As described in the first embodiment and the second embodiment, the image processing apparatus having each of these functions is an image processing that matches the purpose of the user's imaging action, and is more natural. It is possible to execute image processing that can express a realistic landscape.

Further, for example, in the above-described embodiment, only the snow cover portion is changed with respect to the original image. However, since the amount of reflected light increases as the snow increases, the brightness is increased even in areas without snow. May be corrected.
Further, for example, in the above-described embodiment, the target subject is specified from the landscape image in which snow is not piled up. You can also adopt this method.

  For example, in the above-described embodiment, the image processing apparatus to which the present invention is applied has been described as an example configured as an imaging apparatus such as a digital camera. However, the present invention is not particularly limited to an imaging apparatus, and is applicable to any electronic device that can execute the above-described image processing regardless of whether or not an imaging function is provided (data of a captured image may be acquired from another apparatus). can do. Specifically, for example, the present invention can be applied to a personal computer, a video camera, a portable navigation device, a portable game machine, and the like.

  The series of processes described above can be executed by hardware or can be executed by software.

  When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium. The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only constituted by the removable medium 31 of FIG. 1 distributed separately from the apparatus main body in order to provide the program to the user, but also in a state of being incorporated in the apparatus main body in advance. It is comprised with the recording medium etc. which are provided in this. The removable medium is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preinstalled in the apparatus main body is configured by, for example, the ROM 12 in FIG. 1 in which the program is recorded, the hard disk included in the storage unit 20 in FIG.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.

DESCRIPTION OF SYMBOLS 1 ... Imaging device, 11 ... CPU, 12 ... ROM, 13 ... RAM, 18 ... Operation part, 20 ... Storage part, 31 ... Removable media, Drive, 41 ..Original image acquisition unit, 42... Target subject specifying unit, 43... Snow image generation unit, 44... Composition unit, 45. ... Growth image generation unit, 81 ... Growth information storage unit

Claims (14)

Original image acquisition means for acquiring an original image to be processed;
Subject specifying means for specifying the position and shape of a subject portion on which a predetermined deposit in the original image acquired by the original image acquiring means can be deposited ;
Wherein in correspondence with the position and shape of the object portion which is the predetermined deposit identified may be deposited by subject specifying unit, by combining by placing the object image indicating the predetermined deposit on said original image Combining means for generating a combined image;
An image processing apparatus comprising: Further comprising first specifying means for specifying the type of deposit to be synthesized on the original image;
The subject specifying means specifies the position and shape of the subject portion suitable for the synthesis of the type of deposit designated by the first designation means,
The synthesizing unit arranges an object image indicating the type of deposit designated by the first designation unit on the original image in correspondence with the position and shape of the subject portion on which the deposit can be deposited. The image processing apparatus according to claim 1, wherein: Second designation means for designating a deposition amount of the predetermined deposit ;
The synthesizing unit corresponds to the deposition amount designated by the second designation unit on the original image in correspondence with the position and shape of the subject portion on which the predetermined deposit specified by the subject specifying unit can be deposited. The image processing apparatus according to claim 1, wherein an object image indicating a predetermined amount of the predetermined deposit is arranged and synthesized. The predetermined deposit, further comprising an object image generating means for generating an object image in a state of being arranged in a corresponding shape to the shape of the object portion which is the predetermined deposit identified may be deposited by the object specifying means ,
The synthesizing unit arranges and synthesizes the object image generated by the object image generating unit at a position of a subject portion where the predetermined deposit specified by the subject specifying unit can be deposited. Item 4. The image processing apparatus according to Item 3. Imaging means for capturing an image of a subject and acquiring captured image data;
Display means for displaying the captured image obtained as data by imaging by the imaging means;
Recording means for recording data of the captured image obtained by imaging by the imaging means;
Further comprising
The original image acquisition unit acquires data of the captured image obtained by imaging by the imaging unit as data of the original image to be processed,
The display means displays the synthesized image generated by the synthesizing means as the captured image,
The image processing apparatus according to claim 4 , wherein the recording unit records data of the synthesized image generated by the synthesizing unit as data of the captured image. The recording means records the composite image data in response to a recording instruction operation by a user after the composite image is displayed by the display means.
The image processing apparatus according to claim 5.   After the composite image is displayed by the display means, until the recording instruction operation by the user is performed, the composite amount specified by the second specification means by the user and the object image by the object image generation means 7. The image according to claim 6, further comprising control means for executing control for repeating generation of the composite image, generation of the composite image by the composite means, and display of the composite image by the display means. Processing equipment. The synthesizing unit is an object indicating a state in which the predetermined deposit corresponding to the deposition amount specified by the second specifying unit is accumulated on the upper portion of the predetermined subject portion specified by the subject specifying unit. 5. The image processing apparatus according to claim 3 , wherein an image is arranged on the original image and synthesized. The predetermined deposit is snow;
The image processing apparatus according to claim 8. The synthesizing unit generates an object image indicating a state in which the snow is accumulated by executing a predetermined snowfall simulation.
The image processing apparatus according to claim 9. The subject specifying means specifies a three-dimensional shape of a subject portion on which the predetermined deposit can be deposited ,
The compositing unit is configured to synthesize an object image indicating the predetermined deposit based on a three-dimensional shape of a subject portion on which the predetermined deposit specified by the subject specifying unit can be deposited. The image processing apparatus according to claim 1, further comprising: The subject specifying unit generates a saliency map based on a plurality of feature amounts based on the original image, estimates a point-of-interest area based on the saliency map, and determines the predetermined point based on the point-of-interest area. Identify the location and shape of the subject part where the deposit can accumulate ,
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. In the image processing method executed by the image processing apparatus,
An original image acquisition step of acquiring an original image to be processed;
A subject specifying step for specifying a position and a shape of a subject portion where a predetermined deposit in the original image acquired by the original image acquiring step can be deposited ;
The subject specified in correspondence with the position and shape of the object portion which is the predetermined deposit identified may be deposited by step, by combining by placing the object image indicating the predetermined deposit on said original image A compositing step for generating a composite image;
An image processing method comprising: Computer
Original image acquisition means for acquiring an original image to be processed;
Subject specifying means for specifying the position and shape of a subject portion on which a predetermined deposit in the original image acquired by the original image acquiring means can be deposited ;
Wherein in correspondence with the position and shape of the object portion which is the predetermined deposit identified may be deposited by subject specifying unit, by combining by placing the object image indicating the predetermined deposit on said original image A compositing means for generating a composite image;
Program to function as.

Images