JP5440366B2 - Image processing apparatus and method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, method, and program, and in particular, a technique that makes it possible to process a captured image so as to be a more natural and realistic image that matches the purpose of a user's imaging action. About.

2. Description of the Related Art Conventionally, there is a field of display technology in which an electronic display device displays an image for viewing and a car navigation device displays an image for information confirmation such as a map. In the field of such display technology, effect display according to time change and weather change is performed (see Patent Documents 1 to 4).
On the other hand, there is a field of imaging technology in which a user takes an image using a camera. Also in the field of such imaging technology, with the advent of digital cameras, it is possible to display captured images, and in recent years, it has also become possible to process and display captured images.

Japanese Patent Laid-Open No. 05-199491 Japanese Patent Application Laid-Open No. 07-210123 Japanese Patent Laid-Open No. 08-16586 JP 2002-286461 A

However, in the field of imaging technology, in general, an imaging action in which a user takes an image using a digital camera is a main action rather than an action in which the user confirms a captured image obtained as a result. Here, the imaging action is an action accompanied by a user's subjective purpose, and is an active action of creating an image that achieves the purpose. For this reason, it is required that the captured image, which can be said to be created by the user, be processed in accordance with the purpose of the active action called the imaging action.
On the other hand, in the field of display technology, an image to be displayed is often an image created by another person other than the user. And the act of appreciating and confirming images created by others is the main act. In other words, the field of display technology has been developed as a main act is a passive act in which a user views or confirms a unilaterally presented image rather than an active act such as an imaging act. It is.
Therefore, even if the technology in the field of display technology, for example, the technology described in Patent Documents 1 to 4 is applied to the field of imaging technology as it is, it is difficult to apply processing that matches the purpose of the user's imaging action to the captured image. is there.

For example, a user visits a predetermined place to image a landscape such as autumn leaves or cherry blossoms, but the user visits at a time slightly off from the best time, and the autumn leaves or cherry blossoms are not yet in full bloom. Let's assume a case where only the landscape can be imaged.
The purpose of the user's imaging action in this case should have been to capture a landscape in which the autumn leaves and cherry blossoms were in full bloom. However, in this case, only a captured image showing a landscape in which autumn leaves and cherry blossoms are not yet in full bloom is obtained. Such a captured image is an image in which the purpose of the imaging action is not achieved. For this reason, it is required to process the captured image so as to achieve an image in which the purpose of the imaging action is achieved, that is, an image in which the scenery in which autumn leaves and cherry blossoms are in full bloom is reflected.
However, simply applying the techniques described in Patent Documents 1 to 4 makes it difficult to meet such requirements. That is, the techniques described in Patent Documents 1 to 4 are the first technique for automatically selecting and displaying a picture or image to be displayed in accordance with a change in time or a change in weather, or a display of color, brightness, or the like. This is a second technique in which the adjustment data is preset and the entire image is corrected according to the current season, date, weather, and the like. In the former first technique, it is necessary to prepare patterns and images to be selected and displayed in advance. However, it is not realistic to prepare images of trees in the real world in advance. In the latter second technique, not only the leaves and flower portions of the tree but also the color and brightness of the entire image are corrected, so that only unnatural and unrealistic correction can be performed.

Conventionally, in 3D computer graphics (CG) and architectural landscape simulation software, the architectural landscape viewed from various viewpoints is rendered, the surrounding landscape such as trees, and the shade due to seasonal changes, etc. There is something that can be simulated.
However, it is clear that the images obtained by these simulation software are virtual and artificial images compared to captured images (photo images) obtained as a result of imaging the real world, which is unnatural and realistic. It was lacking.
Furthermore, it is assumed that these simulation softwares perform simulation drawing based on things different from the real world, such as design data and a three-dimensional shape model. For this reason, it is very difficult to apply these simulation softwares as they are for processing a captured image obtained as a result of imaging a real world landscape or the like.

  In summary, there is a demand for processing a captured image so as to be a more natural and realistic image that matches the purpose of the user's imaging action. However, the conventional techniques including Patent Documents 1 to 4 cannot sufficiently meet the demand. For this reason, the realization of the technique which can fully respond to the said request is calculated | required.

  The present invention has been made in view of such a situation, and an object thereof is to process a captured image so as to be a more natural and realistic image that matches the purpose of the user's imaging action. .

According to one aspect of the invention,
In an image processing apparatus capable of capturing and reproducing an image including a specific region whose characteristics change depending on time and position,
Holding means for holding time variation characteristic information stored in advance in association with the position and time, the characteristics of the specific region;
When playing back an image including the captured specific area, the characteristic of the specific area is read from the time change characteristic information, the read characteristic is compared with the characteristic when the image is captured, and the captured image is captured. Correction means for correcting the characteristics of
Playback means for playing back an image including the corrected specific area;
An image processing apparatus is provided.

  According to another aspect of the present invention, an image processing method and a program corresponding to the above-described image processing apparatus according to one aspect of the present invention are provided.

  ADVANTAGE OF THE INVENTION According to this invention, a captured image can be processed so that it may become a more natural and realistic image according to the objective of a user's imaging action.

It is a block diagram which shows the structure of the hardware of the imaging device which concerns on one Embodiment of this invention. It is a chromaticity diagram of the xyY color system showing the time change of the chromaticity of deciduous trees. It is a figure which shows an example of the result of a time-dependent change correction process which the imaging device of FIG. 1 performs. It is a functional block diagram which shows the functional structure about the function which implement | achieves a time-dependent change correction process among the functions which the imaging device of FIG. 1 has. 6 is a flowchart for explaining a flow of a temporal change correction process executed by the imaging apparatus of FIG. 4. 6 is a flowchart illustrating an imaging process executed by the imaging processing apparatus of FIG. 1 and including the temporal change correction process of FIG. 5 as a subroutine process. 6 is a flowchart illustrating a display process executed by the imaging processing apparatus of FIG. 1 and including a time-dependent change correction process of FIG. 5 as a subroutine process. It is a figure which shows an example of the GUI image used by the time-dependent change correction process of FIG. It is a figure which shows an example of the result of the correction | amendment using the time color change characteristic information for every some partial object divided from the predetermined color change object with time. An example of a color state transition based on a stochastic process, showing a change in the color of a leaf when the temporal change target is a deciduous tree, is shown. FIG. 4 is a diagram illustrating an example of a result of a temporal change correction process performed by the imaging apparatus of FIG. 1 and an example different from FIG. 3.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a hardware configuration of an imaging apparatus 1 as an embodiment of an image processing apparatus according to the present invention. The imaging device 1 can be configured by a digital camera, for example.

  The imaging apparatus 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an imaging unit 16, and an operation unit 17. A display unit 18, a storage unit 19, a communication unit 20, a GPS (Global Positioning System) unit 21, and a drive 22.

The CPU 11 executes various processes according to programs recorded in the ROM 12. Alternatively, the CPU 11 executes various processes according to a program loaded from the storage unit 19 to the RAM 13.
The RAM 13 also appropriately stores data necessary for the CPU 11 to execute various processes.

  For example, in the present embodiment, programs for realizing the functions of the captured image acquisition unit 51 to the time change characteristic information determination unit 59 in FIG. 4 to be described later are stored in the ROM 12 and the storage unit 19. Therefore, when the CPU 11 executes processing according to these programs, each function of the captured image acquisition unit 51 to the time change characteristic information determination unit 59 of FIG. 4 to be described later can be realized.

  The CPU 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output interface 15 is also connected to the bus 14. An imaging unit 16, an operation unit 17, a display unit 18, a storage unit 19, a communication unit 20, and a GPS unit 21 are connected to the input / output interface 15.

  Although not shown, the imaging unit 16 includes an optical lens unit and an image sensor.

The optical lens unit is configured by a lens that collects light, for example, a focus lens or a zoom lens, in order to photograph a subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

The image sensor includes a photoelectric conversion element, an AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) a subject image at regular time intervals to accumulate image signals, and sequentially supplies the accumulated image signals to the AFE as analog signals.
The AFE executes various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. Through various signal processing, a digital signal is generated and output as an output signal of the imaging unit 16.
Hereinafter, the output signal of the imaging unit 16 is referred to as “captured image data”. Therefore, captured image data is output from the imaging unit 16 and is appropriately supplied to the CPU 11 and the like.

The operation unit 17 includes various buttons including a release button 41 and receives a user instruction operation.
The display unit 18 displays various images.
The storage unit 19 is configured by a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores captured image data output from the imaging unit 16. The storage unit 19 also stores various data necessary for various image processing, such as image data, various flag values, threshold values, and the like.
The communication unit 20 controls communication performed with other devices (not shown) via a network including the Internet.
The GPS unit 21 receives GPS signals from a plurality of satellites for GPS (Global Positioning System), and information on latitude and longitude indicating the current position of the imaging device 1 based on the GPS signals (hereinafter referred to as “GPS position information”). ").

  A drive 22 is connected to the input / output interface 15 as necessary, and a removable medium 31 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted. And the program read from them is installed in the memory | storage part 19 as needed. The removable medium 31 can also store various data such as image data stored in the storage unit 19 in the same manner as the storage unit 19.

In the present embodiment, the imaging apparatus 1 having such a hardware configuration executes a temporal change correction process.
The temporal change correction process refers to a process of correcting the temporal change target image included in the captured image in accordance with a change in the temporal direction of the temporal change target existing in the real world.

Here, what changes a predetermined feature in the real world in the time direction according to a certain rule is referred to as a “time-change target” in this specification.
It should be noted that the “constant law” that changes the predetermined characteristics of the object that changes over time is not limited to any kind of law including any natural law, and may not be mathematically expressed. For example, the color becomes red over time. It may be a simple law that tends to become. In addition, a change in a predetermined feature of a temporal change target may be determined by only one type of law, or in addition to a plurality of laws, uncertain elements such as random numbers may be combined to determine a predetermined feature of the temporal change target. In some cases, changes are dictated.

  In the following description, for convenience of explanation, “color” is focused as “predetermined feature”. That is, an example in which the color changes in the time direction according to a certain rule will be described below as an example of the temporal change target. Such a device is hereinafter referred to as a “color change target with time”.

For example, deciduous trees such as maple leaves fall in autumn in Japan, that is, the color of the leaves gradually changes to red, yellow, brown, etc. over time. It is.
FIG. 2 is a chromaticity diagram of the xyY color system showing temporal changes in chromaticity of deciduous trees. Specifically, FIG. 2 (A) is a chromaticity diagram of the xyY color system showing the temporal change in chromaticity of a ginkgo that is one type of deciduous tree, and FIG. 2 (B) is another type of deciduous tree. It is a chromaticity diagram of the xyY color system showing the time change of chromaticity of a certain yellow maple.
In FIG. 2A, curves LdA, LmA, and LuA indicate that the chromaticity of the leaves of each layer of the ginkgo changes in the direction of the arrow from October to December. That is, the curve LdA shows the time change of the chromaticity of the leaf of the lower part of the ginkgo. A curve LmA shows a temporal change in the chromaticity of the leaf of the middle layer of the ginkgo. Curve LuA shows the time variation of the chromaticity of the leaves of the upper layer portion of the ginkgo.
In FIG. 2B, curves LdB, LmB, and LuB indicate that the chromaticity of the leaves of each layer of yellow maple changes in the direction of the arrow from October to December. That is, the curve LdB shows the time change of the chromaticity of the leaf of the lower part of the maple leaf. A curve LmB indicates a temporal change in the chromaticity of the leaf of the middle part of the maple leaf. Curve LuB shows the time variation of the chromaticity of the leaves of the upper layer portion of the Irohakaede.
As shown in FIG. 2, it can be seen that the color of the leaves of both ginkgo and yellow maple changes with time. Specifically, when compared within the same tree, here, the curves LdA, LmA, and LuA in FIG. 2A are compared, or the curves LdB, LmB, and LuB in FIG. By comparison, it can be seen that the law of change has almost the same tendency, although the chromaticity change and timing differ slightly depending on each layer. On the other hand, when comparing between different types of trees such as ginkgo and yellow maple, here, the curves LdA of FIG. 2A and the curves LdB of FIG. 2B, the curves LmA of FIG. When comparing the curves LmB of 2 (B) and the curves LuA of FIG. 2 (A) and the curves LuB of FIG. 2 (B), the law of change depends on the kind of trees such as Ginkgo and Acer palmatum. You can see that they are different.
In addition to such deciduous trees, examples of color change objects over time include flower trees such as ume and cherry, overhead space (sky), and the like. In other words, in Japanese flowers such as ume and cherry blossoms, the color near the branches gradually changes to red or pink by gradually blooming from early spring to spring in Japan. In addition, in the overhead space, the sky color gradually changes over time during the day.

When such an image of a color change target with time is included in the captured image, the imaging apparatus 1 specifies a region including the image of the color change target with time as a correction target. Hereinafter, such a region is referred to as a “specific region”.
In addition, the imaging device 1 acquires a time different from the imaging time or a difference time from the imaging time as the correction condition. When acquiring a time different from the imaging time as the correction condition, the imaging device 1 obtains a difference time between the current time and the correction time using the imaging time as the current time and the acquired time as the correction time. Hereinafter, the difference time obtained in this way and the difference time acquired as the correction condition are referred to as “conditional difference time”.
Then, the imaging device 1 corrects the coordinates of the color space of the data in the specific area according to the color change that occurs in the real world during the condition difference time for the color change target with time.
Such a series of processes is a temporal change correction process.

Further, with reference to FIG. 3, the temporal change correction process will be specifically described below.
FIG. 3 shows an example of the result of the temporal change correction process.
For example, as a result of imaging a scene including a house and a deciduous tree at a time TN slightly before the best time to see autumn leaves, a captured image 41N including a house image 42 and a deciduous tree image 43N as shown in FIG. Is obtained. Note that the apparatus that captures the captured image 41N may be the imaging apparatus 1 or another apparatus.
As described above, since the imaging time (current time) of the captured image 41N is a time TN slightly before the time when the autumn leaves are in full bloom, it is assumed that the user desires to obtain an image at the time when the autumn leaves are in full bloom. . In such a case, the user can set a future time TF corresponding to the best time to see the autumn leaves by operating the operation unit 17 (FIG. 1).
In this case, the imaging device 1 obtains the difference time between the imaging time TN and the set time TF as the condition difference time.
Moreover, the imaging device 1 detects the leaf part of the deciduous tree image 43N as a specific region based on the data of the captured image 41N.
Then, the imaging apparatus 1 corrects the coordinates of the color space of the data in the specific area according to the change in the leaf color that occurs in the real world during the condition difference time for the deciduous tree.
Thereby, the data of the corrected image 41F shown in FIG. 3A is obtained from the data of the captured image 41N shown in FIG. In other words, when the time elapses from the time TN of imaging to the time TF, the autumn leaves of the deciduous tree advance in the real world, and the leaves of the deciduous tree increase in red, yellow, brown, and the like. Therefore, the corrected image 41F includes a deciduous tree image 43F that has been corrected so that most of the leaf portions detected as the specific region are red, yellow, brown, or the like. On the other hand, the house image 42 in the captured image 41N is not detected as a specific area, assuming that the real-world house does not change in color over time. As a result, the house image 42 is included in the corrected image 41F without being corrected.

  In addition, when viewed on a daily basis, the color of a real-world house may change from morning to night. However, in the example of FIG. 3, the time change in the color of the leaves of the deciduous tree in the real world means a time change over a long span such as autumn to winter in Japan. The time change of the unit is ignored. However, it is of course possible to execute the temporal change correction process in consideration of such a daily time change.

Here, in order to enable correction according to the temporal change of the color in the real world for the temporal color change target, information indicating the characteristics of the temporal color change in the real world for the temporal color change target is captured. It is assumed that the device 1 holds it in advance. Such information is hereinafter referred to as “temporal color change characteristic information”.
For example, with respect to a color change target with time in the real world, information indicating the time transition of the color and information indicating the law of the time change of the color are examples of the time color change characteristic information.
Specifically, for example, curve LdA shown in FIG. 2 (A), LMA, specific information capable of LUA, can be employed as a time change in color characteristic information of ginkgo.
In this case, the imaging apparatus 1 uses the ginkgo leaf portion in the captured image as the specific area, and corrects the coordinates of the color space of the specific area using the temporal color change characteristic information of the ginkgo. That is, the imaging apparatus 1 uses the chromaticity coordinates of the point corresponding to the imaging time TN (gray point in the figure) of the curves LdA, LmA, and LuA shown in FIG. 2A and the corrected time TF. The correction content is determined based on the chromaticity coordinates of the point corresponding to (black point in the figure). Then, the imaging apparatus 1 corrects the coordinates of the color space of the data in the specific area according to the determined correction content.

Furthermore, the user can set not only the future time but also the past time with respect to the imaging time TN. For example, even when the past time TP is set with respect to the imaging time TN, the above-described series of processing is executed as the temporal change correction processing.
Thereby, the data of the corrected image 41P shown in FIG. 3C is obtained from the data of the captured image 41N shown in FIG.
That is, when the time returns from the imaging time TN to the time TP (elapses in the reverse direction), the deciduous tree in the real world is in a state before the autumn leaves, so the color of the leaves increases in green. Therefore, the corrected image 41P includes the deciduous tree image 43P corrected so that most of the leaf portions detected as the specific region are green. On the other hand, the house image 42 in the captured image 41N is not detected as a specific area, assuming that the real-world house does not change in color over time. As a result, the house image 42 is included in the corrected image 41P as it is without being corrected.

  FIG. 4 is a functional block diagram showing a functional configuration of the imaging apparatus 1 that executes such a temporal change correction process.

  As illustrated in FIG. 4, the imaging device 1 includes a captured image acquisition unit 51, an imaging condition acquisition unit 52, a correction condition acquisition unit 53, a condition difference acquisition unit 54, and a specification so as to realize a temporal change correction process. An area detection unit 55, a change data acquisition unit 56, a specific area correction unit 57, a corrected image generation unit 58, a time change characteristic information determination unit 59, and a time change characteristic information storage unit 60 are provided.

In the present embodiment, each of the captured image acquisition unit 51 to the time change characteristic information determination unit 59 is a combination of the hardware called the CPU 11 and the program (software) stored in the ROM 12 or the like in the configuration shown in FIG. It is configured.
In addition, the time change characteristic information storage unit 60 is configured as an area in the RAM 13 or the storage unit 19 of the imaging apparatus 1 or the removable medium 31 in the configuration illustrated in FIG. 1. Note that the time change characteristic information storage unit 60 is held in a device other than the imaging device 1, such as a server, and the imaging device 1 controls the communication unit 20 when executing the temporal change correction process. You may make it access the time change characteristic information storage part 60 hold | maintained at this apparatus.

Details of each of the captured image acquisition unit 51 to the time change characteristic information determination unit 59 will be described with reference to FIG. 5 as appropriate.
FIG. 5 is a flowchart for explaining the flow of the temporal change correction process executed by the imaging apparatus 1 of FIG.

In step S1, the time variation characteristic information determination unit 59 sets, for example, a scene such as a mountain range, a forest, or a tree as one unit, or sets each type of tree or vegetation as one unit, and sets a time color for each set unit. Set each change characteristic information.
In addition, the above-mentioned unit is an illustration, Arbitrary units can be employ | adopted. In this case, hierarchized units may be employed.
For example, when a predetermined type of deciduous tree such as ginkgo or yellow maple is used as a unit, the unit can be further divided (hierarchized) into a plurality of units in the spatial direction. That is, even with the same date and time, the progress of autumn leaves differs between the Kyushu region in the south of Japan and the Tohoku region in the north. Therefore, each region of Japan can be adopted as a unit below the predetermined type of deciduous tree. In this case, for a predetermined type of deciduous tree, time color change characteristic information indicating a time change in leaf color is set for each region in Japan.
The form of temporal color change characteristic information is not particularly limited. For example, the time transition of HSB values (each value of hue H, saturation S, and lightness B), the time transition of chromaticity in XYZ chromaticity coordinates, L * a * b * Time transition of color space coordinates, etc. can be adopted.
In this way, the time color change characteristic information set for each unit by the time change characteristic information determination unit 59 is stored in the time change characteristic information storage unit 60.

Note that the processing in step S1 and the processing in step S2 and later described below do not have to be continuous in time, and it is sufficient that the processing in step S1 is completed before the processing in step S2. . That is, after the process of step S1 is executed in advance and the time color change characteristic information for each unit is stored in the time change characteristic information storage unit 60, the imaging apparatus 1 performs step S2 at an arbitrary timing. Subsequent processing can be executed.
Therefore, when the temporal change correction process is executed as a subroutine process of an imaging process shown in FIG. 6 or a display process shown in FIG. 7 described later, the time color change characteristic information for each unit is already stored in the time change characteristic information storage unit 60. If so, the process of step S1 can be omitted as appropriate.

In step S2, the captured image acquisition unit 51 acquires captured image data.
Here, the imaging time (current time) of the captured image need not be a time synchronized with the processing timing of step S2, and may be an arbitrary time before the processing of step S2. That is, the captured image acquisition unit 51 can acquire captured image data captured at an arbitrary timing in the past.
The captured image data need not be captured by the image capturing apparatus 1 and may be captured by another apparatus.
Therefore, the captured image acquisition unit 51 directly acquires the captured image data output from the imaging unit 16, and also the captured image data transmitted from another device and received by the communication unit 20, and the removable media 31. Data of the captured image read by the drive 22 can be acquired as appropriate.
In the following, in order to facilitate understanding of the flow of the flowchart of FIG. 5, the data of the captured image 41N including the house image 42 and the deciduous tree image 43N as shown in FIG. An example of processing at each step to be described later when is acquired is referred to as a “specific example”.

In step S3, the imaging condition acquisition unit 52 acquires imaging conditions related to the data of the captured image acquired in the process of step S2.
The imaging condition includes at least a time condition of the captured image such as an imaging time (current time) of the captured image and a difference time between the reference time (for example, the best time to see autumn leaves) and the imaging time.
As a specific example, since the imaging time of the captured image 41N is a time TN slightly before the best time to see autumn leaves, the time TN is acquired as at least one of the imaging conditions.
The other conditions are arbitrary, and in the present embodiment, the imaging conditions such as the imaging position are further included in the imaging conditions. For example, when the captured image is captured by the image capturing apparatus 1, GPS position information output from the GPS unit 21 (FIG. 1) at the time of capturing is acquired as the position condition of the captured image. Further, when the user operates the operation unit 17 (FIG. 1) and inputs position information about the captured image, the input position information is acquired as a position condition of the captured image.

In step S4, the correction condition acquisition unit 53 acquires correction conditions for the captured image data acquired in the process of step S2.
The correction condition includes at least a time condition of the corrected image, such as a correction time and a difference time between a reference time (for example, the best time to see autumn leaves) and the correction time.
As a specific example, for example, when the user sets a future time TF corresponding to the best time to see autumn leaves by operating the operation unit 17 (FIG. 1), the future time TF is the correction condition. Acquired as at least one.
Other conditions are arbitrary, and in the present embodiment, position conditions such as a place represented by a corrected image are further included in the correction conditions. For example, when the location represented by the corrected image is the location where the imaging device 1 is present at the time of processing in step S4, the GPS position information output from the GPS unit 21 at the time of processing in step S4 is the correction condition. Is acquired as a position condition which is one of the following. When the user operates the operation unit 17 and inputs corrected position information, the input position information is acquired as a position condition that is one of the correction conditions.
The correction condition acquisition method is not particularly limited, and for example, a method of acquiring a condition set by the user operating the operation unit 17 as the correction condition can be employed. Further, for example, it is possible to employ a method of acquiring, as the correction condition, a condition set by the imaging apparatus 1 by autonomous determination without the user's operation, that is, a condition set automatically. Furthermore, these methods may be used alone or in combination.

In step S5, the condition difference acquisition unit 54 acquires a difference (hereinafter referred to as “condition difference”) between the imaging condition acquired in the process of step S3 and the correction condition acquired in the process of step S4.
As described above, each of the imaging condition and the correction condition includes at least a time condition. Therefore, the difference time between the imaging time (current time) and the correction time, that is, the condition difference time is acquired as at least one of the condition differences.
As a specific example, the difference time (= TF−TN) between the imaging time TN and the future time TF is acquired as the condition difference time.
In the present embodiment, each of the imaging condition and the correction condition includes at least a position condition. Therefore, the difference between the position condition of the captured image and the corrected position condition, that is, information indicating the deviation before and after correction such as latitude and longitude (such information is hereinafter referred to as “conditional difference position”) is also Acquired as one of the condition differences.

In step S6, the captured image acquisition unit 51 converts the captured image data into a color area image format.
That is, in the present embodiment, the captured image data is a collection of color information of each pixel in a color space (RGB space or XYZ space) in which lightness information (luminance information) and color information are not separated. For this reason, if the data of the captured image is used as it is, it is difficult to distinguish the features for each region, which may affect the accuracy of data extraction for a specific region in step S7 described later.
Therefore, the captured image acquisition unit 51 uses color space (HSV space or L * a * b * space) in which lightness information (luminance information) and color information are separated as color information of each pixel constituting the captured image data. Etc.) color information. An image expressed in a color space in which lightness information and color information are separated is called a “color region image”.
As a specific example, the data of the captured image 41N shown in FIG. 3B is converted into a color area image form. However, in the following description of the specific example, for convenience of explanation, description will be made with reference to FIG.

In step S <b> 7, the specific area detection unit 55 detects specific area data from the captured image data.
The specific region is not particularly limited as long as it is a region including an image of a color change target with time. For example, area data including images such as sky, mountain ranges, forests, trees, sea river lakes, and human faces is detected as specific area data.
Furthermore, the number of specific areas detected in the process of step S7 is not particularly limited, and may be a single area or an arbitrary plural area.
When a plurality of specific areas are detected, any one specific area may be an area associated with another specific area, or may be an area independent of the other specific areas. In other words, each type of the plurality of specific areas may be the same as or different from other specific areas. Here, for convenience of explanation, it is assumed that a specific area is detected for each type of color change target with time.
As a specific example, only the data of the leaf portion of the deciduous tree image 43N in the captured image 41N shown in FIG.

In step S8, the specific area correction unit 57 selects a predetermined one of the one or more specific areas detected in the process of step S7 as a process target (hereinafter referred to as “target area”). To be extracted).
As a specific example, a leaf portion (specific region) of the deciduous tree image 43N is extracted as a region of interest in the captured image 41N illustrated in FIG.

In step S <b> 9, the specific area correction unit 57 selects a predetermined one of the pixels constituting the specific area extracted as the attention area in the process of step S <b> 7 as a processing target (hereinafter, “ This is referred to as a “pixel of interest”.
As a specific example, a predetermined one pixel is set as a pixel of interest in a pixel group constituting a leaf portion (region of interest) of the deciduous tree image 43N of the captured image 41N shown in FIG. 3B.

In step S10, the change data acquisition unit 56 selects appropriate temporal color change characteristic information based on the condition difference acquired in the process of step S5 and the type of the specific area set as the attention area in the process of step S8. Is determined from the time change characteristic information storage unit 60, and change data for the target pixel is acquired based on the specified time color change characteristic information.
In step S11, the specific area correction unit 57 converts the data of the target pixel into the change data.
Here, the change data refers to the chromaticity coordinates (one type of pixel value) after updating the target pixel, and may be data indicating the temporal color change characteristic information itself, or based on the temporal color change characteristic information. May be generated. The color space of the chromaticity coordinates acquired as the change data is not particularly limited. However, if the color space is specified by the color area image converted in the process of step S6, the change data is directly corrected for the target pixel. Since it can employ | adopt as a later pixel value, it is suitable.
As a specific example, for example, the deciduous tree image 43N included in the captured image 41N acquired in the process of step S2 is a ginkgo image, and information that can specify the curves LdA, LmA, and LuA shown in FIG. However, it is assumed that the time color change characteristic information is set in the process of step S1. In this case, in the process of step S10, the chromaticity coordinates of the point corresponding to the imaging time TN (gray point in the figure) of the curves LdA, LmA, and LuA shown in FIG. Based on the chromaticity coordinates of the point corresponding to the time TF (black point in the figure), change data of the target pixel is acquired. Then, in the process of step S <b> 11, the pixel-of-interest data is acquired as the change data.

In step S12, the specific area correction unit 57 determines whether or not the target pixel is the last pixel in the target area.
If there is a pixel that has not yet been set as the target pixel in the target region, it is determined NO in step S12, the process returns to step S9, and the subsequent processes are repeated.
That is, each time each pixel in the attention area is set as the attention pixel, the loop processing of Steps S9 to S12 is repeatedly executed, and the chromaticity coordinates indicating the color suitable for the condition difference are the change data of the attention pixel. And the data (pixel value) of the pixel of interest is updated to change data each time.
Thereafter, when all the pixels in the attention area are set as the attention pixels and the data of all the pixels are converted into the respective change data, it is determined as YES in Step S12, and the process proceeds to Step S13.
As a specific example, all of the pixel groups constituting the leaf part (attention area) of the deciduous tree image 43N of the captured image 41N shown in FIG. 3B are set as the attention pixel to constitute the leaf part. When the data of all the pixels in the pixel group is converted into each change data, it is determined as YES in Step S12, and the process proceeds to Step S13.

In step S13, the specific area correction unit 57 determines whether or not the attention area is the last specific area among the one or more specific areas detected in the process of step S7.
If there is a specific area that has not yet been set as the attention area among the one or more specific areas detected in the process of step S7, it is determined as NO in step S13, and the process returns to step S8. The subsequent processing is repeated.
That is, every time one or more specific areas detected in the process of step S7 are set as the attention area, the loop process of steps S9 to S13 is repeatedly executed, and the data (pixels) of each pixel constituting the attention area is repeatedly executed. Value) is converted into each of the chromaticity coordinates indicating the appropriate color for the condition difference.
After that, when all of the one or more specific areas detected in the process of step S7 are set as the attention area and the correction for all the attention areas is completed, it is determined as YES in step S13, and the process proceeds to step S14. Proceed to
As a specific example, since the specific region of the captured image 41N shown in FIG. 3B is only the leaf portion of the deciduous tree image 43N, when the correction is completed after the leaf portion is set as the attention region, the step It is determined as YES in S13, and the process proceeds to step S14.

In step S14, the corrected image generation unit 58 combines the data of each of the one or more corrected specific regions obtained as a result of repeating the loop processing of steps S8 to S13 with the data of the captured image. Generate corrected image data.
As a specific example, the corrected data of the leaf portion of the deciduous tree image 43N shown in FIG. 3B is combined with the data of the captured image 41N, and as a result, the data of the corrected image 41F shown in FIG. (However, the data format at this stage is a color area image).

  In step S15, the corrected image generation unit 58 appropriately performs post-processing such as anti-aliasing processing, smoothing processing, and sharpening processing on the corrected image data.

In step S16, the corrected image generation unit 58 converts the corrected image data from the color area image form to the original form, that is, the color information form in the RGB space or XYZ space.
As a specific example, as shown in FIG. 3A, the data of the corrected image 41F is changed from a color area image form not shown in FIG. 3 to a color information form shown in FIG. Converted.

Thereby, the temporal change correction process ends.
In this way, the captured image data is corrected, and as a result, corrected image data is obtained in which each of the one or more specific areas has changed to a color suitable for the condition difference.
As a specific example, data of the corrected image 41F shown in FIG. 3A is obtained from the data of the captured image 41N shown in FIG. In other words, when the time elapses from the imaging time TN to the future time TF, the autumn leaves of the deciduous tree advance in the real world, and the colors of the leaves of the deciduous tree are red, yellow, brown, and the like. Therefore, the corrected image 41F includes a deciduous tree image 43F that has been corrected so that most of the leaf portions detected as the specific region are red, yellow, brown, or the like. On the other hand, the house image 42 in the captured image 41N is not detected as a specific area, assuming that the real-world house does not change in color over time. As a result, the house image 42 is included in the corrected image 41F without being corrected.

The temporal change correction process has been described above with reference to FIGS. 2 to 5.
The data of the corrected image obtained as a result of such aging correction processing can be used for various types of processing. In other words, the temporal change correction process is independently executed before and after various processes, and can be a various subroutine process.
For example, the temporal change correction process may be a subroutine process of an imaging process or a display process. Note that the imaging process refers to a series of processes until recording of captured image data. Display processing refers to a series of processing until a captured image is displayed.
In the imaging process including the temporal change correction process as a subroutine process, it is possible to record the corrected image data obtained as a result of the temporal change correction process. Further, in the display process including the temporal change correction process as a subroutine process, it is possible to display a corrected image obtained as a result of the temporal change correction process.
Hereinafter, each of the imaging process and the display process including the temporal change correction process as a subroutine process will be individually described in the order.

  FIG. 6 is a flowchart showing the flow of the imaging process to which the present invention is applied and including the temporal change correction process as a subroutine process.

  For example, when the user performs an operation for instructing imaging on the operation unit 17 (FIG. 1), the imaging process is started when the operation is performed. That is, the following processing is executed.

In step S21, the CPU 11 in FIG. 1 performs through imaging and through display.
That is, the CPU 11 continues the imaging operation by the imaging unit 16, and the data of the captured images (hereinafter referred to as “frame images” as appropriate) sequentially output from the imaging unit 16 at predetermined time intervals during that time are stored in the RAM 13 and the memory. It is temporarily stored in a memory such as the unit 19. Such a series of processing of the CPU 11 is “through imaging” here.
Further, the CPU 11 sequentially reads the data of each frame image temporarily recorded in the memory at the time of through imaging, and sequentially displays the corresponding frame image on the display unit 18. Such a series of processes of the CPU 11 is “through display” here. The frame image displayed as a through image is hereinafter referred to as a “through image”.

  In step S22, the CPU 11 sets or acquires predetermined imaging conditions.

In step S23, the CPU 11 determines whether or not the release button 41 (FIG. 1) is in a half-pressed state.
If the user has not pressed the release button 41 halfway, it is determined as NO in Step S23, and the process returns to Step S23. That is, until the user half-presses the release button 41, the determination process in step S23 is repeated, and the imaging process enters a standby state.

Thereafter, when the user half-presses the release button 41, it is determined as YES in Step S23, and the process proceeds to Step S24.
In step S24, the CPU 11 executes AF processing (Automatic Focus processing: autofocus processing) and the like based on shooting conditions and the like.

  In step S25, the CPU 11 performs a temporal change correction process (FIG. 5) on the live view image data.

In step S <b> 26, the CPU 11 causes the display unit 18 to display a through image that has been corrected by the temporal change correction process in step S <b> 25.
The display form in this case is not particularly limited, but in this embodiment, a display form as shown in FIG.

  In step S27, the CPU 11 determines whether or not the release button 41 is fully pressed.

If the user has not fully pressed the release button 41, it is determined as NO in Step S27, and the process proceeds to Step S28.
In step S28, the CPU 11 determines whether or not the release button 41 has been released.
When the user's finger or the like is released from the release button 41, it is determined as YES in Step S28, and the imaging process is ended.
On the other hand, when the user's finger or the like is not released from the release button 41, it is determined as NO in Step S28, the process is returned to Step S23, and the subsequent processes are repeated. That is, as long as the release button 41 is half-pressed, the loop processing from step S23 YES to step S28 is repeatedly executed.

Thereafter, when the user visually recognizes the corrected image changed to the desired color, for example, visually recognizes the corrected image expressing the appearance of the autumn leaves, and presses the release button 41 fully, YES is determined in step S27. If it is determined that there is, the process proceeds to step S29.
In step S29, the CPU 11 controls the imaging unit 16 to acquire captured image data. The acquired captured image data is temporarily stored in a memory such as the RAM 13 or the storage unit 19.

In step S30, the CPU 11 performs a temporal change correction process (FIG. 5) on the captured image data acquired in the process of step S29.
In this case, if the condition difference used in the immediately preceding step S25 is used, the processes in steps S1 to S5 in FIG. 5 can be omitted as appropriate.

In step S31, the CPU 11 causes the display unit 18 to display a captured image (corrected image) corrected by the temporal change correction process in step S30.
The display form in this case is not particularly limited, but in this embodiment, a display form as shown in FIG.

In step S32, the CPU 11 records the captured image data acquired in the process of step S29 and the captured image (corrected image) data corrected by the temporal change correction process in step S30 on the removable medium 31 or the like. (Hereinafter referred to as “image recording process”).
Thereby, the imaging process is completed.

The imaging process including the temporal change correction process as a subroutine process has been described above.
Next, a display process including the temporal change correction process as a subroutine process will be described.

  FIG. 7 is a flowchart showing the flow of the display process to which the present invention is applied and which includes the temporal change correction process as a subroutine process.

  For example, when the user performs an operation to instruct display of a predetermined image on the operation unit 17 (FIG. 1), the display process is started when the operation is performed. That is, the following processing is executed.

In step S41, the CPU 11 in FIG. 1 acquires data of the display image using the image instructed by the user as the display image.
Note that the display image data is not necessarily captured by the imaging apparatus 1 and may be captured or generated by another apparatus.
Therefore, the CPU 11 can acquire the data of the captured image recorded on the removable medium 31 as well as the data of another image recorded on the removable medium 31 as display image data. Further, the CPU 11 can appropriately acquire various types of image data transmitted from other devices and received by the communication unit 20 as display image data.

  In step S42, the CPU 11 causes the display unit 18 to display the display image acquired as data in the process of step S41.

In step S43, the CPU 11 determines whether or not a correction condition setting operation has been performed.
In the present embodiment, the user can set a correction condition for a display image, that is, a time condition such as a correction time, and a position condition such as a correction position by operating the operation unit 17. A specific example of the operation for setting the correction condition will be described later with reference to FIG.

  If such a correction condition setting operation has not been performed, it is determined as NO in Step S43, and the process proceeds to Step S46. Therefore, in this case, since the temporal change correction process is not executed, the original display image continues to be displayed on the display unit 18 as it is.

  In step S46, the CPU 11 determines whether an instruction to end the display process has been given.

If an instruction to end the display process is given, it is determined as YES in step S46, and the display process ends.
On the other hand, if no instruction to end the display process is given, it is determined as NO in step S46, the process returns to step S43, and the subsequent processes are repeated.
That is, when the display processing end instruction is not given, the loop processing of Steps S43NO and S46NO is repeated until the correction condition setting operation is performed, the display processing enters a standby state, and the original display image remains as it is. It continues to be displayed on the display unit 18.
Thereafter, when a correction condition setting operation is performed, it is determined as YES in Step S43, and the process proceeds to Step S44.

  In step S44, the CPU 11 performs a temporal change correction process (FIG. 5) on the display image data.

In step S45, the CPU 11 displays a display image corrected by the temporal change correction process in step S44.
The display form in this case is not particularly limited, but in this embodiment, a display form as shown in FIG.

Thereafter, the process proceeds to step S46, and the subsequent processes are repeated.
That is, the user can change the correction condition until a display image corrected to a desired color is displayed. When the correction condition is changed, it is determined as YES in the process of step S43, the display image is corrected based on the corrected correction condition in the process of step S44, and the changed image is processed in the process of step S45. A display image after correction based on the correction condition is displayed.

  Further, the display process will be specifically described with reference to FIG.

  FIG. 8 shows an example of a GUI (Graphical User Interface) image used in the temporal change correction process.

The GUI image 101 is used when executing a temporal change correction process for changing the color of leaves of a deciduous tree, accepts a time information setting operation as a correction condition, and is obtained as a result of the temporal change correction process according to the correction condition. The corrected image can be displayed.
In the GUI image 101, an image display area 111, a date and time display area 112, a correction condition change bar 113, a fall foliage display area 114, and a sample fall foliage image display area 115 are provided in this order from the top in FIG.

In the image display area 111, an image to be subjected to the temporal change correction process is displayed.
In FIG. 8, the white line that separates the images displayed in the image display area 111 is a boundary line that distinguishes the specific area from other areas, or distinguishes two specific areas.

The date and time display area 112 displays a date and time including at least the imaging time (current time) and the correction time.
Here, the shooting time is represented by a downward-pointing triangular pointer 121.

The correction condition change bar 113 indicates a time zone that can be set as the correction time.
A pointer 122 in the correction condition change bar 113 indicates the correction time.
The user operates the operation unit 17 (FIG. 1) to move the pointer 122 freely left and right in the correction condition change bar 113 while referring to the display contents of the date display area 112, for example, so that a desired date and time can be obtained. Can be set as the correction time.

In the autumn foliage display area 114, the degree of the best autumn leaves is set as a reference “0”, the numerical value before the best time is expressed as a negative value, and the numerical value after the best time is displayed as a positive value is displayed. Is done. The solid line drawn between “-7” and “+7” indicates that this is the best time to see the autumn leaves.
For example, “−28” is displayed in the autumnal best season display area 114 below “10/21” displayed in the date and time display area 112. This means that as of October 21, the best time to see the autumn leaves is “-28”, that is, the stage before the best time to see, and the color change needs to be about 28 levels later. It means that.
Therefore, when the user sets the correction time by moving the pointer 122 in the correction condition change bar 113, the user can refer to the display contents of the autumnal best season display area 114. That is, the user can easily set the best time as the correction time by moving the pointer 122 in the correction condition change bar 113 between “−7” and “+7”.

In the sample autumnal image display area 115, a leaf sample image showing the degree of autumnal leaves in a stepwise manner is displayed.
Accordingly, when the user sets the correction time by moving the pointer 122 in the correction condition change bar 113, the user can refer to the display contents of the sample autumnal image display area 115. That is, the user can easily set the desired time of autumn leaves as the correction time by moving the pointer 122 in the correction condition change bar 113 onto the sample image of the leaf of the desired color.

When the display image data is acquired in the process of step S41 of the display process in FIG. 7 in a state where the GUI image 101 is displayed as the screen of the display unit 18, the display image is displayed in the image display area 111. Is displayed.
In this case, the process of step S3 in FIG. 5, that is, the acquisition process of the imaging condition is executed at this stage, and the date and time display area 112 is located at the position indicating the imaging time specified from the acquired imaging condition. The pointer 121 is displayed.
Although not shown, the position of the pointer 122 in the correction condition change bar 113 is displayed at a default position, for example, a position below the pointer 121 indicating the imaging time.

Thereafter, when the user operates the operation unit 17 (FIG. 1) to move the pointer 122 to the position shown in FIG. 8 within the correction condition change bar 113, it is determined YES in the process of step S43, A time course correction process is executed in the process of step S44.
That is, in the process of step S5 in FIG. 5, a time corresponding to the distance between the pointer 121 and the pointer 122 shown in FIG. 8 is calculated as the condition difference time.
And the display image currently displayed on the image display area 111 of FIG. 8 is handled as a picked-up image, and some area | regions of the area | regions divided along the white line shown in FIG. 8 by the process of step S7. For example, data of a region including a deciduous tree group is detected as data of a specific region.
The loop processing of steps S8 to S13 is repeated for each of these several specific area data to correct the chromaticity coordinates of each pixel in each specific area. Thereafter, the processing of steps S14 to S16 is executed, and the data of the corrected display image (corrected image) is generated.

Then, the corrected display image is displayed in the image display area 111 of FIG. 8 in the process of step S45 of FIG.
That is, among the leaf sample images in the sample autumnal image display region 115, an image having a specific region that is close to the color of the leaf sample image below the position of the pointer 122 in FIG. 8 is displayed as a corrected display image. It is displayed in the image display area 111.
In other words, in the example of FIG. 8, “−7” within the range of the best time to see the autumn leaves is displayed in the date and time display area 112 below the position of the pointer 122. Accordingly, an image having a specific area in which the degree of best viewing of autumn leaves is a color of about “−7” is displayed in the image display area 111 as a corrected display image.

At this time, if the user determines that the corrected display image does not have a desired color, the user operates the operation unit 17 (FIG. 1) and moves the pointer 122 within the correction condition change bar 113 to an appropriate value. It is good to move to a position.
Thereby, since the correction condition is changed, it is determined as YES in the process of step S43, the display image is corrected based on the corrected correction condition in the process of step S44, and in the process of step S45, A display image after correction based on the corrected correction condition, that is, a display image closer to the user's desired color is displayed.

  The display processing using the GUI image 101 in FIG. 8 has been described above, but the GUI image 101 can be used in various other processes. For example, the GUI image 101 can be used in exactly the same way in the processes of steps S25 and S30 of the imaging process of FIG.

As described above, when the imaging device 1 according to the present embodiment holds temporal color change characteristic information indicating the temporal color change characteristics in the real world for the temporal color change target, The fourth function can be realized.
That is, the first function is a function for acquiring captured image data to be corrected. The second function is a function for acquiring the difference between the imaging time of the captured image to be corrected and the correction time as the condition difference time. The third function is a function of detecting, as a specific area, a region including a color change target with time from a correction target captured image. The fourth function is a function for correcting the data in the specific area so as to correspond to the color change occurring in the real world during the condition difference time for the color change target over time based on the time color change characteristic information. .
As a result, the following effects (1) to (3) can be obtained.

(1) For example, as the color change target with time, deciduous trees that gradually fall in the autumn in Japan, or the whole gradually changes to the flower color according to the flowering rate in the spring of Japan Trees can be employed.
Here, for example, the user has visited a predetermined place to capture a landscape such as autumn leaves and cherry blossoms, but has visited a period slightly away from the best time to see autumn leaves and cherry blossoms still in full bloom. Let's assume a case where only a landscape that has not been captured can be imaged.
Even in such a case, the user simply operates the imaging device 1 of the present embodiment to image a landscape in which the autumn leaves and cherry blossoms are not yet in full bloom. It is possible to easily obtain a corrected image in which the color of the cherry tree or cherry tree changes to the best time. In other words, the corrected image is a landscape image in which autumn leaves and cherry blossoms are in full bloom (a landscape image that matches the purpose of the user's imaging action), and is a natural and realistic landscape image. Therefore, by recording such corrected image data, it is possible to reduce or cover imaging failures (meaning that the purpose of the imaging action has not been achieved).
In this way, the captured image can be processed so as to be a more natural and realistic image that matches the purpose of the user's imaging action.

(2) Further, the correction time can be set freely.
For this reason, when the correction image corrected by the initially set correction time is not a desired landscape image for the user, the user can perform an operation of finely adjusting the correction time back and forth. Since a new corrected image is obtained each time such an operation is performed, the user can easily obtain a desired landscape image in a short time.
That is, the user can easily obtain a desired landscape image in a short time regardless of the shooting state of the captured image, for example, regardless of the imaging position or imaging time (season or date).

(3) Further, in order to set such a correction time, the imaging apparatus 1 of the present embodiment can display the GUI image 101 of FIG. 8 and accept a user operation.
This allows the user to easily recognize the relative relationship between the shooting date / time (current time), the date / time to be corrected (correction time), and the best time to see autumn leaves and flowering, and the degree of autumn leaves and flowering is easy. Visible to.
Furthermore, the user can correct the desired degree of autumn leaves and flowering by performing simple operations such as sliding the correction date / time mark (pointer 122 in the example of FIG. 8) to the left or right or back and forth on the time scale. It becomes possible to do.

  It should be noted that the present invention is not limited to the above-described embodiment, and modifications, improvements, etc. within a scope that can achieve the object of the present invention are included in the present invention.

For example, in the embodiment, the time color change characteristic information for each correction color change target is adopted, but the type of the time color change characteristic information is not particularly limited to this.
Specifically, for example, as described above, even in the same deciduous tree, the change in leaf color varies depending on various factors such as the position and layer of leaves, the amount of solar radiation, and shielding, and spatial Or, it becomes complicated that is not uniform in time.
Therefore, temporal color change characteristic information for each of a plurality of partial objects divided from predetermined temporal color change objects may be employed. For example, when the correction color change target is a forest or a tree, each of the upper layer, middle layer, and lower layer is adopted as a partial target, or each of a sunlit part and a shaded part is adopted as a partial target. May be.
FIG. 9 shows an example of the result of correction using temporal color change characteristic information for each of a plurality of partial objects divided from predetermined time-dependent color change objects.
FIG. 9A shows an example of the result of correction when the temporal color change target is a tree and each of the upper layer portion, the middle layer portion, and the lower layer portion is adopted as a partial target. The image of the top tree indicates the image of the tree included in the captured image captured at the first time. The image of the tree at the center shows the image of the tree after correction when the second time when a predetermined time has elapsed from the first time is the correction time, and each of the upper layer portion, the middle layer portion, and the lower layer portion It can be seen that the change in color (degree of autumn leaves) is different. The image of the bottom tree shows the corrected tree image when the third time after a predetermined time has passed from the second time as the correction time. Similarly, the upper layer portion and the middle layer portion It can be seen that the color changes of the lower layer and the lower layer are different.
FIG. 9B shows an example of a correction result when the color change target with time is a tree, and each of a part with a high sun, a part with a medium sun, and a part that is shaded is adopted as a part target. The image of the top tree indicates the image of the tree included in the captured image captured at the first time. The image of the tree at the center shows an image of the tree after correction when the second time when a predetermined time has elapsed from the first time is set as the correction time. It can be seen that the color change (degree of autumn leaves) of each part is different. The image of the tree at the bottom shows the corrected tree image when the third time after a predetermined time from the second time is used as the correction time. It can be seen that the change in the color of each of the sunlit part and the shaded part is different.
FIG. 9C shows an example of correction results when the color change target over time is a mountain forest (a collection of trees) and each of the upper layer portion, the middle layer portion, and the lower layer portion is adopted as a partial target. Yes. The image of the top forest is an image of the forest included in the captured image captured at the first time. The image of the forest in the center shows the image of the forest after correction when the second time after a predetermined time has elapsed from the first time as the correction time, and each of the upper layer part, the middle layer part, and the lower layer part It can be seen that the change in color (degree of autumn leaves) is different. The image of the forest at the bottom shows the image of the forest after the correction when the third time after a predetermined time from the second time is used as the correction time. Similarly, the upper and middle layers It can be seen that the color changes of the lower layer and the lower layer are different.

For example, in the above-described embodiment, the color change that occurs in the real world for the color change target with time is actually measured. However, the present invention is not particularly limited thereto, and the color state transition based on the stochastic process may be calculated. .
FIG. 10 shows an example of a color state transition based on a stochastic process showing a change in the color of a leaf when the temporal change target is a deciduous tree.
That is, FIG. 10 shows an example of a stochastic model (Markov model) that assumes that the stage of development of autumn leaves or the chromaticity of leaves changes to the next stage or chromaticity according to the Markov process.
In the example of FIG. 10, an ellipse indicates one state, and four types of states of GY (yellowish green), Y (highly saturated yellow), y (lowly saturated yellow), and yr (yellowish red) are shown. Is illustrated. The number attached to the arrow indicates the state transition probability (State Transitional Probability) when transitioning from the original state of the arrow to the state of the arrow (including when maintaining the same state).

For example, in the above embodiment, the temporal color change target in which the color in the real world changes in the time direction according to a certain rule is adopted as the temporal change target. However, a predetermined feature in the real world changes in the time direction according to the constant law. Any object can be adopted as long as it does.
Specifically, for example, if a tree space occupancy such as the size, growth, and prosperity of a tree is adopted as a predetermined feature, the predetermined feature changes in the time direction according to the law that it becomes larger or higher. I will do it. For this reason, it is also possible to employ a temporally changing object whose space occupancy in the real world changes in the time direction according to a certain law. Such a temporal change target is hereinafter referred to as a “temporal change target”.

Further, with reference to FIG. 11, the temporal change correction processing for the temporal size change target will be specifically described below.
FIG. 11 shows an example of the result of the temporal change correction process in the temporal size change target.
For example, assume that data of a captured image 81N including a house image 82 and a tree image 83N as illustrated in FIG. 11B is obtained as a result of capturing a scene including a house and a tree at time TN. Note that the apparatus that captures the captured image 81N may be the imaging apparatus 1 or another apparatus.
Here, it is assumed that the user desires to obtain an image in which the tree has grown. In such a case, the user can set a future time TF by operating the operation unit 17 (FIG. 1).
In this case, the imaging apparatus 1 obtains the difference time between the imaging time TN and the future time TF as the condition difference time.
Further, the imaging device 1 detects the tree image 83N as a specific region based on the data of the captured image 81N.
And the imaging device 1 correct | amends the data of a specific area according to the change of the space occupation rate (a magnitude | size and height) which arises in the real world during the condition difference time about a tree.
Thereby, the data of the corrected image 81F shown in FIG. 11A is obtained from the data of the captured image 81N shown in FIG. That is, when the time elapses from the time TN of imaging to the time TF, tree growth proceeds in the real world. For this reason, the corrected image 81F includes a tree image 83F having a larger space occupancy (a larger size) than the tree image 83N detected as the specific region. On the other hand, the house occupancy in the captured image 81N is not detected as a specific area, assuming that the space occupancy does not change with time for a real-world house. As a result, the house image 82 is included in the corrected image 81F as it is without being corrected.
Furthermore, the user can set not only the future time but also the past time with respect to the imaging time TN. For example, even when the past time TP is set with respect to the imaging time TN, the above-described series of processing is executed as the temporal change correction processing.
Thereby, the data of the corrected image 81P shown in FIG. 11C is obtained from the data of the captured image 81N shown in FIG.
In other words, when the time returns from the imaging time TN to the time TP (elapses in the reverse direction), the trees in the real world are in a pre-growth state, and thus the space occupancy (size and height) decreases. For this reason, the corrected image 81P includes a tree image 83P having a smaller space occupancy (smaller in size) than the tree image 83N detected as the specific region. On the other hand, the house occupancy in the captured image 81N is not detected as a specific area, assuming that the space occupancy does not change with time for a real-world house. As a result, the house image 82 is included in the corrected image 81P without being corrected.

For example, in the above-described embodiment, the correction time used in the temporal change correction process is instructed by the user operating the operation unit 17, but is not particularly limited thereto, and the imaging apparatus 1 is not mediated by the user's operation. The correction time may be set autonomously, that is, automatically.
Specifically, for example, the condition difference acquisition unit 54 (FIG. 4) or the like predicts a change stage of a predetermined feature (color or space occupancy) for the temporal change target, and sets a predetermined one of the predicted change stages. The time indicating the stage may be determined as the correction time.
For example, when a deciduous tree is adopted as a color change target over time, the condition difference acquisition unit 54 and the like can predict the progress of autumn leaves and determine the best time as a correction time.
Note that the prediction method in this case is not particularly limited. For example, the condition difference acquisition unit 54 or the like predicts the progress of autumn leaves from data on the average date in the past for each region, or actual measurement data such as monthly average temperature for each region. The best time can be determined as the correction time. In this case, a predetermined prediction formula is created in advance from the actual measurement data, and the condition difference acquisition unit 54 and the like predict the progress of the autumn leaves by performing a prediction calculation using the prediction formula, and determine the best time. The correction time may be determined.

For example, in the above-described embodiment, the correction target image data of the temporal change correction process is captured image data. However, the present invention is not limited to this, and any image data may be used as long as the temporal change target image is included. Can be corrected.
In this case, the captured image acquisition unit 51 directly acquires the captured image data output from the imaging unit 16 as a correction target, as well as image data transmitted from another device and received by the communication unit 20, or removable. Image data read from the medium 31 by the drive 21 can be appropriately acquired as a correction target.

  For example, in the above-described embodiments, the image processing apparatus to which the present invention is applied has been described as an example configured as an imaging apparatus such as a digital camera. However, the present invention is not particularly limited to the imaging device, and can be applied to any electronic device that can execute the above-described image processing regardless of the presence or absence of an imaging function. Specifically, for example, the present invention can be applied to a personal computer, a video camera, a portable navigation device, a portable game machine, and the like.

  The series of processes described above can be executed by hardware or can be executed by software.

  When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium. The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only constituted by the removable medium 31 of FIG. 1 distributed separately from the apparatus main body in order to provide the program to the user, but also in a state of being incorporated in the apparatus main body in advance. It is comprised with the recording medium etc. which are provided in this. The removable medium is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preinstalled in the apparatus main body includes, for example, the ROM 12 in FIG. 1 in which the program is recorded, the hard disk included in the storage unit 19 in FIG.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.

  DESCRIPTION OF SYMBOLS 1 ... Imaging device, 11 ... CPU, 12 ... ROM, 13 ... RAM, 16 ... Imaging part, 17 ... Operation part, 18 ... Display part, 19 ... Storage unit, 20 ... communication unit, 21 ... drive, 31 ... removable media, 51 ... captured image acquisition unit, 52 ... imaging condition acquisition unit, 53 ... correction condition acquisition unit, 54 ... Conditional difference acquisition unit, 55 ... Specific region detection unit, 56 ... Change data acquisition unit, 57 ... Specific region correction unit, 58 ... Correction image generation unit, 59 ... Time Change characteristic information determination unit, 60... Time change characteristic information storage unit

Claims (11)

In an image processing apparatus capable of capturing and reproducing an image including a specific region whose characteristics change depending on time and position,
Holding means for holding time variation characteristic information stored in advance in association with the position and time, the characteristics of the specific region;
When playing back an image including the captured specific area, the characteristic of the specific area is read from the time change characteristic information, the read characteristic is compared with the characteristic when the image is captured, and the captured image is captured. Correction means for correcting the characteristics of
Playback means for playing back an image including the corrected specific area;
An image processing apparatus comprising: The correction means includes
  The position indicated by the captured image is the current position, the position indicated by the corrected image is the correction position, and the difference between the current position and the correction position is acquired as a condition difference position.
  Furthermore, the difference acquisition means for acquiring the time indicated by the captured image as the current time, the time indicated by the corrected image as the correction time, and acquiring the difference between the current time and the correction time as the condition difference time;
  The image processing apparatus according to claim 1, further comprising: It said holding means Ri Contact holds the time change characteristic information for each of a plurality of types of the specific area,
Before SL correcting means, for each type of the specific area, based on the time change characteristic information corresponding to the type, respond to changes in raw cheat the characteristics between the condition differential time for the particular region of the type To correct the data of the specific area corresponding to the type,
The image processing apparatus according to claim 2 . The holding means holds the time variation characteristic information for each of a plurality of partial objects divided from a predetermined specific area ,
Said correcting means, said each plurality of sub-objects, based on the time change characteristic information corresponding to the partial object, so as to correspond to a change in the raw cheat the characteristics between the condition differential time for the parts subject , Correcting the data of the partial target image in the specific area,
The image processing apparatus according to claim 2 or 3.
The correction means includes
Furthermore, the change in raw cheat the characteristics between the condition differential time for the particular region, is calculated as the state transition of the feature based on stochastic processes,
On the basis of the calculated state transition, the image processing apparatus according to any one of claims 2 to 4 for correcting the data of the specific area. An operation means for receiving an operation in which the user designates the correction time as an absolute time or a relative time with respect to a predetermined reference time;
The difference acquisition means recognizes the correction time specified by the user and received by the operation means, and acquires the difference between the current time and the correction time recognized as the condition difference time.
The image processing apparatus according to claim 2. The difference acquisition means
Furthermore, the stage of change of the feature is predicted,
Determining a time indicating a predetermined stage of the predicted stages of change as a correction time, and obtaining a difference between the current time and the determined correction time as the condition difference time;
The image processing apparatus according to claim 2. The object of the specific region is a plant, and the characteristic that changes in the time direction according to a certain rule in the real world is a color of at least a part of the plant.
The image processing apparatus according to claim 1. The object of the specific region is a plant, and the characteristic that changes in the time direction according to a certain law in the real world is a space occupancy of at least a part of the plant.
The image processing apparatus according to claim 1. An image processing apparatus that can capture and reproduce an image including a specific region whose characteristics change depending on time and position ,
A holding step of holding time change characteristic information stored in advance in association with the position and time of the characteristics of the specific region;
When playing back an image including the captured specific area, the characteristic of the specific area is read from the time change characteristic information, the read characteristic is compared with the characteristic when the image is captured, and the captured image is captured. A correction step for correcting the characteristics of
A reproduction step of reproducing an image including the corrected specific area;
An image processing method including: A computer that controls an image processing apparatus that can capture and reproduce an image including a specific region whose characteristics change depending on time and position ,
A holding function for holding time-varying characteristic information stored in advance in association with the position and time of the feature of the specific region;
When playing back an image including the captured specific area, the characteristic of the specific area is read from the time change characteristic information, the read characteristic is compared with the characteristic when the image is captured, and the captured image is captured. Correction function to correct the characteristics of
A playback function for playing back an image including the corrected specific area;
A program that realizes