JP2013197980A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide: an image processing apparatus, an image processing method and a program with which components of an image can be suitably arranged and converted according to a composition.SOLUTION: An image processing method comprises: extracting from image data one or a plurality of object regions and a characteristic portion in a background (steps S11-S13); setting a composition in an image display region according to the image data on the basis of the extracted object and characteristic portion in the background (step S14); arranging at least one of the object region and the background in the image display region on the basis of the set composition (steps S15-S18); and outputting image data in which the object region and the background are arranged (step S19).

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  Conventionally, various compositions suitable for images such as photographs and paintings are known. However, it is not always easy to photograph with an appropriate composition when photographing with the imaging device. Therefore, in recent years, a technique for evaluating a composition based on a through image at the time of imaging by an imaging apparatus has been disclosed (Patent Document 1). In addition, a technique is disclosed in which a plurality of images and objects are arranged and combined based on a composition to generate one piece of image data (Patent Document 2).

  On the other hand, as a technique related to the invention of the present application, in recent years, there is a technique for estimating a defective portion of an image based on surrounding image data and performing a reproduction restoration (Non-Patent Document 1).

JP 2011-135527 A JP 2011-2875 A

M. Bertalmio et al., “Image Inpainting”, Proc. ACM SIGGRAPH2000, p.417-424, 2000.

  However, if the arrangement of components such as the subject in the image once taken and drawn is out of the proper composition, if there is only one subject, it can be handled by trimming etc. If there is a problem in the positional relationship or the relative position between the subject and the background, there is a problem that it cannot be corrected.

  An object of the present invention is to provide an image processing apparatus, an image processing method, and a program capable of appropriately arranging and converting the constituent elements of an image according to the composition.

In order to achieve the above object, the present invention
Subject extraction means for extracting one or a plurality of subject areas and feature portions in the background from the image data;
Composition setting means for setting the composition in the image display area related to the image data based on the subject area and the background feature portion extracted by the subject extraction means;
Subject placement control means for placing at least one of the subject area and the background in the image display area based on the composition set by the composition setting means;
Image output means for outputting image data in which the subject area and the background are arranged under the control of the subject placement control means;
An image processing apparatus comprising:

  According to the present invention, there is an effect that the components of the image can be appropriately rearranged according to the composition.

1 is a block diagram illustrating an internal configuration of an imaging apparatus according to an embodiment of the present invention. It is a flowchart which shows the process sequence of the image edit process in an imaging device. FIG. 3 is a first diagram illustrating an example of editing imaging data. FIG. 3 is a first diagram illustrating an example of editing imaging data. FIG. 3 is a first diagram illustrating an example of editing imaging data. FIG. 2 is a second diagram illustrating an example of editing imaging data. FIG. 2 is a second diagram illustrating an example of editing imaging data. FIG. 3 is a third diagram illustrating an example of editing imaging data. FIG. 3 is a third diagram illustrating an example of editing imaging data.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing an internal configuration of an imaging apparatus which is an image processing apparatus of the present invention.

  The imaging device 100 is a digital imaging device such as a digital camera. The imaging apparatus 100 includes a lens unit 11, an electronic imaging unit 12, an imaging control unit 13, an image data generation unit 14, an image memory 15, an image processing unit 16, a display control unit 17, and an image output unit. Display unit 18, an operation unit 19 as an operation means, a recording medium control unit 20, a central control unit 21, and the like. Among these, the imaging control unit 13, the image processing unit 16, and the central control unit 21 can be arranged on one custom LSI chip 100a.

  The lens unit 11 includes a plurality of lenses and is configured to be able to perform operations such as zooming and focusing. These zoom operations and focus operations may be performed by the user manually operating a housing that holds the lens, or provided with a zoom drive unit and a focus drive unit, either automatically or based on user operation input It may be operable.

  The electronic imaging unit 12 includes, for example, an image sensor such as a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS). The electronic imaging unit 12 measures the amount of light related to the optical image that has passed through each lens of the lens unit 11 for each pixel arranged in a two-dimensional matrix, converts it into an image signal, and outputs it.

  The imaging control unit 13 includes a timing generator (not shown), a driver, and the like. The imaging control unit 13 sends a control signal to the zoom driving unit and the focus driving unit of the lens unit 11 to operate, thereby adjusting the focal length and the focus position. In addition, the imaging control unit 13 drives the electronic imaging unit 12 with a timing generator and a driver to acquire light amount data of a time corresponding to the shutter speed (exposure time) set in the electronic imaging unit 12, The image signal is output to the image data generation unit 14.

  The image data generation unit 14 performs analog / digital conversion on the light amount data of each pixel output from the electronic imaging unit 12 to generate and output digital image data. At this time, the image data generation unit 14 can perform color development adjustment, balance adjustment, γ correction, and the like set in advance for each of the RGB colors. The output image data is DMA-transferred to the image memory 15 via a DMA (Direct Memory Access) controller (not shown).

  The image memory 15 is, for example, a DRAM (Dynamic Random Access Memory), and temporarily stores the digital image data transferred from the image data generation unit 14. The image memory 15 temporarily stores processing data from the image processing unit 16 and the central control unit 21 during various image processing described later.

  The image processing unit 16 performs various image processing according to the present invention on the generated digital image data. The image processing unit 16 includes a feature region detection unit 16a (subject extraction unit), an arrangement setting unit 16b (composition setting unit, subject arrangement control unit), and an image restoration unit 16c (image restoration unit).

  The feature area detection unit 16a identifies a subject included in the image display area and a feature area in the background. The subject here is taken as a foreground, and includes, for example, the human face, upper body, and whole body, but other things such as non-human organisms, plants, and various structures may also be used. good. There is no need for one subject, and a plurality of subjects can be set. In addition, examples of the background feature area include mountains, coastlines, horizons, and distant buildings. Even in the case of a landscape photograph or the like, a mountain or a building can be appropriately set as a subject. The feature area detection unit 16a holds information related to the feature pattern for detecting these subjects and feature areas. The information related to the feature pattern includes, for example, specific contrast information and color distribution. The feature area detection unit 16a performs a match determination with a characteristic pattern by comparing a value obtained by adding the degree of match with these pieces of information with a predetermined weight with a predetermined reference value.

  The arrangement setting unit 16b selects an appropriate composition similar to this based on the size, shape, and arrangement of the feature area of the subject and background detected by the feature area detection unit 16a, and the appropriate setting of each subject and background. The position is identified based on the composition. The arrangement setting unit 16b refers to the composition setting table 21b related to the setting of various compositions used for selection of the composition held in the ROM or the like of the central control unit 21 as necessary.

The image restoration unit 16c restores the image at the pixel position where the image data is lost as a result of the relative movement of each subject and the background. Various known techniques relating to image restoration (Image Inpainting) can be used to repair the image data deficient area. Here, in these known image restoration techniques, the amount of calculation and the amount of memory used often increase. Therefore, the image restoration unit 16c includes a communication unit that communicates with an external server, and transmits the image data and information related to the image restoration area to the external server to perform restoration processing. It may be configured to receive from a server.
The operation of these image processing units 16 will be described in detail later.

  The display control unit 17 performs control to read out the image data stored in the image memory 15 and display it on the display unit 18 based on a control signal from the central control unit 21. The display control unit 17 displays image data that has been captured and acquired based on a user operation and an image related to image data that has been subjected to various edits. Further, the display control unit 17 can cause the display unit 18 to display a through image (live view image) acquired at a predetermined sampling period and a REC view image related to the shooting automatically immediately after shooting.

  The display unit 18 is, for example, a liquid crystal display unit (LCD). In the display unit 18, the drive driver of the liquid crystal display screen operates based on the control signal input from the display control unit 17, and displays a captured image, an edited image, a through image, a rec view image, and the like on the display screen.

  The recording medium control unit 20 is configured such that the recording medium 20 a is detachable, and controls data exchange between the loaded recording medium 20 a and the image memory 15. The recording medium control unit 20 writes the image data (RAW data) stored in the image memory 15 via the image processing unit 16 and the image data subjected to the encoding / compression processing (for example, JPEG) to the recording medium 20a. I can do it.

  The recording medium 20a is a portable recording medium including a nonvolatile memory, and is, for example, a flash memory or an EEPROM (Electrically Erasable and Programmable Read Only Memory).

  The operation unit 19 includes one or a plurality of operation switches. As the operation switch, for example, a push button switch, a rotation switch, or a touch panel using the display unit 18 is used. When the user operates these operation switches, the operation unit 19 converts the operation input into an electric signal and outputs it to the central control unit 21. Specifically, the operation unit 19 receives an instruction button for instructing selection of an imaging mode or a function according to a menu screen displayed on the display unit 18 and an instruction for adjusting the zoom amount. Such a zoom button is provided.

  The central control unit 21 performs various arithmetic processes and controls each unit of the imaging apparatus 100 in an integrated manner. The central control unit 21 includes a CPU (Central Processing Unit), a RAM, and a ROM (Read Only Memory), and calls and executes various processing programs 21a stored in the ROM as necessary, and performs imaging based on the programs 21a. Each part of the apparatus 100 is controlled. The ROM also includes a composition setting table 21b (composition storage means) described later.

  Next, the process of the picked-up image in the imaging device 100 of this embodiment is demonstrated.

  The imaging apparatus 100 according to the present embodiment has a function of correcting one piece of image data acquired by a shooting operation so as to match a predetermined composition.

FIG. 2 is a flowchart illustrating a control procedure by the CPU for image editing processing in the imaging apparatus 100 according to the present embodiment.
First, when the image editing process is started, the CPU acquires image data from the image memory 15 (step S11). The CPU identifies digital image data selected as an editing target based on a user operation or the like.

  Next, the CPU causes the feature region detection unit 16a to perform a process of extracting a subject from the acquired image data (step S12). The feature area detection unit 16a calls various stored feature patterns and detects portions in the image that match these feature patterns by a predetermined reference level or more. Here, the feature region detection unit 16a detects, for example, a human face, and specifies other parts of the body connected to the face based on the detected face, so that various human body shapes, postures, etc. Corresponding to Similarly, it is possible to detect the entire plant based on the shape of the flower. Further, the detection process related to the unnecessary feature pattern may be omitted by setting the type of the subject in advance by the user. The feature area detection unit 16a extracts all subjects detected for all areas in the image, and stores the position information, type, and the like of these subjects in the image memory 15.

Subsequently, the CPU causes the feature region detection unit 16a to extract the feature portion of the background portion (step S13). The feature area detection unit 16a sets the remaining part of the image data excluding the area set as the subject as a background image, and detects the feature part from the background image. As this characteristic part, a thing that can be detected as a subject in a distant view such as a building, a structure, or the top of a mountain, or a thing that forms a boundary such as a horizon, a horizon, a road, a railway, or a pillar or a fence is preferably selected. Is done.
In the process of step S12, even if it is initially set as a subject, if it is recognized as a background part due to a relative positional relationship with other subjects, the setting information is changed and identified. It can be used as a feature region of the background portion. In addition, regarding whether to set a distant view part as a subject or a feature part of a background image, for example, it is possible to acquire focus information from meta information related to image data and add it to the determination material. .

The CPU causes the arrangement setting unit 16b to select an appropriate composition based on the information of all the extracted subject and background feature portions, that is, the arrangement, shape, size, and the like (step S14). The composition stored in the arrangement setting unit 16b and made selectable includes, for example, a horizontal composition, a vertical composition, a diagonal composition, a diagonal composition, a radiation composition, a curved composition, an S composition, a triangular composition, an inverted triangular composition, and a contrast. In addition to well-known representative compositions such as a symmetric composition, a tunnel composition, and a pattern composition, various other compositions can also be included. Further, the information related to these compositions is not limited to information stored in advance in the arrangement setting unit 16b. For example, by storing other composition information in the recording medium 20 a and attaching it to the recording medium control unit 20, it can be acquired from the outside of the imaging apparatus 100. In addition, for example, it is possible to select a plurality of combinations that can be combined, such as a combination of a horizontal line composition and a contrast / symmetric composition. For example, the CPU can match the composition information with each detected subject and the background feature and select a composition having a high similarity as a whole. In addition, the CPU may select a combination of parts having a high degree of similarity, or may adapt a subject having a low degree of similarity or a characteristic part of the background to a composition having a part having a high degree of similarity. The composition can be selected by determining whether it is possible.
Alternatively, as a result of the matching, the CPU extracts a plurality of compositions or combinations of compositions that satisfy a condition that the degree of similarity as a whole or partially exceeds a predetermined level and causes the display unit 18 to display the composition and the combination. The selection may be made through an operation input to the operation unit 19.

  In addition to the composition selection, the CPU also selects information related to the screen configuration. The screen configuration here includes, for example, information related to a screen division method (division ratio) and subject arrangement symmetry. As specific division ratios, there are three divisions, golden divisions, divisions by root ratio (for example, silver ratio 1: √2), and the like. The CPU may further store and select area ratio information, fractal dimension information, color distribution information, color redundancy information, hue information, and the like as other information related to the screen configuration.

  When one or more compositions are selected, the CPU causes the arrangement setting unit 16b to perform processing for setting the arrangement of the subject and the background according to the composition based on the selected composition settings (step S15). In the process of setting the arrangement, the arrangement setting unit 16b can perform each process of translation, rotation, and enlargement / reduction of each subject and background. The same effect as trimming can be obtained by enlarging the background. The final change of the image size itself may be executed during the editing process, or may be performed separately.

  At this time, not all subjects and backgrounds can be set to optimal positions. For example, a case where the overlapping (contact) portions of a plurality of subjects in the original image cannot be separated, or a case where inconvenient duplication occurs when each subject is placed at an optimal position, is conceivable. In addition, due to restrictions related to the image restoration technique, it is necessary to move each subject and background within a range in which image restoration that is difficult to estimate from surrounding pixel data is not performed as much as possible. Therefore, the arrangement setting unit 16b calculates an evaluation value related to the deviation from the optimum position related to the selected composition, and the evaluation values of each subject and background after arrangement are all or a predetermined number in descending order of importance. / Search for an arrangement that can satisfy a predetermined reference value set in advance at a predetermined ratio, and perform the arrangement.

  In addition, when a plurality of compositions or composition combinations can be selected, the arrangement setting unit 16b performs possible rearrangement for each composition or composition combination, and the higher evaluation value is finally selected. It is good also as composition and arrangement.

  When the arrangement of each subject and background is set according to the composition, the CPU moves at least one of the subject and background to the set position (step S16). Then, the CPU specifies an image defect portion where an image is missing as a result of this movement (step S17).

The CPU sends a command to the image restoration unit 16c to perform a defective portion repair process for repairing the identified defective portion (step S18). As described above, the image restoration unit 16c transmits, to the external server, the image data in which the subject and the background have been moved using the communication unit and the information related to the missing portion, and performs various arithmetic processes related to the restoration by the external server. It is good to make it happen.
Here, in the restoration technique related to image restoration (Image Inpainting), the defective portion is repaired by internal interpolation or extrapolation using various weighting and interpolation methods based on pixel data around the defective portion. For example, each pixel data is handled as numerical data based on luminance (brightness), and continuity is taken into account, that is, optimal restoration data is obtained by minimizing the numerical change (finding the minimum of the numerical gradient). Can do. Alternatively, the degree of similarity with the neighboring pixel data can be evaluated, and the restoration target pixel data can be estimated and obtained by weighted averaging or the like based on the distance from the restoration target pixel data to each of the surrounding pixels and the luminance data of the pixel. .

  In such image restoration, if an attempt is made to repair a missing portion between a relatively moved background portion and a subject portion as it is, an unnatural continuous restoration image is obtained between the subject and the background. Will be. Accordingly, the image restoration unit 16c cuts out the image data of each subject portion and temporarily stores it together with the moved position data, and after repairing the missing portion including the subject portion cut out based only on the background image, the subject portion The process can be advanced to overwrite the image data.

  When the correction and restoration of the image is completed, the CPU writes the image data into the image memory 15, and controls the display control unit 17 as necessary to display it as the edited image data on the display unit 18 (step S19). ). Then, the CPU ends the image editing process. Instead of being displayed on the display unit 18 as edited image data, an edited image based on the edited image data may be printed out from the printing unit.

Next, the operation content will be described with reference to a specific example of image editing.
3 to 5 are views showing an image editing example 1 by this image editing process.

  First, the image shown in FIG. 3A is acquired (step S11). Subsequently, yachts are extracted as the subjects O11 to O13, respectively (step S12). Further, as shown in FIG. 3B, the lake shore and the horizon are extracted as the background feature portion B11 (step S13).

  Next, the yacht as the identified subject and the layout of the lake shore and the horizon as the characteristic features of the background are confirmed, and an optimum composition is selected (step S14). Here, as shown in FIG. 4A, the horizontal division is performed on the horizon, and the left subject O11 and the right subjects O12 and O13 are each set to a root ratio of 1: √2 (white silver ratio) with respect to the central vertical line. ) Position.

  Once the background and subject arrangement are determined, the moving direction and moving distance of these image portions are then calculated (step S15). Here, as shown in FIG. 4B, the left subject O11 is moved to the upper right (arrow M11), the right subjects O12 and O13 are moved substantially to the left (arrow M12), and the background features B11 is moved to the lower left (arrow M13) (step S16). Then, an image defect portion when these relative movements are performed is specified (step S17). As shown in FIG. 5A, first, a partial defect I11 occurs based on the relative movement within the image area. Further, due to the movement of the background, a defect E11 occurs in the outer frame portion corresponding to the cut image portion R11. These defects I11 and E11 are repaired based on the surrounding background image data (step S18). In this case, the missing portion I11 in which surrounding pixel data exists is repaired by internal interpolation, and the missing portion E11 corresponding to the outer frame portion is interpolated by extrapolation.

  Finally, each moved subject is overwritten on the restored background image. As a result, as shown in FIG. 5B, an image corrected based on the selected composition and repaired with the missing portion due to relative movement is completed and output (step S19).

  6 and 7 are diagrams showing an image editing example 2 by image editing processing.

  First, the image data shown in FIG. 6A is acquired (step S11). As shown in FIG. 6B, subjects O21 to O25 and background feature portions B21 and B22 are extracted from the image. (Steps S12 and S13).

  Then, a composition is selected based on these subjects and the background (step S14). Here, the composition of the central road is a radiation composition, and the background feature portion B22 is located in the center and forms an appropriate vertical dividing line, so there is no need to move the background.

  Next, the subjects O21 to O25 are moved based on the composition (steps S15 and S16). The persons of the subjects O21 and O22 are separately identified, but these subjects are connected and cannot be divided and moved. Therefore, O21 and O22 are moved to a position closest to the optimal arrangement as a whole. Further, the right subjects O21 and O22 are too close to the center with respect to the vertical dividing line, whereas the left subjects O23 to O25 are too close to the left end as a whole, resulting in an unbalanced arrangement. . Therefore, as shown in FIG. 7A, the subjects O21 to O24 are translated to the right so as to conform to the composition of the route ratio 1: √2 (arrow M20).

  These subjects O21 to 25 are appropriately arranged in the vertical direction and are not moved in the vertical direction. For example, even if the positions of the subjects O21 and O22 are below the ideal position. It is not preferable to translate these subjects upward. The subjects O21 and O22 are photographed only halfway through the legs in the original image data, and it is difficult to restore the newly appearing leg tips even if they are translated upward. Therefore, for example, processing such as enlarging the subjects O21 and O22 while the lower end position of the leg is fixed is performed.

  When the movement destination of the subject is set, the missing portion I20 accompanying the movement is specified (step S17), and these missing portions are repaired by internal interpolation (step S18). Then, the corrected images shown in FIG. 7B are obtained by overwriting the subjects O21 to O25 at the set positions (step S19).

  8 and 9 are diagrams showing an image editing example 3 by image editing processing.

  First, the image data shown in FIG. 8A is acquired (step S11), and then the subject O31 and the background feature portions B31 and B32 are extracted as shown in FIG. 8B (step S12, S13).

  Next, a composition is selected based on the detected subject O31 and background feature portions B31 and B32 (step S14). Here, a horizontal line is formed by the horizontal line of the mountain foot that is the background characteristic portion B31, and a vertical line is formed by the person of the subject O31, and these are arranged in a three-part composition. That is, as shown in FIG. 9A, the background feature portion B31 indicating a horizontal line is arranged so as to coincide with the upper horizontal dividing line, and the face portion of the subject O31 is arranged on the line. The subject O31 is arranged on the left vertical dividing line, and the top of the mountain related to the background feature B32 is arranged on the right vertical dividing line.

  Next, the amount of movement is identified based on the set background feature and the position of the subject (step S15). First, as shown by an arrow M31, by moving the background to the upper right so that the upper region R31 protrudes from the frame of the image, the tops of the background feature portion B31 and the background feature portion B32 that form a horizontal line. The position of is moved. Next, a change amount of the subject O31 is set. Here, if the person of the subject O31 is simply moved in the lower left direction, the lower half of the photographed body is interrupted and the subject balance is deteriorated. (Arrow M32) (step S16).

  Due to these movements, image defect portions E30 and I30 are generated (step S17). Accordingly, the missing portion E30 is repaired by extrapolation, and the missing portion I30 is repaired by internal interpolation (step S18). Then, the corrected image is formed and output by reducing and arranging the subject O31 at the set position (FIG. 9B, step S19).

  As described above, the imaging apparatus 100 according to the present embodiment includes the image processing unit 16, and one or a plurality of subject regions and a feature portion in the background are identified and identified from the image data by the feature region detection unit 16a. An appropriate composition is selected based on the arrangement, size, and shape of the subject area and the characteristic part, and an appropriate arrangement of the subject area and the background is determined by the arrangement setting unit 16b according to the selected composition, and rearrangement is performed. The output image is output. Therefore, even if part or all of the subject area and the background are out of the proper arrangement in the composition, the captured image data can be easily converted into an image rearranged in the appropriate composition.

  In addition, as a result of the rearrangement of each subject area and feature portion, when each portion is relatively moved and a missing portion of the image is generated, the missing portion is estimated and repaired based on surrounding pixel information. It is possible to generate image data that has been naturally subjected to composition conversion.

  Further, when rearranging the subject area and the background, parallel movement, rotational movement, and / or enlargement / reduction deformation can be performed in an appropriate combination, so that the imaging apparatus 100 is inclined at the time of shooting, Appropriate adjustment can be performed even when the subject size is not well balanced.

  Various composition settings used for composition selection are preset in the ROM composition setting table 21b, and an appropriate composition or composition combination is easily selected by the arrangement setting unit 16b while referring to the ROM. be able to.

  Further, when setting the composition, the user can select a favorite one from among a plurality extracted by the arrangement setting unit 16b by an operation input to the operation unit 19. It is divided into multiple background parts (for example, the sea and the sky) where the part that has been captured with the same weight as is adjusted by the user's settings, or which is to be weighted differently depending on the shooting intention In some cases, the intended composition can be selected.

  Further, it is possible to perform editing so as not to produce an unnatural image by detecting and judging a case where separation movement is impossible because a plurality of subjects are in contact with each other, and enabling rearrangement collectively.

  In addition, some of the subjects that are cut across the periphery of the image are difficult to repair naturally because the part not photographed outside the periphery, so the subject should not be added newly. By adding a condition for rearrangement of the image, editing can be performed so that an unnatural subject image does not occur.

  In addition, by setting the subject and the background to be arranged within a range that satisfies a predetermined reference level with respect to the selected composition, it is difficult to make the arrangement in which the captured image data perfectly fits the composition. Even in such a case, it can be moved naturally to a preferred arrangement.

  In the image restoration, after performing the restoration process based only on the background image, each subject is arranged on the restored background image according to the composition. By performing the restoration in this way, it is possible to avoid an unnatural restoration by an internal interpolation between the background and the subject.

The present invention is not limited to the above-described embodiment, and various modifications can be made.
For example, in the above-described embodiment, the captured image by the imaging apparatus has been described. However, the image editing process according to the present invention can also be applied to digital image data drawn or captured on a computer.

  In the above-described embodiment, an example in which the imaging apparatus 100 performs processing related to image editing, or transmission to an external server using a communication unit to perform processing has been described. It is also possible to perform processing relating to the rearrangement and image restoration separately from the imaging device. By performing the editing process on the image data copied to an external computer using the recording medium 20a or the like using another computer, the user can comfortably execute these processing instructions using an input device such as a keyboard. You can view the edited image on a large display. In addition, it is possible to perform processing at high speed and comfortably by utilizing the capabilities of the memory and CPU of the computer.

  Further, in the above-described embodiment, the image of the defective portion is repaired. However, for example, an image that is not repaired may be output so that the change history of the arrangement can be understood for learning purposes. .

  In the above embodiment, image output to the display unit 18 has been described as an example. However, output to a printer or the like using a cable or the like is possible, and a portable recording medium or a network cable is used. You can also output data to an external device.

In the above description, an example in which a ROM is used as a computer-readable medium of the program 21a according to the present invention is disclosed, but the present invention is not limited to this. As other computer-readable media, non-volatile memories such as flash memory and EEPROM (Electrically Erasable and Programmable Read Only Memory), and portable recording media such as CD-ROM can be applied. A carrier wave is also applied to the present invention as a medium for providing program data according to the present invention via a communication line.
In addition, details such as the specific configuration, order, and numerical values shown in the description of the above embodiments can be changed as appropriate without departing from the spirit of the present invention.

Although several embodiments of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and equivalents thereof.
The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.

[Appendix]
<Claim 1>
Subject extraction means for extracting one or a plurality of subject areas and feature portions in the background from the image data;
Composition setting means for setting the composition in the image display area related to the image data based on the subject area and the background feature portion extracted by the subject extraction means;
Subject placement control means for placing at least one of the subject area and the background in the image display area based on the composition set by the composition setting means;
Image output means for outputting image data in which the subject area and the background are arranged under the control of the subject placement control means;
An image processing apparatus comprising:
<Claim 2>
Restoration is performed by estimating an image data missing area generated in the image display area due to the movement of the subject area and / or the background by the subject placement control means based on image data around the image data missing area. The image processing apparatus according to claim 1, further comprising an image restoration unit.
<Claim 3>
3. The image processing according to claim 1, wherein the subject arrangement control unit causes each of the subject region and / or the background to perform at least one of parallel movement, rotational movement, and enlargement / reduction deformation. apparatus.
<Claim 4>
Comprising composition storage means for storing information relating to a plurality of compositions;
The composition setting unit selects a composition based on matching between the composition stored in the composition storage unit and the subject and the background feature portion initially arranged in the image display area. The image processing apparatus according to claim 1.
<Claim 5>
Comprising an operation means for detecting and acquiring a user operation input;
The composition setting means extracts a plurality of compositions and / or composition combinations in which the degree of similarity related to the matching satisfies a predetermined condition from the composition storage means, and extracts the composition based on the operation input information acquired by the operation means 5. The image processing apparatus according to claim 4, wherein one of the composition and / or a combination of compositions is selected.
<Claim 6>
The subject placement control means determines whether or not the plurality of extracted subjects cannot be separated and placed separately. If the separation is not possible, the subject placement is collectively performed on the subject. The image processing apparatus according to claim 1, wherein the image processing apparatus is arranged in an image display area.
<Claim 7>
When the boundary of the one or a plurality of subject areas overlaps with the boundary of the image display area, the subject placement control means is a part of the boundary of the subject area that overlaps with the boundary of the image display area The image processing apparatus according to claim 1, wherein the subject area is arranged so as not to move inside the image display area.
<Claim 8>
The subject placement control means sets the placement so that all the subjects and the background are shifted from an optimal placement based on the composition in which the subject is set and within a predetermined reference range. The image processing apparatus according to any one of 1 to 7.
<Claim 9>
3. The image restoration unit, after performing the restoration based on the background image data, overwrites and updates the image data of the subject area on the restored background image data. The image processing apparatus according to claim 8.
<Claim 10>
An image processing method for performing image data editing processing,
A subject extraction step for extracting one or a plurality of subject regions and feature portions in the background from the image data;
A composition setting step for setting a composition in the image display area related to the image data based on the subject and background feature portions extracted in the subject extraction step;
A subject placement control step of placing at least one of the subject region and the background in the image display region based on the composition set in the composition setting step;
as well as,
An image output step for outputting image data in which the subject region and the background are arranged in the subject placement control step;
An image processing method comprising:
<Claim 11>
Computer
Subject extraction means for extracting one or a plurality of subject areas and feature portions in the background from image data;
Composition setting means for selecting a composition in the image display area according to the image data based on the subject area and background feature portion extracted by the subject extraction means;
Subject placement control means for placing at least one of the subject area and the background in the image display area based on the composition set by the composition setting means;
as well as,
Image output means for outputting image data in which the subject area and the background are arranged under the control of the subject placement control means;
Program to function as.

DESCRIPTION OF SYMBOLS 11 Lens part 12 Electronic imaging part 13 Imaging control part 14 Image data generation part 15 Image memory 16 Image processing part 16a Feature area detection part 16b Arrangement setting part 16c Image restoration part 17 Display control part 18 Display part 19 Operation part 20 Recording medium control Unit 20a recording medium 21 central control unit 21a program 21b composition setting table 100 imaging device 100a chip

Claims (11)

Subject extraction means for extracting one or a plurality of subject areas and feature portions in the background from the image data;
Composition setting means for setting the composition in the image display area related to the image data based on the subject area and the background feature portion extracted by the subject extraction means;
Subject placement control means for placing at least one of the subject area and the background in the image display area based on the composition set by the composition setting means;
Image output means for outputting image data in which the subject area and the background are arranged under the control of the subject placement control means;
An image processing apparatus comprising:   Restoration is performed by estimating an image data missing area generated in the image display area due to the movement of the subject area and / or the background by the subject placement control means based on image data around the image data missing area. The image processing apparatus according to claim 1, further comprising an image restoration unit.   3. The image processing according to claim 1, wherein the subject arrangement control unit causes each of the subject region and / or the background to perform at least one of parallel movement, rotational movement, and enlargement / reduction deformation. apparatus. Comprising composition storage means for storing information relating to a plurality of compositions;
The composition setting unit selects a composition based on matching between the composition stored in the composition storage unit and the subject and the background feature portion initially arranged in the image display area. The image processing apparatus according to claim 1. Comprising an operation means for detecting and acquiring a user operation input;
The composition setting means extracts a plurality of compositions and / or composition combinations in which the degree of similarity related to the matching satisfies a predetermined condition from the composition storage means, and extracts the composition based on the operation input information acquired by the operation means 5. The image processing apparatus according to claim 4, wherein one of the composition and / or a combination of compositions is selected.   The subject placement control means determines whether or not the plurality of extracted subjects cannot be separated and placed separately. If the separation is not possible, the subject placement is collectively performed on the subject. The image processing apparatus according to claim 1, wherein the image processing apparatus is arranged in an image display area.   When the boundary of the one or a plurality of subject areas overlaps with the boundary of the image display area, the subject placement control means is a part of the boundary of the subject area that overlaps with the boundary of the image display area The image processing apparatus according to claim 1, wherein the subject area is arranged so as not to move inside the image display area.   The subject placement control means sets the placement so that all the subjects and the background are shifted from an optimal placement based on the composition in which the subject is set and within a predetermined reference range. The image processing apparatus according to any one of 1 to 7.   3. The image restoration unit, after performing the restoration based on the background image data, overwrites and updates the image data of the subject area on the restored background image data. The image processing apparatus according to claim 8. An image processing method for performing image data editing processing,
A subject extraction step for extracting one or a plurality of subject regions and feature portions in the background from the image data;
A composition setting step for setting a composition in the image display area related to the image data based on the subject and background feature portions extracted in the subject extraction step;
A subject placement control step of placing at least one of the subject region and the background in the image display region based on the composition set in the composition setting step;
as well as,
An image output step for outputting image data in which the subject region and the background are arranged in the subject placement control step;
An image processing method comprising: Computer
Subject extraction means for extracting one or a plurality of subject areas and feature portions in the background from image data;
Composition setting means for selecting a composition in the image display area according to the image data based on the subject area and background feature portion extracted by the subject extraction means;
Subject placement control means for placing at least one of the subject area and the background in the image display area based on the composition set by the composition setting means;
as well as,
Image output means for outputting image data in which the subject area and the background are arranged under the control of the subject placement control means;
Program to function as.

Images