JP2015097322A - Imaging apparatus, control method of the same, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of easily obtaining an image of a suitable composition by automatic trimming with the orientation of each subject considered therein.SOLUTION: The image processing device comprises: trimming means to cut out a part of an image obtained by imaging; composition setting means for setting a composition for trimming by the trimming means; subject detection means capable of detecting plural subjects from the image; and subject orientation determination means for determining the facing orientation of each of the subjects detected by the subject detection means. The composition setting means sets the composition on the basis of the result determined by the subject orientation determination means.

Description

  The present invention relates to a trimming control technique for image data.

  Conventionally, when taking a picture with an imaging device such as a digital camera, the photographer shoots with an awareness of the composition, but it is particularly difficult for beginners of photography to determine a suitable composition by instantly judging the subject and scene, It takes skill to instantly determine a desirable composition that can be sympathized with many people. Therefore, there is a technique for automatically trimming according to the position and orientation of the face in the image so as to obtain a suitable composition (see Patent Document 1). However, according to this method, although the orientation of the face is taken into consideration, it is not effective when there is a distant subject.

  In addition, a technique of performing trimming according to the degree of relationship between the main subject and the sub-subject has been proposed (see Patent Document 2). However, since this method calculates and uses the degree of relationship that is the strength of the relationship between the main subject and the sub-subject, it is not effective when there is no sub-subject.

Japanese Patent No. 4869270 Japanese Patent No. 5084696

  Therefore, the present invention has been made in view of the above problems, and its purpose is to automatically perform trimming so that the situation and atmosphere of the shooting scene remain in the composition without increasing the user's work load. It is to realize a trimming technique that gives a suitable impression.

  In order to solve the above problems and achieve the object, an imaging apparatus according to the present invention includes a trimming unit that cuts out a part of an image obtained by imaging, and a composition setting that sets a composition for trimming by the trimming unit. Means, a subject detection means capable of detecting a plurality of subjects from the image, and a subject orientation determination means for judging a direction in which each subject detected by the subject detection means faces. The setting means controls to set the composition based on the result determined by the subject orientation determination means.

  According to the present invention, an image having a suitable composition can be easily obtained by automatically performing trimming in consideration of the direction of each subject.

It is a block diagram which shows the structure of the imaging device which concerns on this embodiment. It is a flowchart which shows the image imaging operation | movement by the imaging device of this embodiment. It is a flowchart which shows the composition setting process operation by the imaging device of this embodiment. It is a figure (1) for demonstrating the specific example of the composition setting process by the imaging device of this embodiment. It is FIG. (2) for demonstrating the specific example of the composition setting process by the imaging device of this embodiment. It is a figure for demonstrating the three division method. It is a figure which shows the relationship between the aspect similarity and aspect evaluation value by the imaging device of this embodiment. It is a figure which shows the relationship between the size ratio by the imaging device of this embodiment, and a pixel number evaluation value. It is a figure which shows the relationship between the space size ratio and space size evaluation value by the imaging device of this embodiment.

  Hereinafter, embodiments for carrying out the present invention will be described in detail. The embodiment described below is an example for realizing the present invention, and should be appropriately modified or changed according to the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment. Moreover, you may comprise combining suitably one part of each embodiment mentioned later.

<Device configuration>
First, the main configuration of the imaging apparatus according to the present embodiment will be described with reference to FIG.

  In FIG. 1, reference numeral 100 denotes an image processing apparatus. Reference numeral 10 denotes a photographing lens. Reference numeral 12 denotes a shutter having an aperture function. An image sensor 14 converts an optical image of a subject into an electric signal. Reference numeral 16 denotes an A / D converter that converts an analog signal output from the image sensor 14 into a digital signal.

  A timing generation circuit 18 supplies a clock signal and a control signal to the image sensor 14, the A / D converter 16, and the D / A converter 26, and is controlled by the memory control circuit 22 and the system control circuit 50. . An image processing circuit 20 performs predetermined pixel interpolation processing and color conversion processing on the data from the A / D converter 16 or the data from the memory control circuit 22. Further, the image processing circuit 20 performs predetermined calculation processing using image data generated by imaging, and supplies the calculation result to the system control circuit 50.

  Based on the calculation result, the system control circuit 50 controls the exposure control unit 40 and the distance measurement control unit 42 to perform TTL (through-the-lens) AF (autofocus) processing, AE (automatic exposure). Processing, EF (flash pre-emission) processing is realized. Further, the image processing circuit 20 also performs TTL AWB (auto white balance) processing using image data generated by imaging.

  A memory control circuit 22 controls the A / D converter 16, the timing generation circuit 18, the image processing circuit 20, the image display memory 24, the D / A converter 26, the memory 30, and the compression / decompression circuit 32. The data output from the A / D converter 16 is supplied to the memory control circuit 22 via the image processing circuit 20 or directly to the memory control circuit 22, and thereby written in the image display memory 24 or the memory 30. It is.

  Reference numeral 24 denotes an image display memory. Reference numeral 26 denotes a D / A converter. An image display unit 28 includes a TFT-LCD or the like. The display image data written in the image display memory 24 is supplied to the image display unit 28 via the D / A converter 26, whereby an image is displayed on the image display unit 28. An electronic viewfinder function can be realized by sequentially displaying images related to imaging on the image display unit 28. In addition, a playback display function and various display functions can be used by using the image display unit 28. Further, the image display unit 28 can arbitrarily turn on / off the display according to an instruction from the system control circuit 50. When the display is turned off, the power consumption of the imaging apparatus 100 can be significantly reduced. it can.

  A memory 30 stores captured still images and moving images. The memory 30 has a storage capacity sufficient to store a predetermined number of still images and a moving image for a predetermined time. By using this memory 30, a large number of images can be stored at high speed even in continuous shooting or panoramic shooting in which a plurality of still images are continuously shot. The memory 30 can also be used as a work area for the system control circuit 50.

  A compression / decompression circuit 32 compresses or decompresses image data by adaptive discrete cosine transform (ADCT) or the like. The compression / decompression circuit 32 reads an image stored in the memory 30, performs a compression process or an expansion process, and writes the processed data to the memory 30.

  An exposure control unit 40 controls the shutter 12 having an aperture function, and can perform flash light control in cooperation with the flash 48. A distance measuring control unit 42 controls focusing of the photographing lens 10. The exposure control unit 40 and the distance measurement control unit 42 are controlled by the system control circuit 50 based on a calculation result obtained by the image processing circuit 20 calculating image data generated by imaging. Thereby, TTL exposure control and distance measurement control are realized.

  A zoom control unit 44 controls zooming of the photographing lens 10.

  A barrier control unit 46 controls the operation of the barrier (protection unit) 102.

  Reference numeral 48 denotes a strobe, which has a function of projecting AF auxiliary light for brightly illuminating the subject and facilitating focusing. The strobe 48 may be a single unit, or a plurality of strobes may be provided, such as a xenon tube for flash emission and an LED for minute long-second emission, and each may be switched and controlled during main emission.

  Reference numeral 50 denotes a system control circuit that controls the entire imaging apparatus 100. A memory 52 stores constants, variables, programs, and the like for the operation of the system control circuit 50. The memory 52 also stores a program diagram used in AE. The program diagram is a table that defines the relationship between the aperture value for the exposure value and the control value for the shutter speed.

  Reference numeral 54 denotes a display unit such as a liquid crystal display device or a speaker, which displays an operation state, a message, and the like using characters, images, sounds, and the like in accordance with execution of a program by the system control circuit 50. The display unit 54 is installed in a single or a plurality of positions near the operation unit of the imaging apparatus 100 so that the display unit 54 can be visually recognized. A part of the display unit 54 is installed in the optical viewfinder 104.

  Among the display contents of the display unit 54, what is displayed on the LCD or the like includes single shot / continuous shooting display, self-timer display, compression rate display, number of recorded pixels, number of recorded pixels, number of remaining images, shutter There are speed display, aperture value display, etc. In addition, exposure compensation display, flash display, red-eye reduction display, macro shooting display, buzzer setting display, clock battery level display, battery level display, error display, information display with multiple digits, and attachment / detachment of the recording medium 200 There are status display, communication I / F operation display, date / time display, and the like. Among the display contents of the display unit 54, what is displayed in the optical viewfinder 104 includes in-focus display, camera shake warning display, flash charge display, shutter speed display, aperture value display, exposure correction display, and the like.

  Reference numeral 56 denotes an electrically erasable / recordable non-volatile memory, such as an EEPROM.

  Reference numerals 60, 62, 64, and 66 are operation units for the user to instruct the system control circuit 50 to perform various operations. For example, a switch or a dial, a touch panel, pointing by line-of-sight detection, a voice recognition device, or a combination of one or more Consists of. More specifically, 60 is a mode dial switch, which is turned off, automatic exposure mode such as program AE mode, fully automatic mode, manual exposure mode, sleeping face shooting mode, panoramic shooting mode, playback mode, multi-screen playback / erase Each mode such as a mode and a PC connection mode can be switched.

  Reference numeral 62 denotes a shutter switch SW1, which is turned on during operation of a shutter button (not shown), and performs AF (auto focus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, EF (flash pre-flash) processing, and the like. Instruct the start of operation.

  A shutter switch SW2 64 is turned on when the operation of the shutter button is completed, and an exposure process of writing image data into the memory 30 via the A / D converter 16 and the memory control circuit 22 with the signal read from the image sensor 14 being performed. Start operation. Further, a series of development processes using operations in the image processing circuit 20 and the memory control circuit 22, a recording process in which the image data is read from the memory 30, compressed by the compression / decompression circuit 32, and written to the recording medium 200. The operation start of the process is instructed.

  An operation unit 66 includes various buttons and a touch panel. The operation unit 66 includes a menu button, a set button, a macro button, an image display on / off button, a multi-screen playback page break button, a flash setting button, a single shooting / continuous shooting / self-timer switching button, menu movement + (plus) Button, menu shift- (minus) button, playback image shift + (plus) button, playback image- (minus) button, shooting image quality selection button, exposure correction button, date / time setting button, and the like.

  Reference numeral 80 denotes a power supply control unit, which includes a battery detection circuit, a DC-DC converter, a switch circuit that switches blocks to be energized, and the like. The power supply control unit 80 detects the presence / absence of a battery, the type of battery, and the remaining battery level, controls the DC-DC converter based on the detection result and the instruction of the system control circuit 50, and requires the necessary voltage. It is supplied to each part including the recording medium for a period. 82 and 84 are connectors. A power source 86 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like.

  Reference numerals 90 and 94 denote interfaces with recording media such as memory cards and hard disks. Reference numerals 92 and 96 denote connectors for connecting to a recording medium such as a memory card or a hard disk. A recording medium attachment / detachment detection unit 98 detects whether or not the recording medium 200 or 210 is attached to the connectors 92 and 96.

  In the present embodiment, it is assumed that there are two interfaces and connectors for attaching the recording medium. Of course, the interface and the connector for attaching the recording medium may have a single or a plurality of systems, any number of systems. Moreover, it is good also as a structure provided with combining the interface and connector of a different standard. The interface and the connector may be configured using a PCMCIA card, a CF (Compact Flash (registered trademark)) card, or the like based on a standard.

  In addition, when the interfaces 90 and 94 and the connectors 92 and 96 are configured using a PCMCIA card, a CF (Compact Flash (registered trademark)) card, or the like, a LAN card, a modem card, a USB card, IEEE1394. Image data and management information attached to image data are transferred to and from peripheral devices such as other computers and printers by connecting various communication cards such as cards, P1284 cards, SCSI cards, and PHS communication cards. I can meet each other.

  Reference numeral 102 denotes protection means that is a barrier that prevents the imaging unit from being soiled or damaged by covering the imaging unit including the photographing lens 10 of the imaging device 100.

  Reference numeral 104 denotes an optical viewfinder, which can take an image using only the optical viewfinder without using the electronic viewfinder function of the image display unit 28. In the optical viewfinder 104, some functions of the display unit 54, for example, a focus display, a camera shake warning display, a flash charge display, a shutter speed display, an aperture value display, an exposure correction display, and the like are installed.

  A communication unit 110 has various communication functions such as RS232C, USB, IEEE1394, P1284, SCSI, modem, LAN, and wireless communication.

  Reference numeral 112 denotes a connector for connecting the imaging apparatus 100 to another device by the communication unit 110 or an antenna in the case of wireless communication.

  Reference numeral 200 denotes a recording medium such as a memory card or a hard disk. The recording medium 200 includes a recording unit 202 composed of a semiconductor memory, a magnetic disk, and the like, an interface 204 with the imaging device 100, and a connector 206 that connects to the imaging device 100.

  Reference numeral 210 denotes a recording medium such as a memory card or a hard disk. The recording medium 210 includes a recording unit 212 composed of a semiconductor memory, a magnetic disk, and the like, an interface 214 with the imaging device 100, and a connector 216 that connects to the imaging device 100.

<Description of operation>
An image capturing operation by the imaging apparatus of the present embodiment will be described. The processing described here is realized by the system control circuit 50 developing and executing a program recorded in the nonvolatile memory 56 in a work area of the volatile memory 52 such as a RAM.

  The system control circuit 50 performs various shooting preparation operations when the power supply 86 is turned on in the mode in which the mode dial 60 can shoot, and starts displaying a through image on the image display unit 28 when the shooting preparation is completed. Here, the through image is an image captured by the image sensor 14 for display as a finder function before and after taking a still image. In the through image display state, the data sequentially written to the image display memory 24 via the image sensor 14, the A / D converter 16, the image processing circuit 20, and the memory control circuit 22 are stored in the memory control circuit 22 and the D / A. The images are sequentially displayed on the image display unit 28 via the converter 26. Thereby, an electronic viewfinder (EVF) function is realized.

  Next, an image capturing operation in the through image display state will be described with reference to FIG.

  In FIG. 2, S201 is repeated until the shutter switch SW1 is pressed. If the shutter switch SW1 is pressed, the process proceeds to S202.

  The system control circuit 50 performs a distance measurement process to focus the photographing lens 10 on the subject, performs a photometry process, and determines an aperture value and a shutter time (S202).

  If the shutter switch SW1 is released without pressing the shutter switch SW2, the process returns to S201 (S203). When the shutter switch SW2 is pressed (S204), the system control circuit 50 stores the memory directly via the image sensor 14, the A / D converter 16, the image processing circuit 20, the memory control circuit 22, or directly from the A / D converter. Through the control circuit 22, exposure processing for writing image data taken in the memory 30, and the image data written in the memory 30 are read out by using the memory control circuit 22 and the image processing circuit 20 as necessary. A photographing process including a developing process is executed (S205).

  Subsequently, composition setting for trimming the image data input in the image processing circuit 20 is performed (S206).

  Here, the composition setting will be described with reference to FIG. In FIG. 3, a subject is detected from the input image data (S301). If the subject is a person, the region of the subject can be detected by detecting the face. A known method may be used for face detection. For example, the image processing circuit 20 extracts a shape corresponding to the constituent elements of the face area such as the eyes, nose and mouth from the input image data, and calculates the face area from the existing positional relationship of the eyes, nose and mouth. . A known method may be used for subject detection when the subject is not a person. For example, the input image data is divided into regions based on color and brightness, and subject-like weight is weighted according to the size of the divided regions and the distance from the screen center. Then, an area where the weight is equal to or greater than a certain value is set as a subject area.

  Next, it is determined whether or not the subject has been detected (S302). If the subject can be detected, the orientation information of the subject is acquired (S303), and the composition is set using the subject orientation information (S304). If the subject has not been detected in S302, the composition is set without using the subject information (S304).

  The method for acquiring the orientation information of the subject in S303 may be a known method. For example, in the time-sequential input image data, the same subject detected in S301 is compared with the immediately preceding frame, and if the same subject continues to move in a certain direction, the advancing direction is simply set as the subject. And the direction. Even if the subject is not moving, if the face can be detected, the face direction is calculated from the ratio of the size of the left and right eyes or the positional relationship of the eyes, nose, mouth, etc. The direction of the line of sight may be calculated from the relationship between the white and black eyes of the subject's eyes, and the direction of the line of sight may be used as the direction of the subject. Furthermore, when the detected face area is being tracked and the face changes to an oblique face or a side face, and the eyes, nose, and mouth cannot be detected, the subject may be facing back.

  Subsequently, a composition is set (S304). The composition setting will be described with reference to FIGS. 4 and 5. FIG. 4 and 5, four scenes (a), (b), (c), and (d) will be described as an example. Each scene has a subject, and the subjects are indicated by T1 to T10. Furthermore, the directions of the subjects are indicated by D1 to D9. Only T10 is a subject whose direction could not be detected, and D10 does not exist. FIG. 5 shows examples of compositions in the image data of FIG. 4 from W1 to W4, respectively. W1 to W4 are merely examples of the composition, and the composition to be set may not be one.

  First, in the scene (a), since the subjects T1 and T2 are facing in opposite directions, the composition is made different. Then, consider the cutout position. In general, in photography, a composition with a wide space in the direction in which the subject is facing is considered good. In the scene (a), since there is no subject to be cut out together with T1, the three-division method is applied. As shown in FIG. 6, the three-division method is performed at four intersections P1 to P4 of dividing lines obtained by drawing two lines so as to divide each of the horizontal and vertical directions of the field angle to be divided into three. If an important element such as the center of gravity of the subject is placed, a balanced and stable composition can be obtained. W1 is an example in which a space is widened in the direction in which the subject is facing and the three-division method is applied. Similarly, when considering the cutout of T2, T2 is directed further outward at the angle of view end. In such a case, it is impossible to make a wide space in the direction in which the subject is facing. Therefore, it is not necessary to forcibly set the composition.

  Next, in the scene of (b), the subjects T3 and T4 are facing back. When the subject is looking backward and the subject is not particularly found in that direction, the subject may be related to the background and the subject, such as looking at the scenery. Therefore, there is a method for setting the composition so that the background is wide. W2 is an example of a composition including subjects T3 and T4 facing backward and a wide background. In addition, the composition may include only the subject T3 or only T4 so that the background is wide.

  In the scene (c), since the subjects T5 and T6 face each other, there is a high possibility that they are related to each other. Therefore, a composition such as W3 is set so that both subjects fall within the angle of view. In this scene, control is performed so that one of the subjects is not cut off at the edge of the angle of view by being conscious of the three-division method.

  Finally, in the scene (d), the subjects T7, T8, and T9 are facing the direction of the subject T10, and T7, T8, and T9 may be considered as related subjects through T10. W4 is an example of a composition including all subjects related to T10. Of course, a composition including only T10 to which the three-division method is applied, or a composition including only T8 and T9 and T10 among subjects facing the direction of T10 may be set. In (d), T10 is used for the sake of clarity, but there are scenes in which the subject of T10 cannot be detected. Even in such a case, if the orientation of a plurality of subjects intersects in a specific region, it is assumed that there is a subject in that region, and a plurality of subjects facing that region are included. A composition may be set.

  The example of setting the composition has been described above using the scenes (a), (b), (c), and (d). By using these methods, it is possible to set the composition while maintaining the relationship between the subject and the background in (a) and (b) and the subjects in (c) and (d).

  If the subject has not been detected in S301, or if the subject has been detected but the orientation cannot be detected in S303, the composition may be set without using the subject information, or the composition may not be forcibly set. May be. For example, if the subject can be detected but the orientation of the subject cannot be detected, the orientation may be simply applied without being concerned about the orientation. Furthermore, as long as no subject is found, for example, the composition may be simply set around the focal point.

  Subsequently, trimming of the set composition is performed (S207). The system control unit 50 reads out the captured image data written in the memory 30, performs trimming of the image data by the image processing circuit 20 according to the composition set in S206, and writes the trimmed image into the memory 30. (S207). At this time, the composition to be trimmed may be the entire composition set in S206, or 1. The order in which there are many subjects in the composition. 2. Composition in which space is widened in the direction the subject is facing. The priority order and the number of sheets to be trimmed may be determined as in the composition to which the three-division method is applied, and the trimming may be performed up to that number.

Further, all the set compositions may be displayed on the image display unit 28, and only the composition selected by the user may be trimmed. The system control circuit 50 reads the captured image data written in the memory 30, performs various image processing using the memory control circuit 22 and the image processing circuit 20 as necessary, and uses the compression / decompression circuit 32. After performing an image compression process according to the set mode, a recording process for writing image data to the recording medium 200 or 210 is executed (S208).

As described above, according to the present invention, it is possible to detect the subject and the orientation of the subject from the image data obtained by shooting and automatically determine an appropriate trimming range according to the orientation of the subject. It is possible to easily obtain a suitable trimmed image without bothering.

<Modification>
A modification of the above-described embodiment will be described. The difference from the above-described embodiment is the process of S207. In the modified example, the evaluation value is used to determine the composition to be actually trimmed from all the compositions set in S206. In order to determine the composition to be extracted, it is necessary to determine the aspect, number of pixels, coordinates, etc. of the composition to be extracted. When determining these, the evaluation values are calculated, for example, from 0 to 100, and finally the respective evaluation values are combined to determine the composition in descending order of the total evaluation value.

  As an example, the aspect evaluation value, the pixel number evaluation value, the subject position evaluation value, the space size evaluation value, and the subject relationship evaluation value are calculated from the subject information and the composition information set, and all the evaluation values are averaged. To calculate the final evaluation value. The final evaluation value may be multiplied by a random value so that it does not become the same value every time, or the evaluation value may vary depending on the type of subject, such as whether or not the subject is a person. Furthermore, depending on the scene and subject, which evaluation value is used to calculate the final evaluation value may be changed.

  An example of calculating each evaluation value is shown below. The values that appear here are merely examples, and the parameters may be adaptively changed according to the scene or subject.

  The aspect evaluation value is a method of calculating according to, for example, the aspect similarity between the subject and the composition. If the subject is vertically long, the evaluation value of the vertically long aspect is set high. The vertical and horizontal sizes of the subject create a minimum rectangular area that encompasses the subject area, and the vertical and horizontal sizes are calculated from this rectangle. When there are a plurality of subjects, a minimum rectangular region that includes each subject region is created, a minimum rectangular region that encompasses all the subject regions is created, and vertical and horizontal sizes may be calculated from the rectangles. The aspect evaluation value is calculated from the aspect similarity calculated by the following formula.

  FIG. 7 shows an example of calculating the aspect evaluation value corresponding to the aspect similarity.

  For example, there is a method of calculating the pixel number evaluation value according to the proportion of the subject in the cut-out composition. When the size of the subject in the cut-out composition is too large or too small, the evaluation value becomes small. To do. The pixel number evaluation value is calculated from the size ratio calculated by the following equation.

  FIG. 8 shows an example of calculating the pixel number evaluation value corresponding to the size ratio.

  For example, there is a method of calculating the subject position evaluation value according to the center of gravity of the subject in the set composition. The evaluation value is 100 when the subject's center of gravity satisfies the three-division method. A simple value such as 50 may be used.

  There is a method of calculating the space size evaluation value according to the space where there is no subject and the direction in which the subject is facing, comparing the size of the space in the direction where the subject is facing from the center of gravity of the subject and the size of the opposite space, The evaluation value should be increased when the ratio is appropriate. The space size evaluation value is calculated from the space size ratio calculated by the following equation.

  FIG. 9 shows a calculation example of the space size evaluation value corresponding to the space size ratio.

  For example, when there are a plurality of subjects in the set composition, the evaluation value is set to 100 when there is a relationship between the subjects and the evaluation value is set to 30 when there is no relationship. If a plurality of subjects cannot be detected, the subject-related evaluation value need not be used.

  By calculating the evaluation value of each composition set in this way and setting the threshold value of the evaluation value necessary for the upper rank or the composition to be trimmed, the number of trimmings can be reduced, and trimming data is wasted. The increase can be avoided and the burden on the user can be reduced.

<Other embodiments>
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads and executes the program code. It is processing to do. In this case, the program and the storage medium storing the program constitute the present invention.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 10 Shooting lens 12 Shutter 14 Image sensor 16 A / D converter

Claims (10)

Trimming means for cutting out a part of an image obtained by imaging;
Composition setting means for setting a composition for trimming by the trimming means;
Subject detection means capable of detecting a plurality of subjects from the image;
Subject orientation determination means for determining the direction in which each subject detected by the subject detection means is facing;
With
The image processing apparatus, wherein the composition setting means sets the composition based on a result determined by the subject orientation determination means. The image processing apparatus according to claim 1, wherein the number of compositions set by the composition setting unit may be any number. The image processing apparatus according to claim 2, wherein the trimming unit does not necessarily trim all the compositions even when a plurality of compositions are set by the composition setting unit. 4. The subject direction determination unit according to claim 1, wherein the subject direction determination unit determines the direction of the subject based on at least one of a traveling direction of the subject, a traveling direction of the entire image, or a viewing direction of the subject. The image processing apparatus according to any one of the above. 5. The composition setting unit according to claim 1, wherein the composition setting unit does not have to set the composition when the subject direction determination unit cannot determine the direction of the subject. 6. Image processing device. 5. The composition setting unit according to claim 1, wherein the composition setting unit sets the composition so that a space in a direction in which each subject detected by the subject detection unit faces is widened. The image processing apparatus described. 5. The composition setting unit according to claim 1, wherein the composition setting unit sets the composition so that the background is wide when the subject detected by the subject detection unit is turned away. The image processing apparatus described. The composition setting means determines whether the subject is related to another subject based on the result determined by the subject orientation determination means, and sets the composition so that the related subject is included. The image processing apparatus according to any one of claims 1 to 4. The trimming unit may trim any number of the compositions set by the composition setting unit according to a subject or a scene, or may not perform trimming. Item 9. The image processing device according to any one of Items 8 to 9. Composition evaluation value calculation means for calculating an evaluation value for each composition according to the position and orientation of the subject in the composition set by the composition setting means,
10. The trimming unit according to claim 1, wherein the trimming unit determines a priority order of the composition to be trimmed based on the evaluation value calculated by the composition evaluation value calculating unit. Image processing device.