JP5718502B2 - Image creating apparatus and image creating method - Google Patents

Description

  The present invention relates to an image creating apparatus for creating an image and an image creating method therefor.

  In recent years, various types of photographing apparatuses such as a digital still camera and a mobile phone with a photographing function have a moving image creation function. However, it is difficult to capture a moving image desired by the user using such an imaging device. This is because generally moving images are taken for a relatively long time. In addition, when there are a plurality of subjects that the user wants to shoot in the shooting scene, and shooting is performed while paying attention to only one of the subjects, the orientation of the shooting device is changed each time the subject of interest is changed. Need to change. On the other hand, if the angle of view is widened so that all the subjects are photographed, the size of each subject in the image becomes relatively small.

  Patent Document 1 proposes a technique for displaying an entire image and images of faces of individual subjects so that they can be viewed simultaneously. In Patent Document 1, an image of a face part in an input image is detected, and the detected face part image is enlarged, and then the original whole image and the face image are combined and displayed. Yes. With such a technique, the user can easily grasp the correspondence between the person in the entire image and the face of each person.

JP 2008-262416 A

  In the technique of Patent Document 1, a person's face image is enlarged and displayed one by one. Even if a plurality of persons are mixed in a narrow image area in the entire image, The face image is enlarged and displayed for each person. Here, in an image in which a plurality of persons are present simultaneously in a narrow image area, it is considered that there is some relationship between the persons. Therefore, it is desirable that these persons can be simultaneously displayed in the same image area so that the relationship can be understood. In addition, when there are a plurality of such areas, it is considered that changing the size of each image area according to the number of persons in the image area results in a moving image that is easy for the user to see.

  The present invention has been made in view of the above circumstances, and in the case where an image area in which a plurality of human subjects are present exists in the entire image, the size of the image area according to the number of persons in each image area. It is an object of the present invention to provide an image creating apparatus capable of creating an image with a modified image and an image creating method therefor.

  In order to achieve the above object, an image creating apparatus according to a first aspect of the present invention detects a display unit that displays an image and a face included in an image area in the image displayed on the display unit. Generated by the face detection unit, a face image generation unit that generates a face image including the face detected by the face detection unit for each image region, and the face image generation unit according to the number of faces included in the image region. In addition, an enlargement factor determination unit that determines the enlargement factor of each face image, and a face image that is enlarged according to the enlargement factor determined by the enlargement factor determination unit in the background image corresponding to the screen size of the display unit. And an image creating unit that creates a synthesized composite image. When the enlarged face image is selected, the enlarged face image is recorded.

  In order to achieve the above object, an image creating apparatus according to a second aspect of the present invention detects a display unit that displays an image and a face included in an image area in the image displayed on the display unit. Generated by the face detection unit, a face image generation unit that generates a face image including the face detected by the face detection unit for each image region, and the face image generation unit according to the number of faces included in the image region. In addition, an enlargement factor determination unit that determines the enlargement factor of each face image, and a face image that is enlarged according to the enlargement factor determined by the enlargement factor determination unit in the background image corresponding to the screen size of the display unit. An image creating unit for creating a synthesized image, and recording the synthesized image in accordance with a recording instruction.

  In order to achieve the above object, an image creating method according to a third aspect of the present invention is a method in which a face included in an image region in an image displayed on a display unit is detected by a face detection unit, A face image including a face detected by the step is generated for each image area, and the enlargement ratio of each face image generated by the trimming control unit is determined according to the number of faces included in the image area, and the screen of the display unit is determined. In the background image corresponding to the size, a composite image is created by synthesizing the face image enlarged in accordance with the determined enlargement ratio, and when the image area is selected, the selected image is recorded. It is characterized by performing.

  In order to achieve the above object, an image creating method according to a fourth aspect of the present invention is such that a face included in an image region in an image displayed on a display unit is detected by a face detection unit, and the face detection unit A face image including a face detected by the step is generated for each image area, and the enlargement ratio of each face image generated by the trimming control unit is determined according to the number of faces included in the image area, and the screen of the display unit is determined. A synthesized image obtained by synthesizing a face image enlarged in accordance with the determined enlargement ratio is created in a background image corresponding to the size, and the synthesized image is recorded in accordance with a recording instruction.

  According to the present invention, when there is an image area in which a plurality of human subjects are present in the entire image, an image capable of creating an image in which the size of the image area is changed according to the number of persons in each image area A creation apparatus and an image creation method therefor can be provided.

It is a figure which shows the structure of the digital camera as an example of the moving image production apparatus which concerns on one Embodiment of this invention. It is a figure which shows the outline | summary of operation | movement of the digital camera in one Embodiment of this invention. It is a flowchart which shows the moving image imaging operation | movement of the digital camera in one Embodiment of this invention. It is a figure for demonstrating the position of a face image. It is a flowchart shown about a designated positional relationship determination process. It is a figure for demonstrating the designated positional relationship determination process. It is a flowchart shown about an enlargement rate determination process. It is a figure for demonstrating an enlargement rate determination process. FIG. 9A is a diagram when the enlargement ratio is determined according to the number of all faces in the face image, and FIG. 9B is an enlargement ratio determined only according to the number of faces arranged in the horizontal direction in the face image. FIG. It is a flowchart shown about the modification of an enlargement rate determination process. It is a figure for demonstrating the modification of an enlargement rate determination process. 10 is a flowchart illustrating a partially enlarged through image display process. It is a figure for demonstrating a margin part. It is a figure which shows the example of the synthetic | combination method of a whole image.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram illustrating a configuration of a digital camera (hereinafter simply referred to as a camera) as an example of a moving image creation apparatus according to an embodiment of the present invention. 1 includes a control unit 101, an imaging unit 102, a face detection unit 103, a display unit 104, a position designation unit 105, an operation unit 106, a recording unit 107, and a clock unit 108. Have.

  The control unit 101 is a control circuit that comprehensively controls the operation of each block of the camera 100. The control unit 101 performs operation control of the imaging unit 102, the face detection unit 103, the display unit 104, and the like according to the operation of the position specifying unit 105 and the operation unit 106 by the user. In addition, the control unit 101 includes a trimming control unit 1011, a trimming image composition control unit 1012, and an image processing unit 1013.

  A trimming control unit 1011 having a function as a face image generating unit performs trimming control for generating a composite moving image to be described later. In other words, the trimming control unit 1011 determines a trimming range that is a predetermined image area corresponding to each of a plurality of points designated by the user in the image displayed on the display unit 104, and images within each determined trimming range. The image is trimmed from the image that has been subjected to the image processing for through image display or the image obtained through the image sensor 102c, to generate a face image. Note that the “face image” in the present embodiment does not necessarily indicate an image in which one person's face is shown in the image. When a plurality of faces exist in the trimming range, an image including the plurality of faces is also a “face image” in the present embodiment.

  A trimming image composition control unit 1012 having functions as a designated positional relationship determination unit, an enlargement ratio determination unit, and a moving image creation unit is configured to display the face image trimmed by the trimming control unit 1011 and the image displayed on the display unit 104. Control is performed for synthesizing the whole image showing the whole to generate one synthesized moving image. That is, the trimming image composition control unit 1012 determines the composition position of each face image from the positional relationship of each face image, and determines the enlargement ratio of each face image from the number of faces in each face image. The trimming image composition control unit 1012 controls the image processing unit 1013 to create a composite moving image of the face image and the entire image.

  The image processing unit 1013 performs image processing on the image obtained by the imaging unit 102. This image processing includes, for example, color correction processing, gradation correction processing, enlargement / reduction processing, compression processing (still image compression or moving image compression) for an image obtained in the imaging unit 102, expansion processing for a compressed image, and the like. It is. The image processing unit 1013 also has a function as a moving image creation unit together with the trimming image synthesis control unit 1012, and creates a synthesized moving image according to the control of the trimming image synthesis control unit 1012. Further, the image processing unit 1013 has a temporary recording unit 1013a for recording various data such as the position of the trimming range. The temporary recording unit 1013a is configured by a RAM, for example.

  The imaging unit 102 includes a lens 102a, a diaphragm 102b, an imaging element 102c, an A / D converter (ADC) 102d, and the like. The lens 102a is an optical system for forming an image of the light flux from the subject on the photoelectric conversion surface of the image sensor 102c. Here, the lens 102a in the present embodiment is an optical system including a zoom lens, and the focal length is made variable by driving the zoom lens. It is possible to adjust the angle of view by changing the focal length of the lens 102a. The stop 102b limits the amount of light incident on the photoelectric conversion surface of the image sensor 102c. The image sensor 102c has a photoelectric conversion surface in which pixels are two-dimensionally arranged, and converts a light beam incident through a lens into an electric signal (image signal). The ADC 102d converts an analog image signal obtained by the image sensor 102c into digital image data (hereinafter simply referred to as an image). The imaging unit 102 having such a configuration captures a subject under the control of the control unit 101 and acquires an image related to the subject.

  The face detection unit 103 detects a face in the image obtained by the imaging unit 102. The face detection unit 103 detects a face by detecting shadows of face parts such as eyes, nose, and mouth from the image. The face detection result by the face detection unit 103 is used for lens focus adjustment in the imaging unit 102. Further, the face detection unit 103 in the present embodiment also has a function as the line-of-sight detection unit 1031. That is, the gaze detection unit 1031 of the face detection unit 103 detects the gaze of the subject by detecting the orientation of the face and the positional relationship between the black eyes and the white eyes.

  The display unit 104 is a display unit such as a liquid crystal display (LCD) or an organic electroluminescence display (ELD) provided on the back surface of the camera 100, for example. The display unit 104 displays various images such as an image obtained by the imaging unit 102 under the control of the control unit 101.

  The position designation unit 105 is a touch panel configured integrally with the display unit 104, for example, and is an operation unit for designating an arbitrary position on the display screen of the display unit 104. The position specifying unit 105 detects contact of the user's finger or the like with the display screen of the display unit 104 and outputs a signal corresponding to the detected position to the control unit 101. The control unit 101 recognizes the position on the display screen of the display unit 104 that has been touched by the user's finger or the like by identifying the signal from the position specifying unit 105. Note that the control unit 101 recognizes from the output of the position specifying unit 105 that the user's finger or the like has been slid when detecting a continuous contact position change.

  The operation unit 106 is various operation units other than the position specifying unit 105. The operation unit 106 includes a power button, a release button, a zoom switch, a mode dial, and the like. The power button is an operation unit for instructing to turn on or off the power of the camera 100. The release button is an operation unit for instructing photographing with the camera 100 (acquisition of an image for recording). The zoom switch is an operation unit for instructing zoom driving of the lens of the imaging unit 102. The mode dial is an operation unit for switching the operation mode of the camera 100 to a shooting mode (still image, moving image) or a playback mode.

  The recording unit 107 records an image for recording obtained by the imaging unit 102 and compressed by the control unit 101. The recording unit 107 is a memory card that is detachably attached to the camera 100, for example. The clock unit 108 measures various times such as shooting date and time. By associating the shooting date and time with, for example, the header information portion of the image recorded in the recording unit 107, it becomes easier to manage the image recorded in the recording unit 107.

  FIG. 2 is a diagram showing an outline of the operation of the camera 100 in the present embodiment. The subsequent operation is executed when the shooting mode of the camera 100 is switched to, for example, the moving image shooting mode.

  First, when executing the shooting operation by the camera 100, the user sets the camera 100 so that the subject that he / she wants to shoot falls within the angle of view of the lens 102a, as shown in FIG. . The example of FIG. 2A shows a case where the user performs shooting in a situation where a large number of people gather, such as a wedding or a panel discussion. In such a situation, the user has a first subject composed of one person (for example, a person who greets in the case of a wedding or a moderator in the case of a panel discussion) 200a and a second subject composed of a plurality of persons ( For example, there are cases where it is desired to photograph both the bride and groom in the case of a wedding and the debate in the case of a panel discussion. In this case, the user determines the installation position of the camera 100 according to the present embodiment so that both the first subject 200a and the second subject 200b are within the angle of view A of the lens 102a. The angle of view is adjusted (in the case of FIG. 2A, the zoom lens is adjusted to the wide angle side).

  In the camera 100, a live view display (also referred to as a live view display) is executed after the power is turned on. In the through image display, the image acquisition by the imaging unit 102 is executed at a certain timing. Each time an image is acquired in the imaging unit 102, image processing (tone correction processing, color correction processing, reduction processing, etc.) is performed in the image processing unit 1013 of the control unit 101. Accordingly, the size (number of pixels) of the image acquired by the imaging unit 102 is matched with the screen size (number of display pixels) of the display unit 104. Then, the image processed image is displayed on the display unit 104 under the control of the control unit 101. With such a through image display, the user can observe the subject using the display unit 104.

  When the camera 100 is set as shown in FIG. 2A, the through image display of the camera 100 is started, and an image within the angle of view of the lens 102a is displayed on the display unit 104 as shown in FIG. Is done. In a state where the through image display as shown in FIG. 2B is being performed, the user touches, for example, the finger 301 at an arbitrary position on the screen of the display unit 104. As described above, when there is a contact with a certain position on the screen of the display unit 104, a signal corresponding to the position is input from the position specifying unit 105 to the control unit 101. When the control unit 101 recognizes contact with the screen of the display unit 104, the face detection unit 103 recognizes the face of the person at the contact position. When a person's face is recognized at this position, a frame display 104a indicating an image area including the person's face is displayed. In addition, when a through image is displayed, an enlargement start button 104c and a recording button 104k are also displayed. When the user touches the recording button 104k with the finger 301 or the like, normal moving image shooting (processing for recording an image within the angle of view shown in FIG. 2B as a moving image) is performed.

  After the selection as shown in FIG. 2B is made, the user further brings the finger 301 into contact with another position on the screen of the display unit 104 as shown in FIG. Also in this case, when the control unit 101 recognizes contact with the screen of the display unit 104, a frame display indicating an image region including the face of the person at the contact position is made. Here, FIG. 2C shows an example in which the user selects a plurality of persons by sliding the finger 301 in the arrow B direction. When a plurality of human faces are recognized as shown in FIG. 2C, a frame display 104b indicating an image area including all the human faces is displayed.

  Thereafter, the user brings the finger 301 into contact with the enlargement start button 104c. When the control unit 101 recognizes the contact of the finger 301 or the like to the position of the enlargement start button 104c, a partially enlarged through image display is executed. In this partially enlarged through image display, a face image is generated by trimming an image in a trimming area which is an image area including a position designated by the user. The generated enlarged image of each face image and the entire image are combined with a predetermined background image having a size corresponding to the screen size of the display unit 104 to generate a combined image. A live view is displayed.

  FIG. 2D shows an example of a partially enlarged through image display. Here, the display size of the enlarged image 104d and the display size of the enlarged image 104e are changed in accordance with the number of faces (persons) in each image. Also, the combined position of each enlarged image corresponds to the positional relationship of each face in the entire image displayed on the display unit 104. For example, the face image 104a corresponding to the enlarged image 104d shown in FIG. 2D exists at the upper left of the screen of the display unit 104, and the number of faces in the face image 104a is one. Further, the face image 104b corresponding to the enlarged image 104e exists at the upper right of the screen of the display unit 104, and the number of faces in the face image 104b is two. In this case, the enlarged image 104d is combined with the upper left portion of the background image, and the enlarged image 104e is combined with the upper right portion of the background image. Further, the ratio between the size of the enlarged image 104d and the size of the enlarged image 104e is set to 1: 2, which is the same as the ratio of the number of faces. In addition, the entire image 104f is reduced and combined with the remaining portion generated by combining the enlarged images 104d and 104e with the background image in this way. The synthesized moving image generated in this way is displayed as a through image.

  In FIG. 2D, the horizontal direction of the face image, which is a direction parallel to the long side of the display unit 104, is enlarged so as to coincide with the ratio of the number of faces. The vertical length of the face image that is parallel to the short side is enlarged at a constant magnification. Of course, enlargement may be performed so that both the horizontal and vertical lengths coincide with the ratio of the number of faces.

  By performing a partially enlarged live view display as shown in FIG. 2 (d), it is possible to extract and enlarge only the person actually interacting in a wedding or a panel discussion. It is possible to observe in detail the facial expressions of each person actually communicating in a panel discussion or the like. Further, since the size of the enlarged image increases as the number of faces in the face image increases, even if the number of faces in the face image increases, the expression of each face can be easily observed. In addition, it is possible to roughly grasp the positional relationship of the person based on the position of each enlarged image, and it is also possible to grasp the positional relationship of the person in detail based on the entire image.

  Here, when displaying a partially enlarged through image shown in FIG. 2D, an enlargement end button 104g and a recording button 104h are also displayed. When the control unit 101 recognizes the contact of the finger 301 or the like to the position of the enlargement end button 104g, the partially enlarged through image display is ended and the original through image display is restored. When the control unit 101 recognizes the contact of the finger 301 or the like to the position of the recording button 104h, a synthesized moving image in which the enlarged images 104d and 104e and the entire image 104f are arranged as shown in FIG. The moving image is compressed by the image processing unit 1013 of the control unit 101 and then recorded in the recording unit 107.

  2D, when the control unit 101 recognizes the contact of the finger 301 or the like to the position of the enlarged image 104d or the enlarged image 104e during the partial enlarged through image display shown in FIG. 2D, the enlarged image at that position is displayed in FIG. As shown in (e), a full-screen through image is further displayed. Here, during full-screen through image display shown in FIG. 2E, a return button 104i and a record button 104j are also displayed. When the control unit 101 recognizes the contact of the finger 301 or the like to the position of the return button 104i, the entire through image display is terminated and the original partially enlarged through image display is restored. When the control unit 101 recognizes the contact of the finger 301 or the like to the position of the recording button 104j, the enlarged image currently displayed on the display unit 104 is compressed by the image processing unit 1013 of the control unit 101, and then the recording unit 107 is recorded.

  Next, a specific processing flow of the moving image creation method by the camera 100 will be described with reference to FIG. FIG. 3 is a flowchart showing the moving image shooting operation of the camera 100.

  The operation of FIG. 3 is started when the operation mode of the camera 100 is set to the moving image shooting mode. First, the control unit 101 executes through image display (step S101). It should be noted that an enlargement start button 104c and a recording button 104k are also displayed during the live view display. As described above, when the user touches the recording button 104k with the finger 301 or the like, normal moving image shooting is performed. Here, a detailed description of normal movie shooting is omitted.

  After the through image display, the control unit 101 sets a parameter n (n = 1, 2, 3,...) Indicating the number of face images designated by the user to an initial value 1 (step S102). Thereafter, the control unit 101 determines from the output of the position specifying unit 105 whether or not the user has specified the position of the display unit 104 within the screen (step S103). The position designation here includes designation of a plurality of positions by a slide operation. In the determination in step S103, when the position is designated, the control unit 101 detects the face included in the position designated by the user by the face detection unit 103. Then, the position (Xn, Yn) of the face image Fn composed of the image area including the detected face is recorded in the temporary recording unit 1013a. Further, a frame indicating the face image Fn is displayed on the display unit 104 as shown in FIG. 2B or 2C, for example (step S104).

  Here, the position (Xn, Yn) is a predetermined position (for example, the lower left corner) of the display unit 104 (0, 0), and from this (0, 0), the X and Y axes are set in the horizontal and vertical directions. The relative position when set. The position (Xn, Yn) when there is only one detected face is the center position of the face (center position when the face is regarded as a circle), for example, as shown in FIG. And As described above, a plurality of faces are detected when the user performs a slide operation. The position (Xn, Yn) when such a plurality of faces are detected is, for example, a position obtained by averaging a plurality of designated face positions. In addition, as shown in FIG. 4B, the position of the center of gravity of the rectangular image area that circumscribes each face may be set as the position (Xn, Yn).

  After the position recording and frame display of the face image Fn, the control unit 101 adds 1 to the parameter n (step S105), and then performs the determination in step S103 again. Also, in the determination in step S103, if the position is not designated, the control unit 101 indicates that the user has performed the designated designation operation, that is, the user's finger or the like is displayed on the screen of the display unit 104. It is determined whether or not the position of the current enlargement start button 104c has been touched (step S106). If it is determined in step S106 that no designated operation has been performed, the control unit 101 determines whether or not a predetermined time (for example, about 10 seconds) has elapsed (step S107). If it is determined in step S107 that a predetermined time has elapsed, the control unit 101 clears data relating to the position of the face image Fn recorded in the temporary recording unit 1013a (step S108). Thereafter, the process returns to step S101, and the control unit 101 continues the through image display. On the other hand, if it is determined in step S107 that the predetermined time has not elapsed, the control unit 101 displays a through image display without clearing the data related to the position of the face image Fn recorded in the temporary recording unit 1013a. continue.

  Also, in the determination in step S106, when an instruction operation for which position designation has been performed is performed, the control unit 101 performs designated position relationship determination processing for determining the position relationship of each face image existing at the position specified by the user. Is performed (step S109).

  Here, the specified positional relationship determination process will be described. FIG. 5 is a flowchart showing the specified positional relationship determination process. The designated positional relationship determination process is a process for determining the composite position of the enlarged image by roughly determining the positional relationship of the face image designated by the user.

  In FIG. 5, the control unit 101 sets 1 to a parameter n indicating the number of face images (step S201). Next, the control unit 101 acquires position information (Xn, Yn) and (Xn + 1, Yn + 1) of the face images Fn and Fn + 1 recorded in the temporary recording unit 1013a (step S202).

  Thereafter, the control unit 101 determines whether or not the difference between the vertical position Yn of the face image Fn and the vertical position Yn + 1 of the face image Fn + 1 is greater than or equal to a predetermined value (for example, greater than or equal to the face width in the face image Fn) ( Step S203). If it is determined in step S203 that the difference between Yn and Yn + 1 is not equal to or greater than a predetermined value, the control unit 101 determines whether the average of Yn and Yn + 1 corresponds to the position of the lower half of the screen of the display unit 104. (Step S204). That is, in step S204, as shown in FIG. 4A, when the vertical width (number of pixels in the vertical direction) of the display unit 104 is H, (Yn + (Yn + 1)) / 2 is H / 2. Or less. If it is determined in step S204 that the average of Yn and Yn + 1 corresponds to the position of the lower half of the screen of the display unit 104, the control unit 101 displays the enlarged image Fn ′ and the enlarged image when displaying a partially enlarged through image. The composite position in the vertical direction of Fn + 1 ′ is set to the lower side of the screen of the display unit 104 (step S205). On the other hand, when the average of Yn and Yn + 1 does not correspond to the position of the lower half of the screen of the display unit 104 in the determination of step S204, the control unit 101 enlarges the enlarged image Fn ′ and the enlarged image when displaying a partially enlarged through image. The composite position in the vertical direction of the image Fn + 1 ′ is set as the upper side of the screen of the display unit 104 (step S206).

  For example, as shown in FIG. 6A, when two face images F1 and F2 are both at the upper position on the screen of the display unit 104, the average of Y1 and Y2 corresponds to the upper half of the screen of the display unit 104. Position. In this case, the combined position of the enlarged images F1 ′ and F2 ′ is set as the screen upper position as shown in FIG. In practice, it is preferable that the vertical position of the enlarged images F1 ′ and F2 ′ is the upper end position of the screen. This is to make the synthesis range of the entire image as wide as possible. On the other hand, as shown in FIG. 6C, when both the face images F <b> 1 and F <b> 2 are at the lower position on the screen of the display unit 104, the average of Y <b> 1 and Y <b> 2 corresponds to the lower half of the screen of the display unit 104. It becomes. In this case, the combined position of the enlarged images F1 ′ and F2 ′ is set as the screen lower position as shown in FIG. Also in this case, it is preferable to set the combined position of the enlarged images F1 ′ and F2 ′ in the vertical direction as the lower end position of the screen.

  If the difference between Yn and Yn + 1 is greater than or equal to the predetermined value in the determination in step S203, the control unit 101 displays the vertical composite position of the enlarged image corresponding to the face image with the larger Y in the display unit 104. The vertical composition position of the enlarged image corresponding to the face image with the smaller Y is set to the lower side of the screen of the display unit 104 (step S207). For example, as shown in FIG. 6E, when the face image F1 is at the upper position on the screen of the display unit 104 and the face image F2 is at the lower position on the screen of the display unit 104, the difference between Y1 and Y2 Becomes larger. In this case, as shown in FIG. 6 (f), the combined position of the enlarged image F1 ′ is the upper screen position, and the combined position of the enlarged image F2 ′ is the lower screen position. Also in this case, it is preferable that the combined position in the vertical direction of the enlarged image F1 ′ be the upper end position of the screen and the combined position in the vertical direction of the enlarged image F2 ′ be the lower end position of the screen.

  After determining the combined position in the vertical direction of the enlarged images Fn ′ and Fn + 1 ′ by the processing in steps S205 to S207, the control unit 101 compares X and Xn + 1, and whether Xn is smaller than Xn + 1, that is, the face It is determined whether or not the image Fn is present on the left side of the screen of the display unit 104 with respect to the face image Fn + 1 (step S208). If it is determined in step S208 that Xn is smaller than Xn + 1, the control unit 101 sets the horizontal composite position of the enlarged image Fn ′ at the time of partially enlarged live view display as the left side of the display unit 104, and the enlarged image Fn + 1. The horizontal composition position of 'is set as the right side of the screen of the display unit 104 (step S209). On the other hand, if it is determined in step S208 that Xn is not smaller than Xn + 1, the control unit 101 sets the horizontal composite position of the enlarged image Fn ′ at the time of displaying a partially enlarged through image as the right side of the screen of the display unit 104, The combined position of the enlarged image Fn + 1 ′ in the horizontal direction is set to the left side of the display unit 104 (step S210).

  For example, as shown in FIG. 6A, when the face image F1 is at the position on the left side of the screen of the display unit 104 and the face image F2 is at the position on the right side of the screen of the display unit 104, X1 is smaller than X2. Become. In this case, the combined position of the enlarged image F1 ′ is set to the left side position of the screen as shown in FIG. On the other hand, the combined position of the enlarged image F2 ′ is set as the screen right position. For example, as shown in FIG. 6C, when the face image F1 is at the position on the right side of the screen of the display unit 104 and the face image F2 is at the position on the left side of the screen of the display unit 104, X1 is greater than X2. Also grows. In this case, the combined position of the enlarged image F1 ′ is set to the right side position of the screen as shown in FIG. On the other hand, the combined position of the enlarged image F2 ′ is set as the left side position of the screen. Note that it is preferable that the actual horizontal combining position is determined so that enlarged images are arranged sequentially from the screen edge. For example, when it is determined that the composite position of the enlarged image F1 ′ is on the left side and the composite position of the enlarged image F2 ′ is on the right side, the horizontal composite position of the enlarged image F1 ′ is the screen left end position of the display unit 104. The combined position in the horizontal direction of the enlarged image F2 ′ is set as the screen right end position of the display unit 104. After that, for example, when it is determined that the synthesis position of the enlarged image F3 ′ is on the left side, the horizontal synthesis position of the enlarged image F3 ′ is determined at a position adjacent to the right side of the enlarged image F1 ′.

  After determining the horizontal combined position of the enlarged images Fn ′ and Fn + 1 ′ by the processing of steps S209 to S210, the control unit 101 determines whether or not the combined position of all the enlarged images has been determined (step S211). ). If it is determined in step S211 that the combined positions of all enlarged images have not been determined, the control unit 101 adds 2 to n (step S212). Thereafter, the process returns to step S202, and the control unit 101 determines the synthesis position of the next two enlarged images. On the other hand, if it is determined in step S211 that the combined positions of all the enlarged images have been determined, the control unit 101 ends the process of FIG. 5 and returns to the process of FIG.

  By performing the specified positional relationship determination process as described above, it is possible to make the positional relationship of the face image displayed when displaying a partially enlarged through image correspond to the positional relationship of the enlarged image. In the process of FIG. 5, the positional relationship between the face images is roughly determined based only on the position of each face image. On the other hand, you may make it determine the synthetic | combination position of an enlarged image according to the gaze detected by the gaze detection part 1031. FIG. For example, the synthesis position may be determined so that the lines of sight of two face images that are the determination targets face each other.

  In the example of FIG. 5, the relationship between the designated positions is determined every two images. Therefore, a surplus occurs when the number of designated face images is an odd number. In this case, the synthesis position is determined for one face image. As an example of this, in FIG. 5, step S203 is branched to NO, and the process of step S204 is performed to determine the vertical synthesis position. Further, in the determination in step S208, the horizontal combined position is determined depending on whether the position of Xn corresponds to the left side or the right side of the screen. For example, if Xn corresponds to the position on the left side of the screen, the horizontal composition position is determined on the left side of the screen, and if it corresponds to the right side position, the horizontal composition position is determined on the right side of the screen.

  Here, returning to FIG. 3, the description will be continued. After the designated positional relationship determination process in step S109, the control unit 101 performs an enlargement ratio determination process for determining the display size of each enlarged image (step S110). The enlargement ratio determination process will be described. FIG. 7 is a flowchart illustrating the enlargement ratio determination process. The enlargement ratio determination process is a process for determining the enlargement ratio of each face image from the number of faces in each face image.

  In FIG. 7, the control unit 101 sets 1 to a parameter n indicating the number of face images (step S301). Next, the control unit 101 uses the face detection unit 103 to detect the number Nn of faces included in the face image Fn (step S302). If the number of faces is recorded in the temporary recording unit 1013a in the process of step 104 in FIG. 3, it is only necessary to acquire the number Nn of faces recorded in the temporary recording unit 1013a in step S302.

  After acquiring the number of faces Nn, the control unit 101 determines whether or not the acquisition of the number of faces Nn has been completed for all face images (step S303). If it is determined in step S303 that acquisition of the number of faces Nn for all face images has not been completed, the control unit 101 adds 1 to n (step S304). Thereafter, the process returns to step S302, and the control unit 101 detects the number of the next face. On the other hand, in the determination in step S303, when the acquisition of the number of faces Nn for all face images is completed, the control unit 101 divides the screen of the display unit 104 by the ratio of the number of faces N1 to Nn. Then, the enlargement ratio of each face image is determined (step S305). Thereafter, the control unit 101 ends the process of FIG. 7 and returns to the process of FIG.

An example of the enlargement ratio determination method in step S305 will be described. Here, for example, as shown in FIG. 8A, a case where the user designates a face image F1 and a face image F2 arranged along the horizontal direction in the screen of the display unit 104 will be described. Here, it is assumed that the horizontal width of the display unit 104 (actually the number of pixels in the horizontal direction) is L. Further, the horizontal width of the face image F1 (actually the number of pixels in the horizontal direction) is assumed to be l1 in the screen of the display unit 104, and the horizontal width of the face image F2 (actually the horizontal direction) It is assumed that the number of pixels is 12 in the screen of the display unit 104. In this case, L is internally divided by the ratio of the horizontal width l ′ of the enlarged face image (enlarged image) F1 ′ and the horizontal width l2 ′ of the enlarged face image (enlarged image) F2 ′. So that The ratio between l1 ′ and l2 ′ is the number N1 of faces in the face image F1 (2 in the example of FIG. 8A) and the number N2 of faces in the face image F2 (1 in the example of FIG. 8A). Is equal to the ratio. Therefore, the enlargement factors E1 and E2 can be calculated as shown in the following (Equation 1).
E1 = (N1 / (N1 + N2)) × (L / l1)
E2 = (N1 / (N1 + N2)) × (L / l2) (Expression 1) Note that the enlargement ratio En for N face images can be calculated as shown in (Expression 2) below.

En = (Nn / (N1 + N2 +... + Nn)) × (L / ln) (Expression 2) where Nn represents the number of faces of the face image Fn, and ln is the horizontal width of the face image Fn (in practice, The number of pixels in the horizontal direction).

  As shown in FIG. 8A, face images F1 and F2 are trimmed from an image subjected to image processing for through image display, and among the trimmed face images, face image F1 is enlarged E1 times. At the same time, the face image F2 is enlarged to E2 times, and the enlarged images F1 ′ and F2 ′ obtained by the enlargement are combined and displayed as a through image, so that the partially enlarged through image display as described above can be performed. It is. Instead of enlarging the face image that has been subjected to image processing for through image display, as shown in FIG. 8B, enlarged images F1 ′ and F2 ′ are read when the image signal is read from the image sensor 102c. An image signal corresponding to the above may be acquired to generate a composite moving image. Since the image subjected to the image processing for through image display is reduced according to the screen size of the display unit 104, the resolution of the enlarged face image cannot be expected. On the other hand, it is possible to increase the resolution of the enlarged image by using an image directly trimmed from the image sensor 102c for the through image display.

  FIG. 8A shows an example in which composition is performed such that no gap is provided between the enlarged image F1 ′ and the enlarged image F2 ′. On the other hand, as shown in FIG. 2D, the composition may be performed so that a gap is provided between the enlarged image F1 ′ and the enlarged image F2 ′. In this case, the enlargement ratio of the face image F1 and the face image F2 including the gap is calculated.

  In the example of FIG. 7, the number of faces in the face image Fn is simply counted to determine the enlargement ratio. In this case, for example, as shown in FIG. 9A, when the face is arranged in both the horizontal and vertical directions in the face image, unnecessary enlargement of the face image may be performed. Actually, it is better to enlarge at a necessary enlargement ratio as shown in FIG. For this purpose, it is not simply enlarged according to the number of faces, but is enlarged according to the number of faces arranged in a direction along one side of the screen of the display unit 104 such as a horizontal direction or a vertical direction. .

  FIG. 10 is a diagram illustrating a modification of the enlargement ratio determination process. In FIG. 10, the control unit 101 sets 1 to a parameter n indicating the number of face images (step S401). Next, the control unit 101 uses the face detection unit 103 to detect XFn1 to XFnN, which are X coordinates of all the faces included in the face image Fn (step S402). Here, XFn1 to XFnN are not the position Xn described above, but the X coordinates of individual faces included in the face image Fn. If the X coordinate of each face is recorded in the temporary recording unit 1013a in the process of step 104 in FIG. 3, only the X coordinate of each face recorded in the temporary recording unit 1013a is acquired in step S402. good.

  After detecting the X coordinates XFn1 to XFnN of each face, the control unit 101 calculates a difference ΔXn between the maximum and minimum values of XFn1 to XFnN (step S403). In the example of FIG. 11, the face image F1 has one face. Therefore, ΔX1 is zero. Further, the face image F2 has five faces. ΔX2 is a difference between the X coordinate of the rightmost face F22 in the face image F2 and the X coordinate of the leftmost face F21 in the face image F2.

  After calculating ΔXn, the control unit 101 calculates an average FLn of the widths of the faces included in the face image Fn (step S404). For example, in the example of FIG. 11, the face image F1 has one face. Therefore, FL1 is the width of the face (the number of pixels in the horizontal direction). On the other hand, the face image F2 has five faces. Therefore, FL2 is the average of the widths of the five faces (the number of pixels in the horizontal direction). Note that the example of FIG. 11 illustrates the case where the widths of all five faces are equal.

After calculating FLn, the control unit 101 calculates the number Nn of faces arranged in the horizontal direction according to the following (Equation 3) (step S405).
Nn = (ΔXn + FLn) / FLn (rounded up) (Expression 3) For example, in the example of FIG. 11, the number of faces N1 for the face image F1 is (0 + FL1) / FL1 = 1. On the other hand, the number of faces N2 for the face image F2 is (2FL2 + FL2) / FL2 = 3.

  After calculating the number Nn of faces arranged in the horizontal direction, the control unit 101 determines whether or not the acquisition of the number Nn of faces has been completed for all the face images (step S406). If the acquisition of the number of faces Nn for all face images has not been completed in the determination in step S406, the control unit 101 adds 1 to n (step S407). Thereafter, the process returns to step S402, and the control unit 101 detects the number of next faces. On the other hand, in the determination of step S406, when the acquisition of the number n of faces for all the face images is completed, the control unit 101 divides the screen of the display unit 104 by the ratio of the number of faces N1 to Nn. Then, the enlargement ratio of each face image is determined (step S408). In the example of FIG. 11, an example is shown in which the enlargement ratios of the face images F1 and F2 are determined so that the horizontal and vertical divisions are 1: 3. After determining the enlargement ratio, the control unit 101 ends the process of FIG. 10 and returns to the process of FIG.

  By determining the enlargement ratio according to the processing as shown in FIG. 10, the possibility of unnecessary enlargement can be reduced. In the process of FIG. 10, the enlargement ratio is determined by the number of faces along the horizontal direction, but the enlargement ratio may be determined by the number of faces along the vertical direction.

  Here, returning to FIG. 3, the description will be continued. After the enlargement rate determination process in step S110, the control unit 101 performs a partially enlarged through image display (step S111). This partially enlarged through image display process will be described. FIG. 12 is a flowchart showing a partially enlarged through image display process. The partially enlarged through image display process is a process for displaying a combined moving image obtained by combining an enlarged image obtained by enlarging each face image and an entire image with a background image, as a through image.

  In FIG. 12, the control unit 101 trims each face image designated by the user from an image that has been subjected to image processing for through image display. Then, the control unit 101 enlarges each trimmed face image at the enlargement ratio determined by the enlargement ratio determination process, and then determines each enlarged image obtained by the enlargement by the designated positional relationship determination process in the background image. Then, they are synthesized at the respective synthesis positions (step S501). Next, the control unit 101 detects a blank portion remaining when the enlarged images are combined (step S502). For example, when the enlarged image F1 ′ is synthesized at the upper left position of the screen as shown in FIG. 13, the area excluding the portion of the enlarged image F1 ′ shown in FIG. However, the entire image to be combined with the margin part is a rectangular image. Therefore, it is necessary to make the margin part a rectangular area. For this reason, the control unit 101 divides the remaining portion by the synthesis of the enlarged image F1 ′ into a plurality of rectangular regions S1 and S2, and detects each rectangular region as a blank portion. Actually, since a plurality of enlarged images are combined, a remaining portion when a plurality of enlarged images are combined is divided into a plurality of rectangular areas to detect a blank portion.

  After detecting the margin part, the control unit 101 compares the area (number of pixels) of the margin part (step S503). Then, as a result of the area comparison, the control unit 101 synthesizes the entire image with the blank portion having the maximum area (step S504). For example, when the blank portions S1 and S2 are detected as shown in FIG. 14A, the entire image A is synthesized with the blank portion S2. On the other hand, when the margin portions S1 and S2 are detected as shown in FIG. 14B, the entire image A is synthesized with the margin portion S1. Since the entire image is not large enough to fit in the margin, it is necessary to reduce the entire image when combining. At this time, the horizontal direction and the vertical direction of the entire image may be reduced at a uniform reduction rate so that the size of the blank portion is simply accommodated. Further, it is only necessary to know the positional relationship of the face image designated by the user from the entire image displayed when the partially enlarged through image is displayed. Therefore, the aspect ratio of the entire image may be changed, that is, the image may be reduced by changing the reduction ratio between the horizontal direction and the vertical direction. Thereby, it is possible to synthesize the entire image with a size corresponding to the size of the margin.

  After synthesizing the entire image, the control unit 101 displays the synthesized moving image as a through image (step S505). Thereafter, the control unit 101 ends the process of FIG. 12 and returns to the process of FIG.

  Here, returning to FIG. 3, the description will be continued. After displaying the partially enlarged through image in step S111, the control unit 101 determines whether or not to perform moving image recording (step S112). In the determination in step S112, moving image recording is performed when the user's finger or the like touches the position of the recording button 104h displayed on the screen of the display unit 104 or when the release button of the operation unit 106 is pressed. Determine to do. In the determination in step S112, when moving image recording is performed, the control unit 101 compresses the moving image that is displayed as a partially enlarged through image in the image processing unit 1013, and causes the recording unit 107 to record the compressed moving image (step S112). S113). After starting the moving image recording, the control unit 101 determines whether or not to end the moving image recording (step S114). In the determination in step S114, when the user's finger or the like touches the position of a recording end button (not shown) displayed on the screen of the display unit 104, or when the release button of the operation unit 106 is pressed again, It is determined that the image recording is finished. If it is determined in step S114 that the moving image recording is not terminated, the process returns to step S113, and the control unit 101 continues the moving image recording. On the other hand, in the determination in step S114, when the moving image recording is ended, the control unit 101 ends the process of FIG.

  Further, in the determination in step S112, when the moving image recording is not performed, the control unit 101 ends the partially enlarged through image display, that is, the user's finger or the like is displayed on the screen of the display unit 104. It is determined whether or not the position of the enlargement end button 104g has been touched (step S115). If it is determined in step S115 that the partially enlarged through image display is to be terminated, the process returns to step S101. In this case, the control unit 101 performs normal through image display. On the other hand, if it is determined in step S115 that the partial enlarged live view display is not to be ended, the control unit 101 has selected a face image in the partially displayed live view image, that is, the user's finger or the like. It is determined whether or not the position of the face image displayed on the screen of the display unit 104 has been touched (step S116). If it is determined in step S116 that no face image is selected, the process returns to step S111. In this case, the control unit 101 continues the partially enlarged through image display.

  In the determination in step S116, when a face image is selected, the control unit 101 trims the face image portion selected by the user, and the image processing unit 1013 performs image processing on the trimmed face image. As shown in FIG. 2E, the display unit 104 displays the entire screen (step S117). Thereafter, the control unit 101 determines whether or not to perform moving image recording (step S118). In the determination in step S118, when moving image recording is performed, the control unit 101 compresses the moving image of the face displayed on the full screen through the moving image in the image processing unit 1013 and causes the recording unit 107 to record the moving image (step S118). S119). After starting the moving image recording, the control unit 101 determines whether or not to end the moving image recording (step S120). If it is determined in step S120 that the moving image recording is not terminated, the process returns to step S119, and the control unit 101 continues the moving image recording. On the other hand, when the moving image recording is ended in the determination in step S120, the control unit 101 ends the process of FIG.

  In the determination of step S118, when moving image recording is not performed, the control unit 101 determines whether the return operation has been performed, that is, the position of the return button 104i on which the user's finger or the like is displayed on the screen of the display unit 104. It is determined whether or not a contact has been made (step S121). If it is determined in step S121 that a return operation has been performed, the process returns to step S111. In this case, the control unit 101 performs partial enlarged through image display. On the other hand, if it is determined in step S121 that no return operation has been performed, the process returns to step S117. In this case, the control unit 101 continues to display the entire face image of the face image.

  As described above, according to the present embodiment, by displaying a through image of an image obtained by synthesizing an enlarged image obtained by extracting and enlarging a face image from the entire image and the entire image, expressions and reactions of a plurality of subjects are displayed. Etc. can be observed in detail. In addition, since the enlargement ratio of the face image is changed according to the number of faces in the face image, each face can be observed in detail even if the face image includes a plurality of faces. is there. Furthermore, the possibility that the face image is unnecessarily enlarged can be reduced by determining the enlargement ratio of the face image according to the number of faces arranged in a direction parallel to one side of the screen of the display unit 104. It is.

  In addition, by making the arrangement of the enlarged images correspond to the arrangement of the enlarged images in the entire image, it is possible to roughly grasp the positional relationship between the faces even with the enlarged images alone. Further, by sequentially determining the composition position of the enlarged image from the screen edge position of the display unit 104, it becomes easy to secure a margin part for compositing the entire image.

  In the above-described embodiment, the user can designate n face images. However, if the number of face images that can be specified increases, the size of each enlarged image displayed by partially enlarged through image display may be reduced, making it difficult to observe the facial expression of the subject. Therefore, the number of face images that can be specified may be limited so that the enlarged image does not become unnecessarily small. Further, the number of enlarged images displayed on the display unit 104 at a time may be limited, and the enlarged images displayed on the display unit 104 may be sequentially switched every predetermined time.

  Although the present invention has been described above based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications and applications are naturally possible within the scope of the gist of the present invention.

  Further, the above-described embodiments include various stages of the invention, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some configuration requirements are deleted from all the configuration requirements shown in the embodiment, the above-described problem can be solved, and this configuration requirement is deleted when the above-described effects can be obtained. The configuration can also be extracted as an invention.

  DESCRIPTION OF SYMBOLS 100 ... Camera, 101 ... Control part, 102 ... Imaging part, 102a ... Lens, 102b ... Aperture, 102c ... Imaging element, 102d ... A / D converter (ADC), 103 ... Face detection part, 104 ... Display part, 105 ... Position specifying part 106 ... Operation part 107 ... Recording part 108 ... Clock part 1011 ... Trimming control part 1012 ... Trimming image composition control part 1013 ... Image processing part 1013a ... Temporary recording part 1031 ... Gaze detection Part

Claims (11)

A display for displaying an image;
A face detection unit for detecting a face included in an image region in the image displayed on the display unit;
A face image generation unit that generates a face image including the face detected by the face detection unit for each image region;
An enlargement rate determination unit that determines an enlargement rate of each face image generated by the face image generation unit according to the number of faces included in the image region;
An image creation unit that creates a composite image in which a face image enlarged according to the enlargement rate determined by the enlargement rate determination unit is synthesized in a background image corresponding to the screen size of the display unit;
Comprising
An image creating apparatus that records the enlarged face image when the enlarged face image is selected. A display for displaying an image;
A face detection unit for detecting a face included in an image region in the image displayed on the display unit;
A face image generation unit that generates a face image including the face detected by the face detection unit for each image region;
An enlargement rate determination unit that determines an enlargement rate of each face image generated by the face image generation unit according to the number of faces included in the image region;
An image creation unit that creates a composite image in which a face image enlarged according to the enlargement rate determined by the enlargement rate determination unit is synthesized in a background image corresponding to the screen size of the display unit;
Comprising
An image creating apparatus that records the composite image in accordance with a recording instruction.   3. The image according to claim 1, wherein the synthesized image is created by synthesizing a button for terminating the display of the enlarged synthesized image, and the synthesized image is returned to a through image in accordance with an operation instruction of the button. Creation device.   The image creation unit reduces an image showing the entire image displayed on the display unit to a remaining portion generated in the background image by the synthesis of the enlarged face image when creating the composite image. The image creating apparatus according to claim 1, further comprising a composition.   5. The enlargement ratio determining unit determines the enlargement ratio of each face image so that a screen of the display unit is divided by a ratio of the number of faces in each face image. The image creating apparatus according to any one of the above.   The enlargement factor determination unit determines the enlargement factor of each face image so that the screen of the display unit is divided by the ratio of the number of faces arranged along one side of the screen of the display unit in each face image. The image creating apparatus according to claim 1, wherein the image creating apparatus includes: A specified positional relationship determination unit that determines a combined position of each face image generated by the face image generation unit according to the positional relationship of the image region;
The image creation apparatus according to claim 3, wherein the specified positional relationship determination unit determines a synthesis position of each face image at a synthesis position of the face image corresponding to a screen edge of the display unit.   The said image creation part performs the said reduction | decrease so that the aspect-ratio of the image which shows the whole image currently displayed on the said display part may be changed according to the shape of the said remaining part. Image creation device. An imaging unit that captures an image of the subject and acquires an image related to the subject;
The image creating apparatus according to claim 1, wherein the face image generation unit generates the face image from an image acquired by the imaging unit. The face included in the image area in the image displayed on the display unit is detected by the face detection unit,
Generating a face image including a face detected by the face detection unit for each image region;
Determine the enlargement ratio of each face image generated by the trimming control unit according to the number of faces included in the image area,
In the background image corresponding to the screen size of the display unit, create a composite image that combines the face image enlarged according to the determined enlargement rate,
An image creating method comprising: recording the selected image when the image area is selected. The face included in the image area in the image displayed on the display unit is detected by the face detection unit,
Generating a face image including a face detected by the face detection unit for each image region;
Determine the enlargement ratio of each face image generated by the trimming control unit according to the number of faces included in the image area,
In the background image corresponding to the screen size of the display unit, create a composite image that combines the face image enlarged according to the determined enlargement rate,
An image creating method, wherein the composite image is recorded in accordance with a recording instruction.

Images