JP4849163B2 - Image processing apparatus, image processing method, and computer program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a computer program for capturing an image including a subject such as a person, and more particularly, to a camera according to recognition of a specific pattern from a subject included in an image. The present invention relates to an image processing apparatus, an image processing method, and a computer program for performing work automation.

  More particularly, the present invention relates to an image processing apparatus, an image processing method, and a computer program that automatically determine shooting timing using recognition of a smile from a subject included in an image. The present invention relates to an image processing apparatus, an image processing method, and a computer program for performing smile detection by paying attention to an appropriate subject when an image includes a plurality of subjects and automatically determining shooting timing.

  Cameras have a long history as a means of recording visual information. Recently, digital cameras that digitize images captured by solid-state imaging devices such as CCD (Charge Coupled Device) and CMOS (Complementary Mental-Oxide Semiconductor) have been used in place of film cameras that use film or photosensitive plates. Widely used. According to the digital camera, there is an advantage that a digitally encoded image can be stored in a memory, image processing and image management by a computer can be performed, and there is no problem of film life.

  Although the basic camera work is the same for both silver halide cameras and digital cameras, automation technology for camera work is advancing in conjunction with the digitization of captured images. Even the user has a very low possibility of shooting failure.

  For example, auto focus function (AF), automatic exposure (AE), camera shake correction, and the like. In the case of a digital camera, there is an advantage that processing such as AF and AE can be realized not only by an optical system but also by digital processing. In addition, an appropriate color state can be reproduced by performing image quality adjustment on a captured image by digital processing, such as multiplying a captured image by white balance / gain using AWB (Auto White Balance).

  Functions such as auto focus and image quality correction should be processed in accordance with any one or more subjects existing in the image frame. In the old days, the user set an important subject manually. Recently, imaging technology using face detection has been introduced, such as detecting a face from a captured image and automatically focusing on the face, or automatically correcting the image quality according to the face. ing.

  Here, as a method for detecting a face from a captured image signal, for example, a method for matching an input image signal using an average face image as a template is widely known in the art (for example, Patent Document 1). checking). In addition, if the imaged face is tilted, it is difficult to detect the face correctly. Therefore, the face angle is detected, the face image is rotated according to the detected value, and matching processing is performed. Techniques are also known. As a method for detecting the face angle, the center line of the face is detected, the degree of the center line is calculated based on the symmetry of the face, and the angle of the center line having a high degree is calculated as the face angle. For example (see, for example, Patent Document 2).

  Recently, research and development on an auto shutter that determines the timing of shooting using the recognition of a specific pattern from an image of a subject such as a smile is underway.

  For example, there has been proposed a camera that determines whether a main subject satisfies a predetermined imaging condition at a site of interest included in the main subject to be imaged and outputs an imaging timing signal when the imaging condition is satisfied (for example, And see Patent Documents 3 to 5). The attention site here refers to, for example, the shape and color of the eyes and mouth of the person when photographing a person, the feathers of the bird when photographing a bird, and the like.

  Specific criteria for shooting timing are appropriate shooting timings such as “not blinking”, “looking at the camera”, “smiling”, etc. It can be considered as a criterion. In addition, the technology that corrects the photo to improve the photo taken with the person as the subject has been known conventionally, but the photo at the moment when the person is blinking can be corrected to the photo that does not blink, One of the reasons for making smiles a shooting condition is that it is difficult to correct a photograph of a person who is not laughing as if it were laughing.

  For example, the shooting condition of “smiling” is determined based on image information such as the area of the white part of the subject's eyes, the lateral length of the mouth, and the area of the white part of the mouth. Can do. However, when a plurality of subjects are included in an image frame input from the image sensor, it is unclear which subject should be focused on to detect a smile and control the shooting timing.

  For example, when a plurality of faces are detected, the largest or closest face frame is set as the AF area, and when face detection is performed using a through image, the face detection is not performed. Also, a proposal has been made for a digital camera that increases the brightness of a through image and facilitates face detection (see, for example, Patent Document 6). However, the digital camera does not automatically shoot based on smile detection, and the detection face to be set in the AF area is automatically determined by the camera, and the photographer wants to shoot a smile himself. The desired subject cannot be specified.

  In addition, the focus of the photographic lens is adjusted so that the closest face among the detected faces is in focus, and a frame surrounding the focused face is displayed on the liquid crystal display, and the face surrounded by the frame An image pickup apparatus that moves a frame to another face by a user operation such as a cross key when the face is not a desired person's face has been proposed (see, for example, Patent Document 7). This image pickup apparatus determines a leading role based on being close to the camera and performs automatic focusing, but does not perform automatic shooting based on smile detection. In addition, although the photographer can specify a subject that he / she wants to take a smile on his / her own, only a specific subject can be selected as a detection target.

  In the case of automatic focusing, although the characteristics such as the depth of focus of the optical system also depend on each other, it is basically impossible to select a plurality of subjects as detection targets at the same time because there is only one point of image formation. On the other hand, in the case of automatic shooting with smile detection, for example, when a relatively large number of people is included in the angle of view when shooting a scene such as a party, it is not sufficient that one of the members of interest is laughing. And it is sometimes required as a shooting condition that the members are laughing at the same time. In other words, automatic shooting based on smile detection is essentially different from automatic focusing.

JP 2003-271933 A (paragraphs 0051 to 0053, FIG. 6) Japanese Patent Laid-Open No. 9-171560 (paragraphs 0032 to 0044, FIG. 4) JP 2000-347278 A JP 2004-294498 A JP 2006-237803 A JP 2006-33437 A JP 2006-145629 A

  An object of the present invention is to provide an excellent image processing apparatus, image processing method, and computer program capable of suitably performing camera / work automation processing in response to recognition of a specific pattern from a subject included in an image. Is to provide.

  A further object of the present invention is to provide an excellent image processing apparatus and image processing method capable of automatically determining an appropriate shooting timing using recognition of a smile from a subject included in an image, and a computer To provide a program.

  A further object of the present invention is to provide an excellent image processing apparatus capable of performing smile detection by paying attention to an appropriate subject and automatically determining a suitable shooting timing when the input image includes a plurality of subjects. And an image processing method, and a computer program.

The present invention has been made in consideration of the above problems, and a first aspect of the present invention is an image input means for inputting an image formed by imaging a subject;
Image signal processing means for signal processing the input image;
Display means for displaying an input image after signal processing by the image signal processing means;
Face detection means for detecting the face of the subject from the input image after signal processing by the image signal processing means;
A smile detection target face setting unit that sets a target for smile detection from the detected faces detected by the face detection unit;
Display control means for displaying the detection face set by the smile detection target face setting means on the through image displayed by the display means separately from other detection faces;
A smile detection face changing means for changing a detection face of the smile detection target in accordance with a user operation on a display screen that distinguishes the detection face set as the smile detection target and the other detection faces;
It is an image photographing device characterized by comprising.

  An image capturing apparatus according to the present invention is configured to detect a smile level of a detected face that is a smile detection target, and to control a shooting process in response to the detected smile level reaching the smile detection level. Has been. Of course, it is also conceivable to perform processing other than shooting based on the smile detection result.

  At present, image photographing devices such as digital cameras are widely used, and various camera / work automation technologies are being promoted in parallel. Among these, with regard to automatic shooting, research and development have been conducted on an auto-shutter function that determines the timing of shooting using recognition of a specific pattern from an image of a subject such as a smile. Specific criteria for shooting timing are appropriate shooting timings such as “not blinking”, “looking at the camera”, “smiling”, etc. It can be considered as a criterion.

  Here, when a plurality of subjects are included in the image frame input from the image sensor, there is a problem of which subject should be focused on to detect a smile and control the photographing timing.

  On the other hand, the image photographing apparatus according to the present invention is configured such that the photographer detects a smile and freely selects a subject for which a smile is desired. Specifically, from the detected faces detected by the face detecting means, those to be smile detection targets are set, and the detected face set as the smile detection target is distinguished from other detection faces and displayed on the through image as an OSD. However, the face to be detected for smile is changed according to the user's operation on the display screen.

  When the face detection means detects faces of a plurality of subjects from the input image, the smile detection target face setting means determines the importance of each detected face based on the position of each detection face in the image or other information. A priority is assigned to each detected face in accordance with the importance, and a predetermined number of detected faces are set as smile detection targets from the higher priority.

  In addition, in order for a user such as a photographer to cut a detection face that is a smile detection target by a dead person, the display control unit displays a through image with a frame on the detection face that is a smile detection target. To do. Alternatively, the display control means attaches a frame to all detected faces detected by the face detection means, and changes the display format of the frame for the detected face that is a smile detection target or attaches a predetermined icon to the through face. Display an image.

  Here, when two or more detected faces are set as smile detection targets, the shooting control means automatically performs shooting when at least one smile is detected, or smiles of all the selected N people are detected. If detected, shooting may be performed automatically.

  The display screen of the display means may also have a touch panel. In such a case, the smile detection face changing unit may change the detection face of the smile detection target in accordance with a user operation on the touch panel.

  Alternatively, the face detection unit detects faces of a plurality of subjects from the input image and assigns an operation order to each detection face, and the smile detection target face setting unit sets a plurality of detection faces according to the importance of each detection face. Is set as a smile detection target, the smile detection face changing means specifies a detection face designated by a user through a general operation key such as an up / down / left / right key in accordance with an operation order, and detects the detected face. Raise the priority of the face and change it to a smile detection target.

  In addition, when a plurality of detected faces are set as smile detection targets, the priority order of the detection faces designated by the user via the touch panel is raised and changed to smile detection targets.

  Further, when a plurality of detected faces are set as smile detection targets, the smile detection face changing means sets the priority order of the detected faces in the area designated by the user on the screen via the touch panel. You may make it change to the object of smile detection.

  Here, in the case where the number of subjects that can be smile detection targets is limited to N, detection faces exceeding N people are included in an area designated by the user on the screen via the touch panel. The priority for each detected face in the region is determined, a priority is assigned to each detected face according to the importance, and a predetermined number of detected faces from the higher priority are subject to smile detection. Should be set to.

  As described above, according to the image photographing apparatus of the present invention, the photographer can freely select a subject to detect a smile and to take a smile. For example, a smile detection target subject can be arbitrarily designated on a touch panel that displays a through image, and a subject that is a smile detection target and a non-target subject can be distinguished and displayed on the through image. . Therefore, for example, when shooting a scene such as a party, when a relatively large number of people is included in the angle of view, automatic shooting based on smile detection is performed under the condition that two or more members are laughing at the same time. It can be suitably performed.

According to a second aspect of the present invention, there is provided a computer program written in a computer-readable format so as to execute a process for photographing an input image formed on a subject on a computer. ,
An image signal processing procedure for signal processing the input image;
A display procedure for executing the image signal processing procedure and displaying the input image after the signal processing on a predetermined display means;
A face detection procedure for executing the image signal processing procedure to detect the face of the subject from the input image after the signal processing;
A smile detection target face setting procedure for setting what should be a smile detection target from the detected faces detected by executing the face detection procedure;
On the through image displayed on the screen, a display control procedure for displaying the detected face set by executing the smile detection target face setting procedure separately from other detected faces;
A smile detection face changing procedure for changing a detection face of the smile detection target in accordance with a user operation on a display screen that distinguishes the detection face set as the smile detection target and the other detection faces;
Is a computer program characterized in that

  The computer program according to the second aspect of the present invention defines a computer program described in a computer-readable format so as to realize predetermined processing on the computer. In other words, by installing the computer program according to the second aspect of the present invention in the computer, a cooperative action is exhibited on the computer, and the same as the image photographing apparatus according to the first aspect of the present invention. An effect can be obtained.

  Advantageous Effects of Invention According to the present invention, an excellent image processing apparatus, image processing method, and computer program capable of automatically determining an appropriate shooting timing using recognition of a smile from a subject included in an image Can be provided.

  Further, according to the present invention, when an input image includes a plurality of subjects, it is possible to perform smile detection by paying attention to an appropriate subject and automatically determine a suitable shooting timing. An apparatus, an image processing method, and a computer program can be provided.

  According to the image processing apparatus of the present invention, a photographer can detect a smile and freely select a subject for which a smile is desired. For example, a smile detection target subject can be arbitrarily designated on a touch panel that displays a through image, and a subject that is a smile detection target and a non-target subject can be distinguished and displayed on the through image. . Therefore, for example, when shooting a scene such as a party, when a relatively large number of people is included in the angle of view, automatic shooting based on smile detection is performed under the condition that two or more members are laughing at the same time. It can be suitably performed.

  Other objects, features, and advantages of the present invention will become apparent from more detailed description based on embodiments of the present invention described later and the accompanying drawings.

FIG. 1 is a diagram schematically showing a hardware configuration of a digital still camera 1 to which the present invention can be applied. FIG. 2 is a diagram schematically showing a data flow when a camera image is displayed through on the monitor / display 25. FIG. 3 is a diagram schematically showing a data flow when the digital still camera 1 performs image photographing processing. FIG. 4 is a flowchart showing a processing procedure when the digital still camera 1 captures an image. FIG. 5 is a flowchart showing a processing procedure for the digital still camera 1 to perform automatic shooting based on smile detection and notify the subject of shooting timing. FIG. 6 is a diagram illustrating an example of a through image in which only a detection target detection face frame is displayed. FIG. 7 is a flowchart showing a processing procedure for displaying a frame on the detection face to be detected. FIG. 8 is a diagram illustrating an example of a through image displayed by changing only the frame shape of the detected face selected as the smile detection target. FIG. 9 is a diagram illustrating an example of a through image in which an icon is added to the detection face frame to be detected. FIG. 10 is a diagram for explaining an operation for changing the automatically selected smile detection target face to another face. FIG. 11 is a diagram for explaining an operation of changing the automatically selected smile detection target face to another face. FIG. 12 is a diagram for explaining processing for assigning an operation order for key operation to each detected face detected from an image. FIG. 13 is a diagram showing an example of screen transition when the detection face to be processed is changed according to the key operation of the up / down / left / right keys. FIG. 14 is a diagram showing an example of screen transition when changing the detection face of the smile detection target when there are three detection faces to be smile detection targets. FIG. 15A is a diagram illustrating an example of screen transition when a plurality of detected faces are simultaneously taught as smile detection targets. FIG. 15B is a diagram showing an example of screen transition when a plurality of detected faces are simultaneously taught as smile detection targets. FIG. 15C is a diagram illustrating an example of screen transition when a plurality of detected faces are simultaneously trained as smile detection targets. FIG. 15D is a diagram illustrating an example of screen transition when a plurality of detected faces are simultaneously trained as smile detection targets. FIG. 16 is a flowchart illustrating a processing procedure for switching a detection face to be a smile detection target according to a user's touch panel operation.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 schematically shows a hardware configuration of a digital still camera 1 to which the present invention can be applied. The illustrated digital still camera 1 includes an image sensor 11, a preprocessing unit 12, and a camera / digital signal processing unit (DSP) 16, and digitally processes and records an image input via the optical system 10. It is like that.

  The optical system 10 includes a lens for condensing light from the subject on the image sensor 11, a drive mechanism for moving the lens to perform focusing and zooming, and light from the subject for a predetermined time by opening and closing operations. 11 includes a shutter mechanism that is incident on the lens 11 and an iris (aperture) mechanism that limits the direction and range of the light flux from the subject (both not shown). The driver 10a controls driving of each mechanism in the optical system 10 based on a control signal from the CPU 29 described later.

  In the imaging device 11, pixels having a photoelectric conversion effect such as a CCD and a CMOS are two-dimensionally arranged to convert incident light from a subject into an electrical signal. For example, a G checkered RB color coding veneer is disposed on the light receiving side. Signal charges corresponding to the amount of incident light that has passed through each color filter are accumulated in each pixel, and the color of incident light at that pixel position can be reproduced from each signal charge amount of three colors read from each pixel. The analog image signal output from the image sensor 11 is a primary color signal for each of RGB colors, but may be a complementary color system color signal.

  An AFE (Analog Front End) unit 12 includes a CDS / AGC / ADC block 13, a timing generator 14, and a V driver 15. For example, the AFE (Analog Front End) unit 12 is configured as one IC (Integrated Circuit) chip.

  In the CDS / AGC / ADC block 13, after the low noise of the signal received from the image sensor 211 in the pixel signal is suppressed with high accuracy (Correlated Double Sampling (CDS)), the sample is held. Appropriate gain control is performed by an AGC (Automatic Gain Control: automatic gain control circuit), AD conversion is performed, and a digital image signal is output.

  The timing generator 14 generates a timing pulse signal for driving the image sensor 11. The V driver 15 outputs a drive signal for outputting the charge of each pixel of the image sensor 11 in the vertical direction in line units according to the timing pulse signal.

  The camera DSP (Digital Signal Processor) unit 16 includes a camera signal processing unit 17, a resolution conversion unit 18, an image codec processing unit 19, a memory control unit 20, a display control unit 21, and a media control unit 22. For example, it is configured as one IC chip.

  The camera signal processing unit 17 performs preprocessing such as defective pixel correction, digital clamping, and digital gain control on the image signal sent from the AFE unit 12, and then multiplies the white balance and gain by AWB. At the same time, image quality correction processing such as sharpness and contrast adjustment is performed to reproduce the appropriate color state, and RGB screen signals are created by demosaic processing (including image processing such as flash dimming and red-eye reduction) be able to). Further, the camera signal processing unit 17 performs γ correction on the RGB image signal, and converts the image information into a gradation suitable for monitor output, printout, or image recording. Further, the camera signal processing unit 17 detects luminance information based on the output image signal of the AFE unit 13, calculates histograms for each predetermined range, and color detection for detecting color information for each predetermined range. Done. The results of histogram detection and color detection are output to the control unit 27 described later.

  A resolution converter 18 changes the size of the image. The image codec processing unit 19 performs color space conversion of the RGB image signal into a luminance signal and a color difference signal (Y / Cr / Cb), and encodes compression such as JPEG (Joint Picture Experts Group) compression or MPEG (Moving Picture Experts Group). Perform processing.

  The memory control unit 20 controls an access operation for writing and reading data such as captured image information to and from a memory device 24 such as an SDRAM (Synchronous DRAM).

  The monitor / display 25 is composed of, for example, an LCD (Liquid Crystal Display) or another type of thin display panel. The display control unit 21 controls the drive of the monitor / display 25, and the image data (through image) output from the camera signal processing unit 17 and the image data held in the memory device 24 are displayed on the monitor / display 25. The signal is converted into a signal for display and output to the monitor display 25.

  In addition to displaying a through image on the monitor / display 25, a menu screen, various setting screens, various warning information, and the like are combined in response to a request from the control unit 27, which will be described later. Displayed as SCREEN Display). In the present embodiment, a detection frame for identifying a subject arbitrarily designated as a smile detection target by the photographer is included as one of the setting screens displayed on the monitor / display 25 by OSD. It is also possible to display a detection frame by distinguishing between a subject that is a smile detection target and a non-target subject. Details of these will be described later.

  In the present embodiment, a touch sensor is superimposed on the surface of the monitor / display 25 and also serves as a touch panel. The coordinates recognized by the touch panel and the coordinates displayed on the display panel are the same. Therefore, the photographer can directly designate a subject to be detected as a smile, other GUI parts, and the like on the display screen of the monitor / display.

  The media control unit 22 includes a slot for loading a removable recording medium 26 such as a Memory Stick (registered trademark), and writes and reads data to and from the recording medium 26. For example, the media control unit 22 records the captured image file held in the memory device 24 on the recording medium 26.

  The camera DSP unit 16 is connected to the control unit 27 via a bus interface unit (BIU) 23. The control unit 27 includes a UI (User Interface) unit 28, a CPU (Central Processing Unit) 29 that comprehensively controls the operation of the entire apparatus according to a user operation, a RAM 30 that is a main storage memory for the CPU 29, It consists of an EEPROM (Electrically Erasable and Programmable ROM) 31 that holds program codes, device information, and the like in a nonvolatile manner.

  The CPU 29 performs subject smile detection processing using the image whose resolution has been converted by the resolution converter 18. Further, the CPU 29 performs GUI processing for OSD display on the monitor / display 25. In the present embodiment, the GUI process includes a display of a detection frame for identifying a subject arbitrarily designated as a smile detection target by the photographer, and a subject that is a smile detection target and a non-target subject. Separately, a detection frame is displayed.

  The UI unit 28 has an operation function for a user to input an instruction and an output function such as a speaker or an LED lamp. As operation functions, for example, a shutter release button, keys and dials for performing various settings such as on / off of flash emission, and the like, control signals corresponding to user operations on these are input to the CPU 29. In the present embodiment, the UI unit 28 has, as an operation function, up / down / left / right keys as shown in the figure, and operation keys (or cross keys) capable of five types of input (center key) surrounded by these four keys. ).

  Further, the UI unit 28 can feed back information related to the camera work, etc., to the user by using an output function, by outputting a sound such as an alarm, or by lighting or blinking the LED.

  The flash light emitting unit 32 emits a flash according to the flash control information from the control unit 27.

  In the digital still camera 1, the signals received and photoelectrically converted by the image sensor 11 are sequentially supplied to the AFE unit 12, subjected to CDS processing and AFE processing, and then converted into digital signals. The camera signal processing unit 17 corrects the image quality of the digital image signal supplied from the AFE unit 12, and finally converts the digital image signal into a luminance signal (Y) and a color difference signal (RY, BY) and outputs the result.

  The image data output from the camera signal processing unit 17 is converted into a display image signal by the display control unit 21, and a through image is displayed on the monitor / display 25.

  FIG. 2 schematically shows a data flow when a camera image is displayed on the monitor display 25.

  Information on the subject is converted into a digital signal through the optical system 10, the image sensor 11, and the AFE unit 12 and sent to the camera DSP unit 16. The camera DSP unit 16 corrects brightness, color, and the like by image signal processing, further converts the resolution of the image to be displayed on the panel, and displays the resulting image on the monitor display 25. Is done. Such control of the image output processing is realized in such a manner that the CPU 29 in the control unit 27 executes a predetermined program code. Actually, the sharing of control is finely divided, but since it is not directly related to the gist of the present invention, detailed description thereof is omitted here.

  When the control unit 27 is instructed to record an image by a user operation or the like via the UI unit 28, the image data from the camera signal processing unit 17 is supplied to the image codec processing unit 19, and a predetermined compression encoding process is performed. Is stored in the memory device 24 through the memory control unit 20 or stored in the recording medium 26 through the media control unit 22. When a still image is recorded, image data for one frame is supplied from the camera signal processing unit 17 to the image codec processing unit 19. Incidentally, when recording a moving image, processed image data is continuously supplied to the image codec processing unit 19.

  FIG. 3 schematically shows a data flow when the digital still camera 1 performs image photographing processing. FIG. 4 shows a processing procedure when the digital still camera 1 captures an image in the form of a flowchart.

  As described above with reference to FIG. 2, when the user presses the shutter key in the UI unit 28 with the subject displayed on the monitor / display 25 (step S <b> 1), the UI unit 28 The key is detected and a key information signal is sent to the CPU 29. In response to this, the CPU 29 outputs a control signal to the driver 10a and executes a photographing process (step S2). At that time, AF and AE processes are appropriately performed together.

  When an instruction for photographing processing is given through the driver 10a, the subject information is converted into a digital signal through the optical system 10, the image sensor 11, and the AFE unit 12 and sent to the camera DSP unit 16 as described above. In the camera DSP unit 16, brightness, color, and the like are corrected by image signal processing, and resolution conversion is further performed to obtain an image size corresponding to the shooting mode (step S3). Thereafter, the image codec processing unit 19 performs JPEG compression processing on the image (step S4), and then records the image on the recording medium 26 through the media control unit 22 (step S5).

  The digital still camera 1 according to the present embodiment can apply an auto-shutter function that determines the timing of shooting using the recognition of a specific pattern from an image of a subject such as a smile. Smile recognition can be realized by executing a predetermined image recognition processing program in the CPU 29 or mounting an engine for image recognition processing (not shown) in the control unit 27. Since the smile recognition technology itself may be known, detailed description thereof is omitted here.

  Note that the digital still camera 1 uses the output function of the UI unit 28 when the camera performs a shooting process, and performs shooting using an external output from the UI unit 28 such as generation of an alarm sound or lighting or blinking of a lamp. May be notified to the subject.

  The digital still camera 1 according to the present embodiment can perform automatic photographing processing based on detection of a smiling face of a subject. FIG. 5 shows a processing procedure for the digital still camera 1 to perform automatic photographing based on smile detection in the form of a flowchart.

  First, the camera image input from the optical system 10 is displayed on the monitor display 25 according to the same data flow as in FIG. 2 (step S11). At this time, the displayed image is also supplied to the control unit 27.

  In the present embodiment, the photographer can detect a smile and freely select a subject for which a smile is desired to be taken (step S12). Specifically, a subject for smile detection is arbitrarily designated on a monitor display 25 (touch panel) that displays a through image.

  The CPU 29 (or an image recognition engine (not shown)) that executes a predetermined image recognition processing program detects the smile level for the subject specified in the preceding step S12 (step S13). In response to this designation operation, a detection frame for identifying the designated designated subject is displayed on the monitor / display 25 by OSD. At this time, the detection frame may be displayed by distinguishing a subject that is a smile detection target and a non-target subject.

  Here, when the user presses the shutter key in the UI unit 28, the camera photographing process is started (step S14). Specifically, the CPU 29 instructs the photographing process through the driver 10 a, and the subject information is converted into a digital signal through the optical system 10, the image sensor 11, and the AFE unit 12 and sent to the camera DSP unit 16. The camera DSP unit 16 corrects brightness, color, and the like by image signal processing. Here, when the automatic shooting function is turned on based on smile detection, the shutter is not immediately released by pressing the shutter key, but the smile level of the subject reaches a preset smile detection level. When the camera shooting process starts.

  In this way, after the camera photographing process is performed according to the pressing operation of the shutter key or the smile detection result, the image is converted into the image size according to the photographing mode by the resolution conversion unit 18 (step S15), and further the image codec. After the image is subjected to JPEG compression processing by the processing unit 19 (step S16), it is recorded on the recording medium 26 through the media control unit 22 (step S17).

  In the digital still camera 1 according to the present embodiment, a photographer can arbitrarily designate a subject for smile detection, and for example, it is allowed to designate two or more subjects as smile detection targets at the same time. Yes. Then, a subject that is a smile detection target and a non-target subject are distinguished and displayed on the through image, and when the smile level of the subject obtained from the through image satisfies a predetermined shooting condition, the shooting process is automatically performed. Start up automatically. Hereinafter, a photographing process based on smile detection will be described. Note that it is conceivable to equip the control unit 27 with an image recognition engine for subject face detection and detected face smile detection processing, but in the following, these processing will be performed in the CPU 29 for convenience. And

(1) Face Detection Processing The image data whose resolution has been converted for through images by the resolution conversion unit 18 is also supplied to a CPU 29 that executes a predetermined image recognition processing program. For example, face detection is performed by image pattern detection, that is, the face area of each subject is extracted from the image frame. The gist of the present invention is not limited to a specific face detection process. For example, the face detection processing algorithm described in “Learning a real-time arbitrary posture face detector using pixel difference features” (10th Image Sensing Symposium (SSII 2004)) can be applied.

  In face detection, the entire image after resolution conversion is scanned. However, when a large number of subjects are included in the image, it is difficult to perform correction and smile detection processing on all detected faces. Therefore, priorities may be assigned to the detected faces, and detected faces with higher priorities may be designated as smile detection processing targets.

  As a priority setting method, for example, a score indicating the importance of the subject is attached based on information such as the fact that the face is close to the center or the face area is large, and the priority is increased in descending order of score. Good.

  In order for the photographer to know the smile detection face among the faces that are currently being detected, it is preferable that the subject that is the subject of smile detection and the subject that is not the target are displayed separately on the through image. In order to distinguish between a detection target and a non-target subject, for example, a method of displaying a frame only on the detection face of the detection target is conceivable. FIG. 6 shows an example of a through image in which a frame is displayed only on the detection face to be detected.

  FIG. 7 shows a processing procedure for displaying a frame on the detection face to be detected in the form of a flowchart. Here, it is assumed that up to N subjects can be smile detection targets (where N is a positive integer).

  First, the camera image input from the optical system 10 is displayed on the monitor display 25 according to the same data flow as in FIG. 2 (step S21). At this time, the displayed through image information is also supplied to the control unit 27.

  The CPU 29 in the control unit 27 is executing a predetermined face detection program, and performs face detection from the information of the through image (step S22). Here, when M faces are detected (where M is a positive integer and M> N), the importance for each detected face is determined, and priority is given to each detected face based on the determination result. Ranking is given (step S23).

  For detected faces with priorities of 1 to N, a GUI process for smile detection, that is, a frame is OSD displayed on the detected face (step S24). On the other hand, a normal GUI process is performed for the detected faces with the priority orders N + 1 to M (step S25).

  In addition, as another method for distinguishing and displaying a detection target and a non-target subject, a frame is displayed on all detected faces, but only the detection face selected as a smile detection target is displayed in the form (color and thickness). And a method of changing an attribute of a line such as a line type (see FIG. 8) or adding an icon to a frame of a detection face to be detected (see FIG. 9). . In the example shown in FIG. 8, two subjects from the left in the front row are selected as smile detection targets, but only these detection faces are surrounded by a thick line frame, and other detection faces are surrounded by a thin dotted line frame. ing. In the example shown in FIG. 9, two subjects from the left in the front row are selected as smile detection targets, but an icon indicating that they are selected is added only to these detected faces.

  In this way, there are two possible specifications for the frame display method. One is a method of displaying a frame by selecting only the top person with the changed priority for the number of smile detection targets automatically selected. The other is a method in which all faces surrounded by a region are set as new smile detection targets. In the latter case, when all the detected faces are surrounded by a frame, the user (photographer) can not only check the detected faces that are the smile detection targets, but also all the subjects whose faces are detected by the system from the input image. It can also be confirmed.

(2) Smile detection processing There are several possible cases for smile detection specifications. For example, smile detection can be performed for a plurality of detected faces, and the plurality of detection results can be statistically processed. Here, for the sake of simplification of explanation, attention is paid to one face with the highest priority among the detected faces detected above (assuming that the subject is “Mr. A”), and automatically when the person laughs. The description will be given according to the specifications for shooting.

  The smile detection process is described in the same algorithm as the face detection process described above, for example, “Learning a real-time arbitrary posture face detector using pixel difference features” (10th Image Sensing Symposium (SSII 2004)). A face detection processing algorithm can be applied. However, in this case, the dictionary of the image to be referenced is different.

  When a smile is detected on the detected face of the subject of interest, shooting processing such as resolution conversion for shooting the input image, JPEG image compression, and recording of the encoded compressed image on a recording medium is started.

  As described above, when M faces are detected from the input image, N persons are selected as smile detection targets by prioritization (provided that M and N are positive integers, and M ≧ N) (described above). Here, when N is 2 or more, that is, when there are a plurality of subjects for smile detection, shooting is automatically performed when at least one smile is detected, or the smiles of all selected N people are smiled. It is possible to consider a specification such that a picture is automatically taken when a camera is detected. In any case, who is the target N person is determined in the priority order.

  In the present specification, only the case where the photographing process is activated based on smile detection will be described. Of course, it is also conceivable to perform processes other than photographing based on the smile detection result.

(3) Selection of subject In the above description, when a plurality of subjects are included in the image frame input from the image sensor, the subject on which smile detection is performed is automatically selected on the system side. Yes. The degree of importance is determined for each detected face, priorities are assigned based on the determination results, and the detected face with the highest priority is designated as a smile detection target.

  The photographer can designate a subject as a smile detection target via the through image displayed on the monitor / display 25. As the designation method, there are a method of directly instructing a subject using a fingertip or a stylus on a display screen of a monitor display 25 configured as a touch panel, a method of selecting using a cross key provided in the UI unit 28, and the like. . Hereinafter, each method will be described.

  Here, the operation for changing the automatically selected smile detection target face to another face will be described using the input image shown in FIG. 10 as an example. In the same figure, when the detected face surrounded by the frame at the center of the front row in the image is automatically selected as the smile detection target according to the priority order, the subject on the right in the front row is to be changed to the smile detection target. To do.

  As shown in FIG. 11, a touch operation is performed on the vicinity of the face to be changed on the screen of the monitor display 25 constituting the touch panel. Then, the touched coordinates are recognized by position detection. Subsequently, face detection is performed in the vicinity of the coordinates by image pattern detection.

  At this time, if the face is detected, the priority of the detected face is changed to the highest. Accordingly, the priorities of the other detected faces having higher priority than the detected face are lowered by one each.

  Then, when the OSD display process is performed, a frame creation process is performed on the face whose priority has been changed to the highest level based on the new priority change information, and the frame is displayed on the monitor display 25. .

  In the above description, the case where face detection is successful by touching has been described. However, when face detection cannot be performed at the touched position, change processing is not performed.

  In addition, an operation method of selecting a subject for smile detection using the up / down / left / right keys provided on the UI unit 28 instead of the touch panel is also conceivable. Hereinafter, such an operation method will be described.

  In this case, when the face detection process ends, an operation order for key operation is assigned to each detected face. FIG. 12 illustrates this state. In the example shown in the figure, the horizontal scanning of the image is repeated from top to bottom, and the operation order is assigned to each detected face from left to right and from top to bottom.

  When the user operates the up / down / left / right key, the detected face to be processed is changed by moving the operation order back and forth in accordance with the key operation from the currently selected detected face.

  FIG. 13 shows a screen transition example when the detection face to be processed is changed according to the key operation of the up / down / left / right keys. In the figure, the operated keys are indicated by diagonal lines. After the operation order is assigned to each detected face, a mode for changing the face is entered when the center determination key is pressed. Then, the face selection is moved so that the operation order is incremented by 1 with the left key, and the face selection is moved so that the operation order is decremented by 1 with the right key. When the movement is finished, the smile detection target face is determined again with the enter key.

(4) Processing when there are a plurality of smile detection target faces In the description so far, the detection face to be a smile detection target is limited to only one person, but a plurality of detection faces are smile detection targets. It can be.

  FIG. 14 shows an example of screen transition when changing the detected face of the smile detection target when the number of detected faces to be the smile detection target is three.

  First, after performing face detection, priorities are set according to an algorithm such as assigning a score representing the importance level of the subject based on information such as the fact that the face area is close to the center or the face area is large. Up to three subjects are selected as detection targets, and these detection faces are surrounded by a frame on the screen on the monitor display 25 as shown on the left side of the figure.

  At this time, as shown in the center of the figure, the user (photographer) instructs the vicinity of the face of the subject that is not selected as the detection target but wants to be the detection target.

  In response to such an instruction operation, the priority order of the subject desired by the user is changed to the highest priority, and the priority order of each subject that has been higher in priority than the subject is decremented by one each. To do. As a result, the subject having the third priority moves down to the fourth position and is excluded from the smile detection target.

  Then, subjects with the top three new priorities are selected as detection targets, and these detection faces are surrounded by a frame on the screen on the monitor display 25 as shown on the right side of the figure. Compared with the left side of the figure, the leftmost subject in the rear row has been ranked as the highest priority according to the user's touch operation, and the subject in the middle of the rear row has been removed from the detection target and surrounded the detection face. The frame has disappeared.

(5) Subject selection method In the above description, an example in which a subject is selected one by one by touch panel or left / right key operation has been described. Can also be specified.

  FIG. 15 shows an example of screen transition in such a case. Here, it is assumed that two faces are detected as smile detection targets, and frames are displayed for the number of smile detection targets that are automatically selected.

  A user (photographer) performs a touch panel operation that surrounds a face to be specified as a detection target via the touch panel. Then, when an area surrounded by touch coordinates is recognized by position detection, face detection is subsequently performed in the area by image pattern detection.

  If a face image can be detected from the touched area, the priority order of the face is changed to the highest level for the detected face. At this time, if there is one face detected from the same area, that person is given the first priority. Further, when there are a plurality of detected faces, priority is assigned by a predetermined algorithm among them. When the number of detected faces exceeds two (four in the example shown in the figure), the priority is increased for the detected faces with the second highest priority in the same area, but the third and subsequent detected faces are increased. Absent. As the priority is raised, the priority of each subject that has a higher priority than those subjects is lowered.

  Then, when the OSD display process is performed, a frame creation process is performed on the face that has been changed to the smile detection target based on the new priority change information, and a frame is formed on the through image on the monitor display 25. Display.

  In the example shown in FIG. 15A, two subjects from the left in the front row are selected as smile detection targets.

  At this time, as shown in FIG. 15B, the user (photographer) performs an operation to surround four faces from the left in the back row via the touch panel.

  When all the faces surrounded by the region are set as new smile detection targets, as shown in FIG. 15C, the smile detection targets are changed from the left to the left two subjects in the front row to the four detection faces from the left in the back row. Switches. Along with this, a frame surrounding each of the four detected faces from the left in the rear row is displayed, and a frame surrounding the two detected faces from the left in the front row disappears.

  Further, when switching smile detection targets for the number of smile detection targets, that is, only two people, as shown in FIG. 15D, the priority is set higher by a predetermined algorithm from the four people surrounded by the touch panel operation. Only the third and fourth names from the left in the back row are selected as smile detection targets, and a frame surrounding the third and fourth detection faces from the left in the back row is displayed. As a result, the frame surrounding the two detected faces from the left in the front row disappears.

  FIG. 16 shows a processing procedure for switching a detection face to be a smile detection target according to a user's touch panel operation in the form of a flowchart.

  First, face detection processing is performed on the input image on the system side, the importance level of each detected face is determined and prioritized, and a predetermined number of people from the top are automatically selected as smile detection targets (step S31).

  Each selected detection face is framed on the through image. The user (photographer) confirms the subject on which the smile is detected on the monitor / display 25. Here, when it is desired to change the smile target, a touch panel operation is performed so as to surround the face to be designated as the detection target (step S32).

  Then, when a region surrounded by touch coordinates is recognized by position detection, face detection is subsequently performed in the region by image pattern detection (step S33). Here, if a face image can be detected from the touch-operated area, the priority of the face is raised with respect to the detected face (step S34). As the priority is raised, the priority of each subject that has a higher priority than those subjects is lowered.

  Then, when the OSD display process is performed, a frame creation process is performed on the face whose priority has been changed to the highest level based on the new priority change information, and a frame is displayed on the through image on the monitor display 25. Is displayed (step S35).

  As described above, according to the digital still camera 1 according to this embodiment, the photographer can freely select a subject for which a smile is detected and a smile is desired to be taken. For example, a smile detection target subject can be arbitrarily designated on a touch panel that displays a through image, and a subject that is a smile detection target and a non-target subject can be distinguished and displayed on the through image. . Therefore, for example, when shooting a scene such as a party, when a relatively large number of people is included in the angle of view, automatic shooting based on smile detection is performed under the condition that two or more members are laughing at the same time. It can be suitably performed.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiment without departing from the gist of the present invention.

  In this specification, the embodiment applied to the digital still camera has been mainly described, but the gist of the present invention is not limited to this. For example, the present invention can be similarly applied to various information devices having a mobile phone, a PDA (Personal Digital Assistant), or other camera functions.

  In this specification, only the case where the photographing process is started based on smile detection has been described. Of course, it is also conceivable to perform processes other than photographing based on the smile detection result.

  In short, the present invention has been disclosed in the form of exemplification, and the description of the present specification should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

DESCRIPTION OF SYMBOLS 1 ... Digital still camera 10 ... Optical system 10a ... Driver 11 ... Image sensor 12 ... AFE part 13 ... CDS / AGC / ADC block 14 ... Timing generator 15 ... V driver 16 ... Camera DSP part 17 ... Camera signal processing part 18 ... Resolution conversion unit 19 ... Image codec processing unit 20 ... Memory control unit 21 ... Display control unit 22 ... Media control unit 23 ... Bus interface unit 24 ... Memory device 25 ... Monitor / display 26 ... Recording media 27 ... Control unit 28 ... UI unit 29 ... CPU
30 ... RAM
31 ... EEPROM
32 ... Flash light emitting part

Claims (11)

Image input means for inputting an image formed by imaging a subject;
Signal processing means for performing signal processing on the input image;
Display means for displaying the image signal-processed by the signal processing means, and a display screen having a touch panel;
Detecting means for detecting a face of a subject from the image signal-processed by the signal processing means;
Setting means for setting a detection target for detecting a specific pattern of the face among the detection faces detected by the detection means;
Control means for changing a detection face detected in the vicinity of coordinates touched by the user on the touch panel to a detection target of the pattern;
Equipped with,
When the detecting means detects faces of a plurality of subjects from the input image, the setting means determines the importance for each detected face based on the position of each detected face in the image or other information, and the importance A priority is assigned to each detected face according to the degree, and a predetermined number of detected faces from the higher priority are set as detection targets of the pattern,
An image processing apparatus. Image input means for inputting an image formed by imaging a subject;
Signal processing means for performing signal processing on the input image;
Display means for displaying the image signal-processed by the signal processing means, and a display screen having a touch panel;
Detecting means for detecting a face of a subject from the image signal-processed by the signal processing means;
Setting means for setting a detection target for detecting a specific pattern of the face among the detection faces detected by the detection means;
Control means for changing a detection face detected in the vicinity of coordinates touched by the user on the touch panel to a detection target of the pattern;
A direction operation key for instructing a direction on the display screen;
Comprising
The control means assigns an operation order to each of a plurality of detected faces detected from the image, assigns priorities based on the importance of each detected face, and detects a predetermined number from the higher priority order. After setting the target, the priority of the detection face specified in accordance with the user operation order with respect to the direction operation key is raised and changed to the detection target of the pattern.
An image processing apparatus. Image input means for inputting an image formed by imaging a subject;
Signal processing means for performing signal processing on the input image;
Display means for displaying the image signal-processed by the signal processing means, and a display screen having a touch panel;
Detecting means for detecting a face of a subject from the image signal-processed by the signal processing means;
Setting means for setting a detection target for detecting a specific pattern of the face among the detection faces detected by the detection means;
Control means for changing a detection face detected in the vicinity of coordinates touched by the user on the touch panel to a detection target of the pattern;
Comprising
The control means assigns a priority based on the importance of each of the plurality of detected faces detected from the image, and after setting a predetermined number as a detection target of the pattern from the higher priority, the touch panel The detection target of the pattern is changed by raising the priority of the detection face specified by the user via
An image processing apparatus. Image input means for inputting an image formed by imaging a subject;
Signal processing means for performing signal processing on the input image;
Display means for displaying the image signal-processed by the signal processing means, and a display screen having a touch panel;
Detecting means for detecting a face of a subject from the image signal-processed by the signal processing means;
Setting means for setting a detection target for detecting a specific pattern of the face among the detection faces detected by the detection means;
Control means for changing a detection face detected in the vicinity of coordinates touched by the user on the touch panel to a detection target of the pattern;
Comprising
The control means assigns a priority based on the importance of each of the plurality of detected faces detected from the image, and after setting a predetermined number as a detection target of the pattern from the higher priority, the touch panel The priority of the detected face in the area designated by the user on the screen is raised to change the detection target of the pattern,
An image processing apparatus. Pattern degree level detecting means for detecting a pattern degree level of a detected face which is a detection target of the pattern;
Shooting control means for controlling the shooting process in response to the pattern level detected by the pattern level detection means reaching the pattern detection level;
The image processing apparatus according to any one of claims 1 to 4, further comprising a. The control means displays a through image with a frame on the detection face which is the detection target of the pattern;
The image processing apparatus according to claim 1 , wherein the image processing apparatus is an image processing apparatus. The control means attaches a frame to all detected faces detected by the detecting means, and changes the display format of the frame for the detected face that is the detection target of the pattern or attaches a predetermined icon to display a through image. Display,
The image processing apparatus according to any one of claims 1 to 4, characterized in that. In the case where two or more detected faces are the detection targets of the pattern, the photographing control means automatically performs photographing when at least one smiling face is detected, or when the smiling faces of all selected persons are detected. Shooting automatically,
The image processing apparatus according to claim 5. The control means limits the number of subjects that can be the detection target of the pattern to N, and when the detection area includes more than N detection faces, each of the detection faces in the area is detected. Priority is assigned based on importance, and detection faces for N people from the higher priority are set as detection targets of the pattern,
The image processing apparatus according to claim 4 . An image input step for inputting an image formed by imaging a subject;
A signal processing step for signal processing the input image;
Displaying the image signal-processed in the signal processing step on a display screen having a touch panel; and
A detection step of detecting a face of a subject from the image signal-processed in the signal processing step;
A setting step of setting a detection target for detecting a specific pattern of the face among the detection faces detected in the detection step;
A control step of changing a detected face detected in the vicinity of coordinates touched by the user on the touch panel to a detection target of the pattern;
Have
When detecting the faces of a plurality of subjects from the input image in the detection step, the setting step determines the importance for each detected face based on the position of each detected face in the image or other information. A priority is assigned to each detected face according to the degree, and a predetermined number of detected faces from the higher priority are set as detection targets of the pattern,
An image processing method. A computer program written in a computer-readable format so as to execute processing of an input image formed by imaging a subject on a computer having a display means whose display screen also has a touch panel.
Image signal processing means for performing signal processing on the input image;
Detecting means for detecting a face of a subject from the input image after signal processing by the image signal processing means;
Setting means for setting a detection target for detecting a specific face pattern among the detected faces detected by the detection means;
Control means for changing a detected face detected in the vicinity of coordinates touched by the user on the touch panel to a detection target of the pattern;
Function as
When the detecting means detects faces of a plurality of subjects from the input image, the setting means determines the importance for each detected face based on the position of each detected face in the image or other information, and the importance A priority is assigned to each detected face according to the degree, and a predetermined number of detected faces from the higher priority are set as detection targets of the pattern,
Computer program.