JP2009094946A - Image pickup device and portrait right protection method in the same device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent the invasion of the privacy or portrait right of a third person when disclosing a photographic image in an image pickup device. <P>SOLUTION: This image pickup device for acquiring image data by imaging an object, and for detecting a prescribed object from the acquired image data is configured to determine whether or not each of one or more detected objects (K1 to K4) is valid or not based on information including at least one of the size, position, change quantity and focusing evaluation value of the object, and applies portrait right protection processing to the object (for example, K4) determined as an invalid object. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an imaging apparatus, and more particularly to a portrait right protection method for preventing infringement of privacy and portrait rights in a captured image.

  Conventionally, various image processing methods for detecting a specific object such as a face from image data acquired by an imaging device such as a digital camera and using the detected object have been proposed. An image processing apparatus that detects a face area from data and reproduces an image paying attention to the detected face area is disclosed (Patent Document 1).

  On the other hand, when a subject is photographed by an imaging device such as a digital camera, in addition to the subject desired by the photographer, for example, the face of a person unintended by the photographer, such as the face of a person who accidentally passes, is photographed as the subject. Sometimes.

  However, in recent years, the awareness of privacy and protection of portrait rights has increased, so when releasing images obtained by shooting, privacy and protection of portrait rights against third parties appearing in the images are desired. .

Therefore, human gestures and actions are detected from the acquired images, the detected gestures and actions are analyzed, and the optimal abstraction according to user requests and usage scenes is determined for the image area determined according to the analysis results. An imaging device for monitoring (Patent Document 2) that performs processing, or an image processing device that reconstructs a structure by changing parameters of feature points of the structure so that a structure such as a face cannot be specified in an image (Patent) Document 3), an image providing apparatus (Patent Document 4) that automatically arranges a sensor in an area that is not desired to be imaged, automatically performs mosaic processing, and provides an image subjected to mosaic processing has been proposed.
JP 2006-293882 A JP 2006-295251 A JP 2006-318377 A JP 2005-026917 A

  However, since the monitoring imaging device analyzes gestures and actions, it is difficult to determine a person who the photographer does not want from people who perform the same operation, for example. In the image processing apparatus and the image providing apparatus, when determining an object to be subjected to image processing, for example, a person who accidentally passes or a person who has unintentionally entered the field of view is considered. It is difficult.

  The present invention has been made in view of such circumstances, and an imaging apparatus capable of preventing infringement of privacy and portrait rights to a third party when a photographed image is disclosed, and a portrait right protection method in the imaging apparatus. It is intended to provide.

An imaging apparatus according to the present invention includes an imaging unit that captures an image of a subject and acquires image data;
Object detection means for detecting a predetermined object from the image data;
Subject determination means for determining whether each of the one or more objects detected by the object detection means is an effective subject or an invalid subject;
Portrait right protection means for protecting the right of portrait of the object determined to be an invalid subject by the subject determination means is provided.

  In the imaging apparatus according to the aspect of the invention, the subject determination unit may be configured based on information including one or more of the size, position, change amount, and focus evaluation value of the target detected by the target detection unit. It is preferable to make a judgment.

In the imaging apparatus according to the present invention, when the number of the objects that are determined to be effective objects by the object determining unit is larger than the number of objects that are arbitrarily set, the various objects are invalidated from the effective objects in the order of low priority. Move down to the subject,
When the number is less than the number, it is preferable to include effective subject changing means for moving the invalid information from the invalid subject to the effective subject in descending order of priority.

  In the imaging apparatus according to the present invention, the portrait right protection means may perform a mosaic process on the object determined to be the invalid subject.

  In the image pickup apparatus of the present invention, the portrait right protection means may perform a blurring process on the object determined to be the invalid subject.

  In the image pickup apparatus of the present invention, the portrait right protection means may perform a trimming process on the object determined to be the invalid subject.

  Further, the imaging apparatus of the present invention may warn the portrait right protection means when there is the object determined to be the invalid subject.

  In the imaging apparatus of the present invention, the portrait right protection means may make it impossible to print image data including the object when there is the object determined to be the invalid subject. .

  The image pickup apparatus of the present invention can include a plurality of types of portrait right protection means, and a selection means for selecting and using one of them.

According to the portrait right protection method of the present invention, an imaging device that captures an image of a subject, acquires image data, and detects a predetermined object from the acquired image data.
For one or more detected objects, it is determined whether the object is an effective subject or an invalid subject based on information including one or more of the size, position, change amount, and focus evaluation value of the object. And
Portrait right protection processing is performed on the object determined to be an invalid subject.

In addition, the portrait right protection method of the present invention arbitrarily sets the number of objects as subjects,
When the number of the objects determined to be the effective subject is larger than the number of the set objects, the effective subject is moved down from the effective subject to the invalid subject in the order of the low priority of the various information,
When the number is less than the number, the invalid subject can be moved from the invalid subject to the valid subject in order of the priority of the various information.

  According to the imaging device and the portrait right protection method in the imaging device of the present invention, it is determined whether each of the one or more objects detected from the image data acquired by imaging the subject is an effective subject or an invalid subject, Portrait rights protection processing is performed on objects that are judged to be invalid subjects, so portrait rights are automatically protected against third parties by determining subjects that the photographer does not want, ie third parties, as invalid subjects. Since the processing is performed, it is possible to prevent infringement of privacy and portrait rights with respect to a third party when releasing the photographed image.

  In addition, when the number of objects to be used as subjects is arbitrarily set and the number of objects judged to be effective subjects is larger than the set number of objects, the priority of various types of information on the size, position, change amount, and focus evaluation value of the objects The effective subject is decremented from the effective subject to the invalid subject in descending order, and the effective subject detection accuracy is increased by moving from the invalid subject to the valid subject in descending order of the priority of the various information when the number is smaller than the set number of objects. be able to.

  Hereinafter, an embodiment of an imaging apparatus according to the present invention will be described in detail with reference to the drawings. In the following embodiments, a digital camera will be described as an example of the imaging apparatus in the present invention. However, the scope of the present invention is not limited to this, and for example, a security camera, a mobile phone with a camera, a PDA with a camera, etc. It can also be applied to other electronic devices having an electronic imaging function.

  1 and 2 show an example of a digital camera, and are perspective views seen from the back side and the front side, respectively. As shown in FIG. 1, an operation mode switch 11, a menu / OK button 12, a zoom / up / down arrow lever 13, a left / right arrow button 14, A Back button 15 and a display switching button 16 are provided, and a finder 17 for photographing and a liquid crystal monitor 18 for photographing and reproduction are further provided. A shutter button 19 is provided on the upper surface of the main body 10.

  The operation mode switch 11 is a slide switch for switching operation modes of a still image shooting mode, a moving image shooting mode, and a playback mode.

  Each time the menu / OK button 12 is pressed, the LCD monitor 18 displays various menus for setting the shooting mode, flash emission mode, portrait right protection mode, self-timer ON / OFF, recording pixel number, sensitivity, and the like. Or a button for confirming selection / setting based on a menu displayed on the liquid crystal monitor 18. The portrait right protection mode is a mode for preventing privacy and portrait right infringement using the face detection function.

  The zoom / up / down arrow lever 13 tilts the lever in the up / down direction to adjust the telephoto / wide angle during shooting, and moves the cursor in the menu screen displayed on the liquid crystal monitor 18 during various settings to display it. It is a lever for. The left and right arrow buttons 14 are buttons for moving the cursor in the menu screen displayed on the liquid crystal monitor 18 at the time of various settings to be displayed left and right.

  The Back button 15 is a button for stopping various setting operations when pressed and displaying the previous screen on the liquid crystal monitor 18. The display switching button 16 is a button for switching ON / OFF of display on the liquid crystal monitor 18, various guide displays, ON / OFF of character display, and the like when pressed.

  The viewfinder 17 is used by the photographer to look into the subject for composition and focus when photographing the subject. The subject image seen from the finder 17 is projected through the finder window 23 on the front surface of the main body 10.

  The contents set by the operation of each button and switch described above can be confirmed by the display on the liquid crystal monitor 18, the position of the lamp in the finder 17, the position of the slide lever, and the like.

  The liquid crystal monitor 18 functions as an electronic viewfinder by displaying a through image for checking a subject at the time of shooting, and also displays a still image and a moving image after shooting and displays various setting menus.

  Further, as shown in FIG. 2, a photographing lens 20, a lens cover 21, a power switch 22, a finder window 23, a flash light 24 and a self-timer lamp 25 are provided on the front surface of the main body 10, and a media slot 26 is provided on the side surface. Is provided.

  The photographic lens 20 is for forming a subject image on a predetermined image plane (CCD or the like inside the main body 10), and includes a focus lens, a zoom lens, and the like. The lens cover 21 covers the surface of the photographic lens 20 when the power of the digital camera 1 is turned off or in the playback mode, and protects the photographic lens 20 from dirt and dust.

  The power switch 22 is a switch for switching on / off the power of the digital camera 1. The flashlight 24 is used to instantaneously irradiate the subject with light necessary for photographing while the shutter button 19 is pressed and the shutter inside the main body 10 is opened.

  The self-timer lamp 25 is used to notify the subject of the opening / closing timing of the shutter, that is, the start and end of exposure when shooting with the self-timer. The media slot 26 is a filling port for filling an external recording medium 70 such as a memory card. When the external recording medium 70 is filled, data reading / writing is performed.

  FIG. 3 is a block diagram showing the functional configuration of the digital camera 1. As shown in FIG. 3, as the operation system of the digital camera 1, the above-described operation mode switch 11, menu / OK button 12, zoom / up / down arrow lever 13, left / right arrow button 14, Back (return) button 15, and display switching button 16 are provided. The shutter button 19 and the power switch 22 and an operation system controller 74 which is an interface for transmitting the operation contents of these switches, buttons and levers to the CPU 75 are provided.

  Further, a focus lens 20a and a zoom lens 20b are provided as components of the photographing lens 20. Each of these lenses is step-driven by a focus lens driving unit 51 and a zoom lens driving unit 52 including a motor and a motor driver, and is configured to be movable in the optical axis direction. The focus lens driving unit 51 step-drives the focus lens 20 a based on the focus driving amount data output from the AF processing unit 62. The zoom lens driving unit 52 controls step driving of the zoom lens 20 b based on the operation amount data of the zoom / up / down arrow lever 13.

  The diaphragm 54 is driven by a diaphragm driving unit 55 including a motor and a motor driver. The aperture driving unit 55 adjusts the aperture diameter of the aperture 54 based on aperture value data output from the AE (automatic exposure) processing unit 63.

  The shutter 56 is a mechanical shutter and is driven by a shutter drive unit 57 including a motor and a motor driver. The shutter drive unit 57 controls the opening and closing of the shutter 56 according to a signal generated by pressing the shutter button 19 and the shutter speed data output from the AE processing unit 63.

  At the back of the optical system, there is a CCD 58 that is a photographing element. The CCD 58 has a photoelectric surface in which a large number of light receiving elements are arranged in a matrix, and subject light that has passed through the optical system is imaged on the photoelectric surface and subjected to photoelectric conversion. In front of the photocathode, a microlens array (not shown) for condensing light on each pixel and a color filter array (not shown) in which RGB filters are regularly arranged are arranged. Yes.

  The CCD 58 reads out the charge accumulated for each pixel line by line in synchronization with the vertical transfer clock signal and horizontal transfer clock signal supplied from the CCD control unit 59 and outputs them as a serial analog image signal. The charge accumulation time in each pixel, that is, the exposure time, is determined by an electronic shutter drive signal given from the CCD controller 59. An imaging unit (imaging means) 100 is configured including the optical system, the CCD 58, and the CCD control unit 59.

  The analog image signal output from the CCD 58 is input to the analog signal processing unit 60. The analog signal processing unit 60 includes a correlated double sampling circuit (CDS) that removes noise from the analog image signal, an auto gain controller (AGC) that adjusts the gain of the analog image signal, and converts the analog image signal into digital image data. It comprises an A / D converter (ADC) for conversion. The digital image data converted into the digital signal is CCD-RAW data having RGB density values for each pixel.

  The timing generator 72 generates a timing signal. The timing signal is input to the shutter drive unit 57, the CCD control unit 59, and the analog signal processing unit 60, and the operation of the shutter button 19 and the opening / closing of the shutter 56 are performed. The charge capturing of the CCD 58 and the processing of the analog signal processing 60 are synchronized. The flash control unit 73 controls the light emission operation of the flash 24.

  The image input controller 61 writes the CCD-RAW data input from the analog signal processing unit 60 in the frame memory 66. The frame memory 66 is a working memory used when various digital image processing (signal processing) described later is performed on image data. For example, an SDRAM that performs data transfer in synchronization with a bus clock signal having a certain period. (Synchronous Dynamic Random Access Memory) is used.

  The display control unit 71 is for displaying the image data stored in the frame memory 66 on the liquid crystal monitor 18 as a through image. For example, the display control unit 71 combines a luminance (Y) signal and a color (C) signal together. The signal is converted into a composite signal and output to the liquid crystal monitor 18. Through images are acquired at predetermined time intervals and displayed on the liquid crystal monitor 18 while the shooting mode is selected. In addition, the display control unit 71 causes the liquid crystal monitor 18 to display an image based on the image data stored in the external recording medium 70 and read out by the media control unit 67.

  A face detection unit (object detection means) 69 detects a human face from image data stored in the frame memory 68, that is, a through image. Specifically, an area having facial features included in the face (for example, having a skin color, having eyes, or having a face shape) is detected as a face area, and the face is detected by matching both-eye image data. The face may be detected from the positional relationship between eyes and mouth, or the face area may be specified by detecting the outline of the face, detecting color information corresponding to the skin of the face, etc. Alternatively, a known technique can be used.

  When a face is detected by the face detection unit 69 in the through image, a face detection frame K surrounding the detected face is displayed on the through image (see FIG. 6). Further, the face detection unit 69 may set face detection ON / OFF using a face detection button (not shown), for example, or may select face detection ON / OFF using the menu / OK button 12 or the like. It can be changed as appropriate. When the portrait right protection mode is selected, face detection is set to ON.

  In the present embodiment, the following description will be made assuming that a person's face is detected. However, another object such as a special building may be detected.

  The face determination unit (subject determination unit) 80 determines whether each of the one or more faces detected by the face detection unit 69 is an effective face (effective subject) or an invalid face (invalid subject). That is, it is determined whether or not the face detected by the face detection unit 69 is a subject desired by the photographer. At this time, in this embodiment, the determination is made based on information on the size, position, change amount, and focus evaluation value of the face. The present invention is not limited to this. For example, the determination may be made based on only one of the various types of information, or some of the various types of information may be selected, and the selected information may be used based on the selected information. The determination may be made. Further, the determination may be made by adding information different from the above-mentioned various information, and can be changed as appropriate. The determination by the face determination unit 80 will be described in detail later.

  The effective face changing unit (effective subject changing unit) 81 changes the number of faces determined to be effective faces by the face determining unit 80, and the number of effective faces is determined by the user, for example, the menu / OK button 12, When the number of faces is arbitrarily set by operating the zoom / up / down arrow lever 13, left / right arrow button 14, etc., the valid face is changed to the invalid face in descending order of priority of the various information. When the number of faces is less than the number of arbitrarily set faces, the invalid face is moved up from the invalid face to the effective face in the order of higher priority of the various information. This priority will be described later in detail.

  The portrait right protection unit (portrait right protection means) 82 protects the portrait right for the face determined to be an invalid face by the face determination unit 80. Specifically, there are things that perform mosaic processing, blurring processing or trimming processing on invalid faces, warn when there are invalid faces, or make it impossible to print image data containing invalid faces, A plurality of types of portrait right protection units 82 may be provided, or one type of portrait right protection unit 82 may be provided. The portrait right protection unit 82 will be described in detail later.

  The AF processing unit 62 detects a focus position based on the image data, determines a focus setting value (focus drive amount), and outputs focus drive amount data (AF processing). As a focus position detection method, for example, a passive method is used in which a focus position is detected by using a feature that an in-focus evaluation value (contrast value) of an image increases in a focused state.

  The AE processing unit 63 measures subject luminance (photometric value) based on the image data, determines an exposure setting value such as an aperture value and shutter speed based on the measured subject luminance, and obtains the aperture value data and shutter speed data. Output (AE processing).

  The AWB processing unit 64 automatically adjusts the white balance at the time of imaging (AWB processing). Note that the AWB processing unit 64 can adjust the white balance before and after imaging.

  An image processing unit (image processing means) 68 performs gamma correction processing, contour enhancement (sharpness) processing, contrast processing, noise reduction processing, and other image quality correction processing on the image data of the main image, and converts the CCD-RAW data. YC processing is performed for conversion into YC data composed of Y data that is a luminance signal, Cb data that is a blue color difference signal, and Cr data that is a red color difference signal.

  The main image is taken in from the CCD 58 when a photographing instruction is given by the photographer, for example, when the shutter button 19 is fully pressed, and stored in the frame memory 66 via the analog signal processing unit 60 and the image input controller 61. Acquiring the actual image is referred to as actual photographing. The upper limit of the number of pixels of the main image is determined by the number of pixels of the CCD 58. For example, the number of recorded pixels can be changed by image quality settings (settings such as fine and normal) that can be set by the photographer. On the other hand, the number of pixels of the through image or the like may be smaller than that of the main image.

  The compression / decompression processing unit 65 performs compression processing, for example, in a compression format such as JPEG on the image data of the main image that has been subjected to processing such as image quality correction by the image processing unit 68, and generates an image file. Based on the Exif format or the like, a tag that stores additional information such as the shooting date and time and whether the image was shot in the portrait right protection mode is added to the image file. In the playback mode, the compression / decompression processing unit 65 reads the compressed image file from the external recording medium 70 and performs decompression processing. The decompressed image data is output to the display control unit 71, and the display control unit 71 displays an image based on the image data on the liquid crystal monitor 18.

  The media control unit 67 corresponds to the media slot 26 in FIG. 2, and reads an image file or the like stored in the external recording medium 70 or writes an image file.

  The internal memory (storage means) 82 stores various constants set in the digital camera 1, programs executed by the CPU 75, and various types of face detection frames K when the shutter button 19 is half-pressed and fully pressed. The face detection tag in which information is stored is stored. At this time, the various information includes the position, size (area), focus evaluation value, and the like of the face detection frame K. Here, the focus evaluation value is a value obtained by filtering the image data in the inner region of the face detection frame K to obtain a high frequency component and integrating the absolute value of the high frequency component. Further, the size of the face detection frame K, that is, the area of the internal region is calculated based on the number of pixels, for example. The position calculation method will be described later in detail.

  The CPU 75 controls each part of the main body of the digital camera 1 in accordance with operations of various buttons, levers, switches, and signals from the respective function blocks. When a plurality of types of portrait right protection units 82 are provided, it also functions as a selection unit (selection unit) that selects one of them. For example, the portrait right protection unit 82 that the user wants to use is set by operating the zoom / up / down arrow lever 13 or the left / right button 14, and which portrait right protection unit 82 is set by the CPU 75 as the selection unit. By determining whether or not, the portrait right protection unit 82 set is selected.

  The data bus 76 includes an image input controller 61, various processing units 62 to 65, 68, a frame memory 66, various control units 67 and 71, a face detection unit 69, a face determination unit 80, an effective face change unit 81, and portrait right protection. The unit 82, the internal memory 77, and the CPU 75 are connected, and various signals and data are transmitted and received through the data bus 76.

  Next, a process performed at the time of photographing in the portrait right protection mode in the digital camera 1 having the above configuration will be described. 4 is a flowchart of a photographing process of the digital camera 1, FIG. 5 is a flowchart of a face determination process, FIG. 6 is a diagram illustrating an example of a captured image, and FIG. 7 is a diagram for explaining validity determination based on face position information. 8 is a diagram illustrating another example of a captured image.

  When the digital camera 1 is turned on, as shown in FIG. 4A, imaging of the subject is started (step S1), and the display control unit 71 performs display control of the through image. The display of the through image is a process of displaying the image data stored in the frame memory 68 on the liquid crystal monitor 18, and the face detection unit 69 always detects the face from the through image and is detected by the liquid crystal monitor 18. A face detection frame K surrounding the face is displayed.

  Next, the CPU 75 determines whether or not the shutter button 19 has been half-pressed (S1) (step S2). When the shutter button 19 is not half-pressed (step S2; NO), the CPU 75 repeats the process of step S2 until the shutter button 19 is half-pressed, and the shutter button 19 is half-pressed (S1). In this case (step S2; YES), the CPU 75 acquires the face detection tag (S1 face detection tag) of the face detected by the face detection unit 69 at S1 and stores it in the internal memory 77 (step S3). The AE processing unit 63 and the AF processing unit 62 perform AE processing and AF processing, respectively (step S4).

  Next, the CPU 75 determines whether or not the shutter button 19 has been fully pressed (S2) (step S5). If the shutter button 19 has not been fully pressed (step S5; NO), the CPU 75 It is determined whether or not half-pressing is released (step S6). If half-pressing is released (step S6; YES), the process proceeds to step S2. If the half-press of the shutter button 19 is not released (step S6; NO), the process proceeds to step S5, and the processes after step S5 are repeated.

  On the other hand, when the shutter button 19 is fully pressed (S2) in step S5 (step S5; YES), the CPU 75 detects the face detection tag (S2 face detection tag) of the face detected by the face detection unit 69 at S2. Is acquired and stored in the internal memory 77 (step S7), and the imaging unit 100 takes a real image of the subject (step S8). At this time, the image processing unit 68 may perform image processing on the main image acquired by the main photographing.

  Then, as shown in FIG. 4B, the face determination unit 80 performs a face determination process for determining whether the face detected by the face detection unit 69 at S2 is an effective face or an invalid face (step S9).

  In the face determination process, as shown in FIG. 5A, first, the number of S2 face detection tags (m), that is, the number of faces detected by the face detection unit 69 is counted (step S20). The area of K is calculated (step S21). In the present embodiment, the area within the face detection frame K is calculated as the face size. However, the present invention is not limited to this, and for example, the area inside the face outline may be used as an area, or may be changed as appropriate. Is possible. Then, based on the calculated area, the S2 face detection tags are sent to the processes after step S23 in order from the largest area (step S22).

  First, n is initialized (step S23), and then n + 1 is substituted for n (step 24). Next, it is determined whether n is larger than m (n> m) (step S25). If n is equal to or smaller than m (step S25: n> m; NO), n is 1 in addition. It is determined whether or not (n = 1) (step S26).

  If n is 1 (step S26: n = 1; YES), as shown in FIG. 5B, the face first sent by the process of step S22, that is, the face detected by the face detection unit 69 is detected. Among them, since the face has the largest area of the face detection frame K (the face in the face detection frame K1 in FIG. 6 or FIG. 8), the effectiveness determination based on the face size information in step S32 is not performed. It is determined whether or not the face position is valid (step S27).

  As a method for determining the effectiveness of the position of the face, the subject desired by the photographer is usually located at the center of the field of view, so that the face is first detected within the number of effective pixels of the digital camera 1. Also, an edge region that does not make the detected face an effective face is determined. In the present embodiment, for example, the number of effective pixels of the digital camera 1 is set to 3072 × 2304 pixels, and an area of, for example, 300 pixels is set as the edge area from each of the upper, lower, left, and right edges.

Then, the effective area in which the detected face becomes an effective face when the face is detected is an area that is not the edge area, as shown in FIG. 7, when the upper left of the image is set as the origin, the upper left of the effective area The coordinate point T300r and the lower right coordinate point T300s are calculated as in the following equations (1) and (2).

On the other hand, in the image data captured at S2, as shown in FIG. 6, when the face detection unit 69 detects, for example, four faces, face detection frames K1 to K4 that respectively surround the detected faces are displayed. At this time, the coordinates of the upper left R and the lower right L of each of the face detection frames K1 to K4 are, for example,

Then, the center points K1s to K4s of the face detection frames K1 to K4 are calculated by the following formulas (3) to (6) and positioned on the coordinates as shown in FIG. The center point Ks is stored in the face detection tag as position information of the face detection frame K and stored in the internal memory 77 at S1 and S2.

  As a result, the coordinates of K1s, K2s, and K3s are within the effective area, but the coordinates of K4s are within the edge area, and are sent in the descending order of area in step S22 of FIG. 5A. When the face detection tag is the face detection frame K1, K2, K3, the position of the face within this frame is valid as being located outside the edge region, and when the face detection tag is the face detection frame K4, The face position in the frame is determined to be invalid because it is located in the edge region.

  If it is determined that the position of the face is not valid (step S27; NO), this face is determined to be an invalid face, and an invalid face tag is added to the face detection tag for this face (step S31). ). On the other hand, if it is determined that the position of the face is valid (step S27; YES), it is next determined whether or not the amount of change in the face is valid (step S28).

  As a method for determining the effectiveness of the amount of change in the face, the subject to be photographed is usually waiting for being photographed and often does not move, so the amount of change in the face is larger than a predetermined threshold value. In such a case, the face is determined to be invalid, for example, because it is highly likely that the face of a person who has accidentally passed by or entered the field of view unintentionally, or a person that the photographer does not want, etc. Is smaller than a predetermined threshold value, it is determined that the photographer is likely to be a person who wants to be photographed and does not move, and is determined to be effective.

  Note that the amount of change of the face can be the difference Δ (R1, R2) between the face center points Ks1, Ks2 calculated at S1 and S2 and stored in the internal memory 77, and this difference Δ (R1, R2). When R1 and R2 are within the threshold value 10, for example, it is determined that the amount of change in the face is valid. Information on the amount of change in the face is added to the face detection tag and stored.

  If it is determined that the amount of change in the face is not valid (step S28; NO), it is determined that this face is an invalid face, and an invalid face tag is added to the face detection tag for this face (step S28). S31). On the other hand, when it is determined that the amount of change in the face is valid (step S28; YES), it is next determined whether or not the focus evaluation value of the face is valid (step S29).

  As a method for determining the effectiveness of the facial focus evaluation value, the photographer usually focuses on the desired subject, and thus the focused subject has a higher focus evaluation value. Therefore, when the focus evaluation value calculated at S2 and stored in the internal memory 77 is a predetermined threshold value (for example, 1) or more, there is a possibility that the face is in focus and desired by the photographer. If the focus evaluation value is smaller than a predetermined threshold (for example, 1), it is determined that the face is so-called out-of-focus and the possibility that the photographer wants the face is low. .

  If it is determined that the focus evaluation value of the face is not valid (step S29; NO), this face is determined to be an invalid face, and an invalid face tag is added to the face detection tag for this face. (Step S31). On the other hand, when it is determined that the focus evaluation value of the face is valid (step S29; YES), this face is determined to be an effective face, and an effective face tag is added to the face detection tag of this face. (Step S30).

  If it is determined in step S30 or S31 whether the face is an effective face or an invalid face, as shown in FIG. 5A, the process proceeds to step S24 and 1 is added to n (step S24). It is determined whether n is larger than m (n> m) (step S26). If n is equal to or smaller than m (step S25: n> m; NO), n is further 1 (n = 1) or not (step S26).

  Here, the face to be determined this time is the face sent in the second and subsequent steps in the process of step S22, that is, the face with the largest area of the face detection frame K among the faces detected by the face detection unit 69 ( In FIG. 6 or FIG. 8, a face other than the face in the face detection frame K1 (in FIG. 6 or FIG. 8, for example, the second is the face in the face detection frame K2, and the third is the face in the face detection frame K3). Therefore, it is determined in step S32 whether the face size is valid (step S32).

  As a method for determining the effectiveness of the size of the face, the subject desired by the photographer usually occupies a large area in the field of view. Therefore, when the face size calculated at S2 and stored in the internal memory 77 is larger than a predetermined threshold, it is determined that the photographer is likely to be the desired face, and the face size is determined to be valid. If the threshold value is smaller than the threshold value, it is determined that the face is not desired by the photographer and is invalid.

  The predetermined threshold value can be set to ½ of the area of the face detection frame K1 having the largest area, for example, as shown in FIG. That is, if the face detection frame K of the face to be determined with respect to the face detection frame K1 is 1/2 or more in area ratio, it can be determined to be effective. In this case, the size of the face in the face detection frame K2 in FIG. 8 is determined to be effective because area K2 / area K1> 0.5, and the size of the face in the face detection frame K3 is also determined as area K3 / area. Since K1> 0.5, it is determined to be effective. The size of the face in the face detection frame K4 is determined to be invalid because area K4 / area K1 <0.5.

  If it is determined that the face size is valid (step S32; YES), the process proceeds to step S27, and the subsequent processes are repeated. On the other hand, if it is determined that the face size is not valid (step S32; NO), this face is determined to be an invalid face, and an invalid face tag is added to the face detection tag for this face ( Step S31).

  If it is determined in step S30 or S31 whether the face to be determined is an effective face or an invalid face, the process proceeds to step S24 and 1 is added to n as shown in FIG. S24) It is determined whether n is larger than m (n> m) (step S25), and until n becomes larger than the number (m) of S2 face detection tags (step S25; NO), that is, the face The processes from step S26 are repeated until the determination of whether the face is an effective face or an invalid face is completed for all the faces detected by the detection unit 69.

  Then, n becomes larger than the number (m) of the S2 face detection tags (step S25; NO), that is, whether all the faces detected by the face detection unit 69 are valid faces or invalid faces is determined. When completed, the face determination process by the face determination unit 80 ends, and as shown in FIG. 4B, the CPU 75 moves the process to step S10, and the effective face change unit 81 performs the effective face change process (step S10).

  In the present embodiment, as the face determination process, the effectiveness is determined based on all of the face size, position, change amount, and focus evaluation value. However, the present invention is not limited to this. One or more values may be selected from the size, position, change amount, and focus evaluation value. Further, for example, validity determination processing based on the brightness in the face detection frame K, the sharpness of the face, or the like may be added immediately before step S27 in FIG. 5B and before step S30.

  The validity of the brightness in the face detection frame K can be determined using the L * average value in the face detection frame K. As a calculation method, an average value of RGB in each detected face area, that is, the face detection frame K is converted into an XYZ value, and an XYZ value is converted into L * a * b.

First, a digital RGB value is converted into a linear RGB value. When sRGB is assumed, the linear RGB value can be expressed by the following formula (7) with respect to the digital RGB value.

Then, the linear RGB value is converted into the XYZ value by the following equation (8).

  Next, the XYZ value is converted into L * a * b. X, Y, and Z are XYZ conversion values, and Xn, Yn, and Zn are D65.

After luminance linear conversion and XYZ conversion, Lab is calculated by the following equation (9).

  Then, when the L * average value in each face detection frame K calculated as described above is within a range of 40 to 70, for example, it is determined to be effective.

  In addition, the effectiveness of the face sharpness can be determined using a known evaluation method for the sharpness of an image using discrete Fourier transform. Here, FIG. 9A shows the distribution of frequency components of the captured image, and FIG. 9B shows the power spectrum of each spatial frequency as a gray image.

First, the amplitude of each period is obtained using the discrete Fourier transform formula shown in the following formula (10).

  The value obtained at this time is called a spectrum, and the value represented by the square of the absolute value of the amplitude is called a power spectrum. As shown in FIG. 9A, the distribution of frequency components of the actual image can be obtained. . Further, as shown in FIG. 9B, when the power spectrum of each spatial frequency in FIG. 9A is represented by a grayscale image, the vicinity of the center of the coordinate represents a low frequency and the higher the distance from the center, the higher the frequency.

  The determination of the face sharpness is invalidated when it is determined that the face sharpness is low when the face detected power spectrum has a power spectrum value of the set frequency fk of 0.1 or less, for example.

  Next, the effective face changing process in step S10 of FIG. 4B will be described. FIG. 10 is a flowchart of the effective face changing process. In the effective face changing process, as shown in FIG. 10, for example, an arbitrary number input by the user is set as the number of faces (m) (step S40). Next, the number (n) of valid faces determined by the face determination process in step S9 of FIG. 4B is counted (step S41).

  Then, it is determined whether or not the number of arbitrarily set faces (m) is larger than the number of effective faces (n) (step S42). If it is larger (step S42; YES), the invalid face tag is set. The added face detection tag is read (step S43), and the various information of the read face detection tag is moved up to the valid face in the descending order of priority (step S44), and the tag is changed from the invalid face to the valid face.

  Here, the priority of various types of information is that, in the face size information, the subject desired by the photographer usually occupies a large area in the field of view of the image. It is assumed that the priority is higher in descending order of area. In the face position information, since the subject desired by the photographer is usually located at the center of the field of view, for example, it is assumed that the priority is higher in the order of the face closer to the center in the image.

  Further, in the face change amount information, the subject to be photographed usually does not move because it is waiting to be photographed. Therefore, for example, it is assumed that the priority is in descending order of the change amount. In the focus evaluation value of the face, the photographer usually focuses on a desired subject. For example, the priority is set in descending order of the focus evaluation value.

  On the other hand, when the number of faces (m) arbitrarily set in step S42 is smaller than the number of effective faces (n) (step S42; NO), the face detection tag to which the valid face tag is added is read. (Step S45), the various information of the read face detection tag is moved down to the invalid face in the order of low priority (Step S46), and the tag is changed from the valid face to the invalid face.

  In step S42, when the arbitrarily set number of faces (m) is the same number (m = n) as the number of effective faces (n), the CPU 75 does not perform the subsequent processing, and FIG. The process is thereafter performed in step S11. In this way, the effective face changing unit 81 performs the effective face changing process, so that the detection accuracy of the effective face can be increased.

  When the effective face changing process is completed, the CPU 75 moves the process to step S11 as shown in FIG. 4B, and the CPU 75 determines whether or not there is an invalid face on the main image (step S11). If it is determined that there is no invalid face (step S11; NO), the main image that has been subjected to image processing by the image processing unit 68 is displayed on the monitor 18 via the display control unit 71, and the media Recording is performed on the external recording medium 70 via the control unit 69 (step S13). At this time, the compression / decompression processing unit 67 may further perform compression processing on the main image data subjected to image processing to generate an image file.

  Then, the CPU 75 determines whether or not an off operation has been performed by the power switch 22 (step S14). If the off operation has been performed (step S14; YES), the power of the digital camera 1 is turned off, and the process ends. If an off operation has not been performed (step S14; NO), as shown in FIG. 4A, the process proceeds to step S1, and the processes after step S1 are repeated.

  On the other hand, when it is determined in step S11 that there is an invalid face (step S11; YES), the portrait right protection unit 82 performs portrait right protection processing (step S12).

  As the portrait right protection process, as described above, the mosaic process or the blur process can be performed on the invalid face.

The mosaic processing is performed by averaging RGB values in a set of cells, so-called cells, according to the following equation (11). In the following formula (11), n represents the number of pixels in the cells to be averaged.

  Here, FIG. 11A to FIG. 11F show an example of an image of a face area on which mosaic processing has been performed. For example, when the number of pixels in the image of the face area shown in FIG. 11 is 200 × 200 pixels, for example, when averaging is performed with n = 1 × 1, mosaic processing is performed, as shown in FIG. The image is not mosaic-processed, and as the value of n increases, the face details become difficult to confirm as shown in FIGS. 11B, 11C, 11D, and 11E in this order. . When the averaging is performed with n = 200 × 200 and the mosaic processing is performed, the inside of the cell becomes a uniform color.

For the blurring process, a known process performed by a blurring filter using a Gaussian function of the following formula (12) can be used.

  Next, FIG. 12 shows a flowchart of the mosaic or blurring process by the portrait right protection unit 82. As shown in FIG. 11, the portrait right protection unit 82 first determines whether or not the portrait right protection mode is set (step S51). If the portrait right protection mode is set (step S51; YES), an invalid face is determined. The tag is read (step S52), and the above-described mosaic process or blurring process is performed on the image area inside the face detection frame K including the invalid face, that is, the invalid face (step S53). At this time, the mosaic processing and the blurring processing perform processing that can be performed by the portrait right protection unit 82 selected in advance by the CPU 75 as the selection unit as described above.

  When the invalid face is subjected to the mosaic process or the blur process, the main image subjected to the image process is displayed on the monitor 18 via the display control unit 71 and also via the media control unit 69. Recording is performed on the external recording medium 70 (step S54).

  On the other hand, when the portrait right protection mode is not set in step S51 (step S51; NO), the process proceeds to step S54, and the main image subjected to the image processing by the image processing unit 68 is displayed. Processing for displaying on the monitor 18 is performed via the control unit 71 and recording is performed on the external recording medium 70 via the media control unit 69 (step S54). In this way, the mosaic process or the blur process as the portrait right protection process by the portrait right protection unit 82 is performed.

  As a result, it is possible to make it difficult to identify an individual for a face determined to be an invalid face, and thus privacy and portrait rights of a person determined to be an invalid face can be protected.

  In the present embodiment, the mosaic process or the blur process is performed on the image area inside the face detection frame K. However, the present invention is not limited to this, and the face detection unit 69 is included in the face, for example. When an eye is detected as a region having a facial feature (singular point), the above processing can be performed only on the peripheral portion of the eye. When the face detection unit 69 detects the face outline, the above processing can be performed only on the inside of the face outline or on all the areas inside the face detection frame K other than the inside of the face outline.

  As the portrait right protection process, as described above, a warning process can be performed when there is an invalid face. FIG. 13 shows a flowchart of warning processing by the portrait right protection unit 82.

  As shown in FIG. 13, the portrait right protection unit 82 first determines whether or not the portrait right protection mode is set (step S61). If the portrait right protection mode is set (step S61; YES), an invalid face is detected. A warning is given that the image has been taken (step S62). As warning means, for example, an error message prompting re-shooting may be displayed on the liquid crystal monitor 18, a warning sound may be generated, or the error message may be transmitted by voice, It can be changed as appropriate.

  In this way, for example, a passing person is photographed at the time of photographing, and if it is determined that the person's face is an invalid face, a warning is given to shoot again, so that the photographed image of this person is displayed. Displaying on the liquid crystal monitor 18 or recording on the recording medium 70 can be prevented.

  On the other hand, when the portrait right protection mode is not set in step S61 (step S61; NO), the process proceeds to step S63, and the main image subjected to the image processing is displayed via the display control unit 71. Then, the display process is performed on the monitor 18 and is recorded on the external recording medium 70 via the media control unit 69 (step S63). In this way, the warning process as the portrait right protection process by the portrait right protection unit 82 is performed.

  Further, as the portrait right protection process, as described above, the trimming process can be performed on the invalid face. Here, FIG. 14 shows a flowchart of the trimming process by the portrait right protection unit 82, FIG. 15A shows an example of a photographed image before the trimming process, and FIG. 15B shows an example of a photographed image after the trimming process.

  The portrait right protection unit 82 first determines whether or not the portrait right protection mode is set (step S71), and if it is set (step S71; YES), the invalid face tag shown in FIG. Is read (step S72), the position of the invalid face is further read (step S73), the trimming area including the invalid face is calculated, and the trimming area is grayed and displayed on the liquid crystal monitor 18 for confirmation, for example. (Step S74).

  As shown in FIG. 15A, the trimming area is calculated when a face in the face detection frames K1, K2, and K3 as an effective face and a face in the face detection frame K4 as an invalid face are photographed. When the aspect ratio is 4: 5, for example, as shown in FIG. 15B, the maximum area that does not include the face in the face detection frame K4 that is an invalid face and maintains the aspect ratio 4: 5 is used. A region including an invalid face other than the image (a black region in FIG. 15B) is calculated as a trimming region so that the image remains.

  Then, trimming is started based on the calculated trimming area (step S75), and the image subjected to the trimming process is displayed on the monitor 18 via the display control unit 71 and also via the media control unit 69. Recording is performed on the external recording medium 70 (step S76).

  On the other hand, when the portrait right protection mode is not set in step S71 (step S71; NO), the process proceeds to step S77, and the main image subjected to the image processing is displayed via the display control unit 71. Then, the display process is performed on the monitor 18 and is recorded on the external recording medium 70 via the media control unit 69 (step S77). In this way, the trimming process as the portrait right protection process by the portrait right protection unit 82 is performed.

  By performing the trimming process as described above, for example, when a passing person is photographed at the time of photographing and it is determined that this person's face is an invalid face, the person's face is automatically trimmed. Thus, it is possible to prevent an image of this person from being displayed on the liquid crystal monitor 18 or recorded on the recording medium 70.

  This trimming process can be executed when the invalid face is positioned on the top, bottom, left, and right edges of the image. For example, when the invalid face is at the center of the image, the trimming area is calculated by the above method. I can't do it because I can't. In this case, for example, the CPU 75 as the selection unit may reselect the portrait right protection unit 82 that performs processing other than trimming processing.

  As the portrait right protection process, as described above, an unprintable process for making it impossible to print image data including an invalid face can be performed. Here, FIG. 16 shows a flowchart of the unprintable process by the portrait right protection unit 82.

  The portrait right protection unit 82 first determines whether or not the portrait right protection mode is set (step S81), and if it is set (step S81; YES), prints on the image data as shown in FIG. An impossible tag is added (step S82). This non-printable tag makes it impossible to print an image when detected by a printing device such as an ink jet printer. Then, the main image subjected to the image processing is displayed on the monitor 18 via the display control unit 71 and is recorded on the external recording medium 70 via the media control unit 69 (step S83).

  On the other hand, when the portrait right protection mode is not set in step S81 (step S81; NO), the process proceeds to step S83, and the main image subjected to the image processing is displayed via the display control unit 71. Then, the display process is performed on the monitor 18 and is recorded on the external recording medium 70 via the media control unit 69 (step S83). In this way, the non-printable process as the portrait right protection process by the portrait right protection unit 82 is performed.

  When the portrait right protection process described above is completed (step S12), as shown in FIG. 4B, the CPU 75 determines whether or not the power switch 22 has been turned off (step S14). (Step S14; YES), the power of the digital camera 1 is turned off, and the process ends. If an off operation has not been performed (step S14; NO), as shown in FIG. 4A, the process proceeds to step S1, and the processes after step S1 are repeated. In this way, the digital camera 1 captures the subject.

  Next, a printing process for printing an image photographed by the digital camera 1 using a printing device such as an inkjet printer will be described. Here, FIG. 17 shows a flowchart of printing processing by a printing device such as an ink jet printer.

  As shown in FIG. 17, when printing a photographic image, the printing device first inputs a print image (step S91), and then determines whether there is an invalid face tag in the input image file (step S91). S92). If there is no invalid face tag (step S92; NO), the printing device prints an image based on the data of the image file (step S93).

  On the other hand, if there is an invalid face tag (step S92; YES), it is determined whether or not there is a portrait right protection mode tag, that is, whether or not the image is shot in the portrait right protection mode (step S92). If it is not the portrait right protection mode (step S94; NO), the process proceeds to step S93, and the printing device prints an image based on the data of the image file (step S93).

  If it is the portrait right protection mode (step S94; YES), it is determined whether or not there is a non-printable tag in the image file (step S95). If there is a non-printable tag (step S95; YES). For example, an error is displayed on the liquid crystal panel of the printing device, for example (step S96). When an error is displayed, no image is printed.

  On the other hand, if there is no non-printable tag (step S95; NO), portrait right protection printing is performed (step S27). Here, portrait right protection printing refers to a face determined to be an invalid face as described above. Printing an image that has undergone portrait right protection processing such as mosaic processing, blurring processing, or trimming processing.

  In this embodiment, even if the printing device has an invalid face tag, it is printed if it is an image that has undergone portrait right protection processing such as mosaic processing, blurring processing, or trimming processing. When an invalid face is subjected to a portrait right protection process such as a mosaic process, a blurring process, or a trimming process on the invalid face by the digital camera 1, the invalid face tag is deleted so that only the invalid face tag is printed. You may do it. In this case, the image to be printed is an image that has been subjected to the portrait right protection process or an image that does not include an invalid face.

  As a result, it is possible to print only an image that has been subjected to the portrait right protection process on an invalid face or an image that contains an invalid face that does not include an invalid face, so an erroneous printing of an image that contains an invalid face Can be prevented.

  As described above, according to the digital camera 1 of the present embodiment, an effective face or an invalid face is determined for each of one or more faces detected from image data acquired by imaging a subject, and an invalid face is determined. The portrait right protection process is applied to the face that is determined to be, so if the photographer does not want the face, that is, the third party is determined to be an invalid face, the portrait right protection process is automatically applied to the third party. Therefore, it is possible to prevent infringement of privacy and portrait rights for a third party when releasing a photographed image.

  In addition, when the number of faces to be the subject is arbitrarily set and the number of faces determined to be valid faces is larger than the set number of faces, various types of face size, position, variation, focus evaluation value, etc. The effective face is decremented from the effective face to the invalid face in the order of the lower priority of the information, and when the number of faces is smaller than the set number, the effective face is Detection accuracy can be increased.

  Although the digital camera 1 of the present embodiment includes the effective face changing unit 81, the imaging apparatus of the present invention may not include the effective face changing unit 81. In this case, the process of step S10 in FIG. 4B is not performed.

  Note that the imaging apparatus of the present invention is not limited to the digital camera 1 of the above-described embodiment, and can be changed as appropriate without departing from the spirit of the present invention.

Rear view of digital camera Front view of digital camera Functional block diagram of digital camera Flow chart of digital camera shooting process (1) Flow chart of digital camera shooting process (2) Flow chart of face determination processing (part 1) Flow chart of face determination process (part 2) The figure which shows an example of a captured image The figure explaining the validity judgment based on the positional information on the face The figure which shows another example of a captured image (A) Distribution of frequency components of captured image (b) Gray scale image of power spectrum of each spatial frequency Effective face change processing flowchart An example of a face area image that has undergone mosaic processing Mosaic processing or blur processing flowchart Warning process flowchart Trimming process flowchart (A) Captured image before trimming process (b) Captured image after trimming process Unprintable process flowchart Flow chart of printing process by printing device

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 100 Imaging part (imaging means)
69 Face detection unit (object detection means)
75 CPU (selection means)
80 Face determination unit (subject determination means)
81 Effective face changing unit (effective subject changing means)
82 Portrait Rights Protection Department (Portrait Rights Protection Means)

Claims (11)

Imaging means for capturing an image of a subject and acquiring image data;
Object detection means for detecting a predetermined object from the image data;
Subject determination means for determining whether each of the one or more objects detected by the object detection means is an effective subject or an invalid subject;
An imaging apparatus comprising: portrait right protection means for protecting a portrait right for the object determined to be an invalid subject by the subject determination means.   The subject determination means makes the determination based on information including one or more of the size, position, change amount, and focus evaluation value of the object detected by the object detection means. The imaging apparatus according to claim 1, wherein the imaging apparatus is characterized. When the number of the objects determined as effective subjects by the subject determination means is larger than the number of arbitrarily set objects, the effective subjects are moved down from the effective subjects to the invalid subjects in the order of the low priority of the various information,
The imaging apparatus according to claim 2, further comprising an effective subject changing unit that advances the invalid subject to the effective subject in descending order of the priority of the various information when the number is smaller than the number.   The imaging apparatus according to any one of claims 1 to 3, wherein the portrait right protection unit performs a mosaic process on the object determined to be the invalid subject.   The imaging apparatus according to claim 1, wherein the portrait right protection unit performs a blurring process on the object determined to be the invalid subject.   The imaging apparatus according to any one of claims 1 to 3, wherein the portrait right protection unit performs a trimming process on the object determined to be the invalid subject.   The imaging apparatus according to any one of claims 1 to 3, wherein the portrait right protecting means warns when there is the object determined to be the invalid subject.   4. The portrait right protection unit is configured to make it impossible to print image data including the object when there is the object determined to be the invalid subject. The imaging apparatus of Claim 1.   9. The imaging apparatus according to claim 1, further comprising a plurality of types of portrait right protection means, and selection means for selecting and using one of them. In an imaging device that captures an image of a subject, acquires image data, and detects a predetermined object from the acquired image data,
For one or more detected objects, it is determined whether the object is an effective subject or an invalid subject based on information including one or more of the size, position, change amount, and focus evaluation value of the object. And
A portrait right protection method comprising performing a portrait right protection process on the object determined to be an invalid subject. Set the number of target objects as desired,
When the number of the objects determined to be the effective subject is larger than the number of the set objects, the effective subject is moved down from the effective subject to the invalid subject in the order of the low priority of the various information,
The portrait right protection method according to claim 10, wherein when the number is smaller than the number, the invalid subject is moved up from the invalid subject to the valid subject in descending order of priority of the various information.