JP6670496B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing device.

  In general, when an object is imaged using an imaging device such as a camera, depending on the orientation of the camera and the relationship with the object, the captured image (hereinafter, simply referred to as “image” or “photograph”) may be generated. Accuracy may be reduced. For example, there are out-of-focus due to out of focus, overexposure where the amount of light taken into the lens is too large, and underexposure where the amount of light taken into the lens is too large.

  There is also a technique for recognizing a face based on a captured image and a person included in the image. In order to perform face recognition, certain accuracy is required for a captured image.

  This is because human faces are rich in variety and various kinds of information are required for face recognition. For example, it is necessary to make a comprehensive judgment using information that differs for each individual such as race, age, and gender, and information such as imaging information such as the direction and inclination of the person who has taken the image.

  For this reason, face recognition may become difficult when the exposure amount of the captured image is not appropriate or when the sensitivity is low.

  Patent Literature 1 describes correcting an image by adjusting an exposure amount and the like.

Japanese Patent Application No. 2014-239396

  However, in the technique disclosed in Patent Literature 1, when an entire image is viewed, an appropriate image is obtained by the correction, but a part of the image is not necessarily an appropriate image. May not be.

  For example, in the case of performing face recognition of a person from an image in which a person is captured together with a background, an appropriate image is generally obtained, but face recognition is difficult in an image of a part of the face. is there.

  Therefore, an object of the present invention is to provide an image processing apparatus that performs image processing that can appropriately perform face recognition.

In order to achieve the above object, the present invention according to claim 1 is an imaging unit that captures a person including a face to be recognized as an imaging target based on imaging setting information , A face recognition unit that performs face recognition; an information acquisition unit that acquires face information related to the face recognized by the face recognition unit; and attribute information of a person can be estimated based on the face information acquired by the information acquisition unit. and propriety judging means for judging whether or not the previous SL comprises an adjustment means for adjusting the imaging setting information used for imaging the imaging object, the permission determination means, the color tone of the face recognized by the face recognition unit It is determined whether or not the data is within an effective range designated as capable of estimating the attribute information of the person. The adjusting means can recognize the face of the person by the face recognizing means. Judgment means Therefore, when the color tone data is not within the effective range, the imaging setting information is adjusted based on the color tone data of the recognized face, and when the face of the person cannot be recognized by the face recognition means, The imaging setting information is adjusted based on the color tone data.

FIG. 1 is a diagram showing a vending machine having an image processing device according to an embodiment of the present invention. FIG. 1 is a block diagram configured by applying an image processing apparatus according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a state of image processing in the image processing apparatus according to the embodiment of the present invention. 5 is a flowchart illustrating a flow of image processing performed in the image processing device according to the embodiment of the present invention. 5 is a flowchart showing a detailed flow of “adjustment confirmation processing 1” shown in FIG. 5 is a flowchart showing a detailed flow of “adjustment confirmation processing 2” shown in FIG. 4. FIG. 4 is a diagram illustrating an imaging setting information change table used in the image processing apparatus according to the embodiment of the present invention.

  Hereinafter, an embodiment of an image processing apparatus according to the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a diagram illustrating a vending machine having an image processing device according to an embodiment of the present invention, and FIG. 2 is a block diagram configured by applying the image processing device according to the embodiment of the present invention. . Note that this image processing device can perform image processing on an image captured by an imaging device provided for crime prevention, and can also perform a predetermined image.

  In FIG. 1, a vending machine can pay out a beverage (a canned beverage, a PET bottled beverage, and the like) contained in a container and provide a predetermined service by paying a price. For example, when payment is made by a currency such as coins or bills or electronic money, the selected kind of beverage is discharged from a discharge port provided at a lower portion, and predetermined information ( Information is provided by displaying map information, restaurant information, event information, etc.).

  This vending machine is equipped with an image pickup device (image pickup device, photographing device) such as a CCD camera or a CMOS sensor (hereinafter, referred to as an “image pickup unit 101”), and has a predetermined image pickup condition (image pickup start condition). Upon establishment, the imaging unit 101 images an imaging target (imaged object) including a person who is provided with a service at the vending machine based on the imaging setting information. The imaging target may be a person alone, but includes the background in addition to the person.

  Further, the predetermined imaging condition may be that a person has instructed to enjoy the service, or that the person has been continuously located in an imaging range in which an arbitrary person can capture an image for a certain period of time.

  Further, the imaging setting information is setting information used when the imaging unit 101 captures an image of an imaging target, and includes sensitivity (gain) of a CCD camera or a CMOS sensor or the amount of light incident on the CCD camera or the CMOS sensor. It is configured by the exposure time to be adjusted and the like.

  When the imaging conditions are satisfied and the imaging unit 101 images an imaging target based on imaging setting information (initial imaging setting information) specified in advance, an imaging image of the imaging target can be displayed on the display unit 110. is there. Note that the captured image does not necessarily need to be displayed on the display unit 110.

  FIG. 2 shows the imaging unit 100 and the display unit 110. The imaging unit 100 obtains imaging information (corresponding to “color tone data” described later) of the imaging target by imaging the imaging target, and identifies the imaging target using optimal imaging setting information based on the imaging information. This is a unit for imaging. When a person is included in the imaging target, face information including imaging information of a face portion of the person (hereinafter, simply referred to as a “face”) is acquired, and in order to improve recognition accuracy of the face of the person. The imaging target is imaged using the imaging setting information newly set based on the face information. The face at this time is not only a recognition target recognized by the person, but also an attribute estimation target (attribute identification target) for estimating (specifying) what kind of attribute information the person has. At this time, the attribute information (also simply referred to as “attribute”) is information on a person, and is represented by race, age, gender, and the like.

  The imaging unit 100 includes an imaging unit 101 (corresponding to the above “imaging device”) including a lens and an imaging element for imaging an imaging target including a person or the like, and a predetermined image captured by the imaging unit 101. The image processing apparatus includes a signal processing unit 102 that performs predetermined signal processing on an image signal sampled at a timing, and an image processing unit 103 that performs image processing on an image signal.

  In the imaging unit 101, input signals to be imaged are read by each of the imaging elements arranged in an array via various filters, and the read input signals are transmitted to the signal processing unit 102.

  The signal processing unit 102 includes a sampling processing unit 102a, a generator 102b, a register 102c, and an adjustment unit 102d.

  The signal read by the imaging unit 101 is input to the sampling processing unit 102a. The sampling processing unit 102a changes the content of the input signal (the state of the input signal from the state where the signal is input from the imaging unit 101 and the state where the signal is not input). A signal is transmitted to the adjustment unit 102d after performing a sampling process for clarifying the image. The sampling processing unit 102a can remove noise due to an input signal having a low signal level to the maximum. The sampling processing unit 102a receives a clock signal (pulse) from the generator 102b and performs a sampling process at a processing timing based on the clock signal.

  The generator 102b can generate a clock signal for predetermined signal processing, and may transmit the clock signal to the imaging unit 101 in addition to transmitting the clock signal to the sampling processing unit 102a. In this case, the sampling process performed by the sampling processing unit 102a and the imaging process performed by the imaging unit 101 are processed at a timing based on the clock signal. Assuming that the clock signal at this time is “33 ms (millisecond)”, an image of “30 frames” per “one second (1000 ms)” is captured by the imaging unit 101.

  The signal (signal for each frame) that has been subjected to the sampling processing in the sampling processing unit 102a is input to the adjustment unit 102d, and the adjustment unit 102d receives the signal from the sampling processing unit 102a, and based on the signal, When it is determined that the captured image is not appropriate because the imaging setting information is inappropriate, a process of adjusting at least a part of the imaging setting information is performed to maintain a signal level at a constant level. When the adjustment processing is performed, the adjustment unit 102d stores the adjusted imaging setting information in the register 102c. When it is not determined that the captured image is not appropriate (when the captured image is appropriate), the process of adjusting at least a part of the imaging setting information may not be performed.

In addition, the adjusting unit 102d uses the face information (recognized face position coordinates, face position coordinate brightness information) stored in the register 102c by image processing performed in the image processing unit 103 as described below. It is also possible to perform adjustment confirmation processing of imaging setting information. That is, the imaging setting information is information used by the imaging unit 101 when estimating attribute information of a person from a recognized face. If various information is not stored in the register 102c, the adjustment confirmation processing of the imaging setting information is not performed.

  The imaging unit 101 performs imaging of an imaging target based on the imaging setting information stored in the register 102c.

  When such signal processing is performed, the signal processing unit 102 transmits the captured image to the image processing unit 103.

  The image processing unit 103 performs a “face recognition process” on the basis of the received captured image to determine whether a person's face is included in the imaging target from the captured image. The face recognition process is a process of recognizing a face, and is a process of identifying that an imaging target includes a person (a person having a recognized face). The face recognition processing is processing for recognizing that a person is included in the captured image. In the face recognition process, for example, it is possible to divide a captured image into a plurality of rectangular areas and to grasp the features of the face based on the brightness difference between the rectangular areas. In addition, the face recognition processing is performed by forming a three-dimensional model. There is a method of recognizing a face by detecting an organ (eye, nose, mouth, etc.).

  When the face recognition process is performed and the face is recognized, the image processing unit 103 performs a process of specifying the position coordinates of the recognized face. At this time, if a plurality of faces can be recognized, the position coordinates of each face are specified. When face position coordinates are specified by performing face recognition, the image processing unit 103 acquires color tone data of a pixel corresponding to the position coordinates of the face. The color tone data is imaging information used to determine imaging setting information, and is represented by, for example, lightness (brightness information) or saturation. Hereinafter, description will be made based on “brightness information”.

  This brightness information (an example of color tone data, the same applies hereinafter) is represented by 1 byte (8 bits) per pixel when the captured image is a grayscale image, and the gradation of the image is represented by 256 gradations. It is. At this time, the darkest state is “0 gradation”, and the lightest state is “255 gradation”.

  Subsequently, the image processing unit 103 calculates the shading of each pixel, and calculates the average value of the entire image from the calculated shading of each pixel. This average value is used as the brightness information of the captured image. That is, when face recognition is performed by the image processing unit 103 and face position coordinates are specified, brightness information of the face position coordinates is calculated. If the recognition is not possible, the brightness information of the entire captured image is calculated.

As described above, when the brightness information is calculated, the image processing unit 103 stores the brightness information in the register 102c. The position coordinates (face position information) are stored in the register 102c. That is, when the face in the face recognition processing in the image processing unit 103 can be recognized, the register 102c, it could face recognition, the position coordinates of the recognized face, the brightness information of the face position coordinates as face information The state is stored. On the other hand, if the face cannot be recognized, the face cannot be recognized, and the brightness information of the entire captured image is stored. Note that the register 102c stores, as an initial value, “a face cannot be recognized (face recognition flag = OFF)”. The face position information at this time includes not only the position information of the entire face, the position information and shape information of each of the organs (eyes, nose, mouth, etc.) constituting the face, but also information on the positional relationship between the organs. It may be.

  When the brightness information for a predetermined number of frames is stored in the register 102c in this way, the adjustment unit 102d uses the brightness information (in a broad concept, based on the face information) to store the imaging setting information. Perform adjustment confirmation processing.

This adjustment confirmation processing is performed by the imaging setting information used by the imaging apparatus so that the brightness information, which is the color tone data, is within the effective range specified to be able to estimate the attribute information of the person from the recognized face. Is the process of changing More specifically, the adjustment confirmation process is a process of changing imaging setting information such as a gain and an exposure time with a predetermined change width using an imaging setting information change table as shown in FIG.

Therefore, the imaging setting information such as the gain and the exposure time is changed in a predetermined change width , and when the brightness information is within the specified effective range, the face information of the recognized face is also changed. Then, it is determined that the imaging end condition for ending the imaging process has been satisfied because the attribute information of the person can be estimated. Details of the adjustment confirmation processing are shown in FIGS. 5 and 6 and will be described later.

  Further, the predetermined number of frames may be, for example, an arbitrary number of frames or a plurality of frames captured at a time based on a clock signal. The adjustment unit 102d performs the adjustment confirmation process based on the brightness information for a plurality of frames stored in the register 102c. However, the present invention is not limited to this. The adjustment unit 102d may perform the adjustment confirmation process when is stored. That is, the adjustment confirmation process is performed for each single frame that has been imaged.

  FIG. 3 is a diagram illustrating a state of image processing in the image processing apparatus according to the embodiment of the present invention.

  In FIG. 3, FIG. 3A is a diagram showing a captured image showing one frame among the captured images of a predetermined number of frames. In this captured image, a face is appropriately recognized by face recognition processing. 3 shows a captured image that cannot be obtained. FIG. 3B is a diagram illustrating a state in which face recognition processing is performed using the captured image illustrated in FIG. 3A and face position coordinates are extracted.

  FIG. 3C shows the calculation of face setting data (gain, exposure time) after calculating the face tone data (brightness information) as shown in FIG. 3B and using the calculated tone data. FIG. 4 is a diagram illustrating a captured image of an imaging target captured by an imaging unit using the imaging setting information. In the captured image illustrated in FIG. 3C, the entire captured image increases the possibility of face recognition by the imaging setting information set using the color tone data of the face portion, compared to the captured image illustrated in FIG. 3A. It shows the state where it was turned on.

  FIG. 4 is a flowchart illustrating a flow of image processing performed in the image processing apparatus according to the embodiment of the present invention.

  In FIG. 4, when the image processing unit, which is an example of the image processing apparatus, starts processing when, for example, the imaging start condition is satisfied, it performs initialization processing (S401). This initialization process is a process of setting information stored in a register to an initial state.

  After the initialization process (S401), the image processing unit performs a process of imaging an imaging target including a person using the imaging device (S402). When the imaging target is imaged, the image processing unit performs a process of recognizing a face portion from a person included in the captured image (S403).

  Next, the image processing unit determines whether or not the face has been recognized in the face recognition processing (S404). When it is determined in the face recognition process that the face recognition has been performed (YES in S404), the image processing unit performs a process of acquiring information (face information) on the recognized face (S405). The face information at this time is information necessary for estimating attribute information of a person from the recognized face, and corresponds to color tone data of a face portion, position coordinates of the face, and the like.

  When the face information is obtained in this way, the image processing unit performs a process of storing the face information (S406). As a result, the face information for the captured image is stored. When the imaging target is imaged in frame units, face information is stored in frame units.

Subsequently, it is determined whether or not to save the face information of the predetermined number of frames (S407), if it is not determined that the stored face information corresponding to a predetermined number frames (NO in S407), the image processing unit Subsequently, it is determined whether or not the imaging end condition associated with the end of the estimation of the attribute information of the person is satisfied (S411). In contrast, if it is determined that the stored face information corresponding to a predetermined number frames (YES in S407), the image processing unit performs the "adjustment confirmation process 1" (S408). If the predetermined number of frames is a single frame, these processes are performed for each frame.

  The "adjustment confirmation process 1" is a process as shown in FIG. 5, in which new imaging setting information is calculated based on color tone data included in face information.

  Next, in the process of determining whether or not the face has been recognized in the face recognition process (S404), if it is not determined that the face has been recognized in the face recognition process (NO in S404), the image processing unit determines whether or not the captured image has been captured. A process for acquiring color tone data from the whole is performed (S409), and an "adjustment confirmation process 2" is performed (S411).

  The “adjustment confirmation process 2” is a process as shown in FIG. 6, and is a process of calculating new imaging setting information based on color tone data of the entire captured image.

In this way, when the adjustment confirmation process 1 is performed, if the adjustment confirmation process 2 was performed, if you store the number of face images predetermined frame, the image processing unit in either of these cases, the person It is determined whether or not the imaging end condition accompanying the end of the estimation of the attribute information is satisfied (S411). The imaging end condition is that processing necessary for estimating the attribute information of the person has been completed. That is, it indicates that appropriate imaging setting information for estimating the attribute information of the person has been set.

  FIG. 5 is a flowchart showing a detailed flow of the “adjustment confirmation process 1” shown in FIG. This “adjustment confirmation processing 1” is processing for calculating imaging setting information for imaging the entire captured image from color tone data that is a part of face information.

  In FIG. 5, the image processing unit acquires color tone data, which is a part of face information, from each pixel of the face part recognized by the face recognition processing (S501). This color tone data is specified by the gradation value as described above.

  Upon acquiring the color tone data, the image processing unit calculates “average color tone data” of the entire captured image from the color tone data of each pixel (S502). This “average tone data” is indicated as brightness information of the face portion.

  Subsequently, the image processing unit determines whether or not the "average tone data", that is, the brightness information of the face portion is within a predetermined effective range and within an effective range necessary for estimating attribute information of a person. Is determined (S503). When determining that the “average tone data” is within the predetermined range (YES in S503), the image processing unit ends the processing. On the other hand, when it is not determined that the “average tone data” is within the predetermined range (NO in S503), the image processing unit subsequently changes the imaging setting information (gain, exposure time) by a predetermined change width. The changed imaging setting information is calculated (S504). Then, the image processing unit stores (registers) the imaging setting information in the register (S505).

  FIG. 6 is a flowchart showing a detailed flow of the “adjustment confirmation process 2” shown in FIG. In the “adjustment confirmation processing 2”, imaging setting information for imaging the entire captured image is calculated from the color tone data of the entire captured image.

  In FIG. 6, the image processing unit acquires color tone data from each pixel of the entire captured image (S601).

  Upon acquiring the color tone data, the image processing unit calculates “average color tone data” of the entire captured image from the color tone data of each pixel (S602). This “average tone data” is indicated as brightness information of the face portion.

  Subsequently, the image processing unit determines whether or not the "average tone data", that is, the brightness information of the face portion is within a predetermined effective range and within an effective range necessary for estimating attribute information of a person. Is determined (S603). When determining that the “average tone data” is within the predetermined range (YES in S603), the image processing unit ends the process. On the other hand, when it is not determined that the “average tone data” is within the predetermined range (NO in S603), the image processing unit subsequently changes the imaging setting information (gain, exposure time) by a predetermined change width. The changed imaging setting information is calculated (S604). Then, the image processing unit stores (registers) the imaging setting information in the register (S605).

  As described above, the optimum imaging setting information is stored in the register by the respective adjustment confirmation processes as shown in FIGS. 5 and 6, and it is possible to estimate the attribute information of the person using the imaging setting information. Become.

  FIG. 7 is a diagram illustrating an imaging setting information change table used in the image processing apparatus according to the embodiment of the present invention.

  7, the imaging setting information change table includes an imaging setting information item name 70, a priority 71, a setting range 72, and a change width 73.

  The imaging setting information item name 70 is an item constituting imaging setting information. The priority 71 is information for designating an item to be prioritized when any one of the imaging setting information item names 70 is changed, and the setting range 72 is a range that can be taken by each item constituting the imaging setting information. The change width 73 is a change width for changing each item constituting the imaging setting information.

  In FIG. 7, “gain (sensitivity)” and “exposure time (shutter speed)” are shown as the imaging setting information item name 70, and “high (high)” “1)” and “exposure time (shutter speed)” indicate “low (0)”. This indicates that “gain (sensitivity)” is changed prior to “exposure time (shutter speed)”.

  As shown in the change width 73 at this time, the “gain (sensitivity)” is changed at a change width of “5”, and the “exposure time (shutter speed)” is changed at a change width of “500”. Indicates that it is being changed. In the setting range 72, setting ranges in which “gain (sensitivity)” and “exposure time (shutter speed)” can be taken are shown, and “gain (sensitivity)” is a range of “0 to 255”. The "exposure time (shutter speed)" is in the range of "0-65535".

  From the above, when face information (color tone data (particularly, brightness information)) of a recognized face is not within a predetermined range when face recognition is performed in a captured image obtained by capturing an imaging target, “gain (sensitivity)” Is changed with a change width of “5” in priority, and then “exposure time (shutter speed)” is changed with a change width of “500”. The “gain (sensitivity)” and the “exposure time (shutter speed)” may be changed alternately, or by changing only one of them, the face information of the face (color tone data (particularly, , Brightness information)) may be within a predetermined range.

  As described above, by changing the “gain (sensitivity)” and the “exposure time (shutter speed)” that are the imaging setting information, it is possible to obtain appropriate brightness information for estimating attribute information of a person. .

  The embodiment described above is an embodiment of the present invention, and the present invention is not limited to the embodiment, and can be appropriately modified and implemented without changing the gist thereof.

Reference Signs List 10 vending machine 100 imaging unit 101 imaging unit 102 signal processing unit 103 image processing unit 110 display unit

Claims (3)

Imaging means for imaging a person including a face to be recognized as an imaging target based on imaging setting information ,
Face recognition means for performing face recognition of a person imaged by the imaging means,
Information acquisition means for acquiring face information about the face recognized by the face recognition means,
Availability determination means for determining whether or not estimation of attribute information of a person is possible based on the face information acquired by the information acquisition means,
Comprising an adjustment means for adjusting the imaging setting information used for imaging of the prior SL imaging target,
The availability determination means,
Determine whether the color data of the face recognized by the face recognition means is within an effective range designated as capable of estimating attribute information of a person,
The adjusting means,
When the face of the person can be recognized by the face recognition unit, and the color tone data is not within the effective range by the availability determination unit, the imaging setting information is adjusted based on the recognized color tone data of the face,
When the face of the person cannot be recognized by the face recognition unit, the imaging setting information is adjusted based on color tone data of the entire imaging target.
Image processing device. The imaging setting information includes:
Including the sensitivity of the imaging means, exposure time,
The adjusting means,
The image processing apparatus according to claim 1, wherein at least one of the sensitivity and the exposure time for the face acquired by the information acquisition unit is adjusted. The image processing apparatus according to claim 1 or 2 further comprising a display means for displaying the imaging target based on the image pickup setting information.