JP2009251679A - Imaging apparatus and image recognition method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To efficiently and quickly recognize an object image which has newly appeared in an image while following up a detected object image in an image in continuous images. <P>SOLUTION: This imaging apparatus includes a function for retrieving the object image in one portion of the region of an image when retrieving any object image other than the object image detected in continuous images in the image. Thus, it is possible to reduce the retrieval processing quantity of each image when retrieving any object image other than the detected object image, and to efficiently and quickly recognize an object which has newly appeared in the image. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging apparatus and an image recognition method, and more particularly to an imaging apparatus and an image recognition method that can recognize a predetermined object image from continuous images.

  2. Description of the Related Art Conventionally, many imaging devices such as digital cameras have an autofocus function for focusing on a subject. Recently, a facial feature (facial shape, facial color, eye, nose, etc.) is extracted from the screen and the focus is automatically adjusted to that face (face detection autofocus). Cameras are also being developed. With such a camera, if the face detection autofocus function is turned on, the camera automatically focuses on the face just by pointing the camera at the person. At this time, for example, a square frame is displayed around the face displayed on the liquid crystal screen of the camera, and it can be confirmed that the face is detected.

  In such an apparatus, one problem is how to speed up recognition and tracking processing of a face image (hereinafter also simply referred to as a face) in a continuous image with a small amount of calculation processing. Therefore, various techniques for speeding up face recognition and tracking processing in continuous images have been proposed (see, for example, Patent Document 1).

  In Patent Document 1, when a recognized face is tracked, a face detection result (specifically, position information of a face) in a past image is referred to. A technique for limiting the above has been proposed. In the technique of Patent Document 1, the recognition processing speed is increased by reducing the amount of recognition processing during face tracking.

JP 2004-318331 A

  In the technique described in Patent Document 1 described above, tracking of a face that has already been detected in an image can be processed at high speed. However, when another face appears in the image, the face appears newly. The process for recognizing the face is not sufficiently considered. When such a situation occurs, in order to detect a newly appearing face in the image, it is usually necessary to search the entire image, and there is a problem that processing efficiency is poor.

  The present invention has been made to solve the above problems. An object of the present invention is to provide an imaging apparatus and an image recognition method capable of efficiently recognizing a target image newly appearing in an image while tracking a target image such as a face image that has already been detected. Is to provide.

  In order to solve the above problems, an imaging apparatus of the present invention searches an object image that has an imaging element and outputs an image signal, and an object image that meets a specific condition from the image signal output by the imaging unit. And an image processing unit that performs predetermined image processing based on the detection result. The imaging device according to the present invention obtains an image signal when the image processing unit searches for the second object image in the image while searching for and tracking the first object image as the object image. It is set as the structure provided with the control part which controls to search for a 2nd target object image, restrict | limiting to the one part area | region of the image to be performed.

  In order to solve the above problems, the image recognition method of the present invention is a method including the following procedure. First, a first object image is searched from an image signal obtained by imaging (first step). Next, the first object image searched in the first step is tracked (second step). Next, while tracking the first object image in the second step, the second object image is limited to a partial area of the image and searched.

  In the present invention, when searching for an object image (second object image) other than the detected object image (first object image) in the image, the object is detected in a partial area of the image. Search for an image. Therefore, it is possible to reduce the processing amount for each image when detecting an object image different from the detected object image, and to detect the other object image more quickly.

  As described above, in the imaging apparatus and the image recognition method of the present invention, when searching for an object image different from the detected object image in the image, a partial area of the image is searched. The processing amount for each image when detecting another object image can be reduced. Therefore, according to the present invention, an object image newly appearing in an image can be efficiently recognized in a short time while tracking a detected object image.

  Hereinafter, an example of an embodiment of an imaging apparatus of the present invention will be specifically described with reference to FIGS. In this embodiment, a mobile communication terminal having a function of recognizing a face shown in a continuous image will be described as an example of the imaging device. The mobile communication terminal here is a terminal device that performs wireless communication with a radiotelephone base station called a so-called mobile phone terminal. However, the present invention is not limited to this embodiment.

[Device configuration]
First, the configuration of the mobile communication terminal according to the present embodiment will be described. A block diagram of the mobile communication terminal according to the present embodiment is shown in FIG. As shown in FIG. 1, the mobile communication terminal 1 includes an antenna 11, a control unit 12, a communication circuit 13 connected to the antenna 11, a display unit 14, an operation unit 15, a storage unit 16, and a clock unit. 17. The mobile communication terminal 1 also includes a speaker 18, a microphone 19, an audio processing unit 20, a wireless LAN (Local Area Network) communication circuit 21, and a wireless LAN antenna 22.

  Furthermore, the mobile communication terminal 1 includes a camera 24. The camera 24 includes an imaging device, a lens mechanism that causes image light to enter the imaging device, and an imaging signal processing unit that processes and outputs an image signal obtained by imaging with the imaging device.

  Imaging with the camera 24 is performed continuously, for example, every fixed frame period. By recording the image signal continuously obtained in this frame period as it is, a so-called video camera in which a moving image signal is recorded is obtained. Also, by recording only one frame of image signal, a so-called still camera can be obtained in which a still image signal is recorded. In addition, as a lens mechanism with which the camera 24 is provided, for example, an autofocus mechanism is provided, and it is possible to automatically focus on a target subject. As the target subject, for example, as described later, a face recognized from an image based on an image signal can be applied.

  Further, the mobile communication terminal 1 includes a control line 25 and a data line 26 as shown in FIG. The control line 25 is a signal line through which a signal for controlling each device connected to the control line 25 flows. As shown in FIG. 1, some device units in the mobile communication terminal 1 are connected to the control unit 12 through a control line 25, and processing in each device unit is performed under the control of the control unit 12. The data line 26 is a signal line for transferring data between device units connected thereto. Although not shown in FIG. 1, the mobile communication terminal 1 includes a power supply unit, and power is supplied from the power supply unit to each device unit.

  The control unit 12 includes an arithmetic control device such as a CPU (Central Processing Unit), for example, and controls each unit constituting the mobile communication terminal 1. The control unit 12 also functions as an image processing unit that searches and recognizes a face in the image and a control unit that controls the image search and recognition processing.

  The communication circuit 13 transmits and receives a transmission signal to and from a mobile phone base station (not shown) via the antenna 11 under the control of the control unit 12. The communication circuit 13 also modulates and demodulates radio waves exchanged with the mobile phone base station.

  The display unit 14 includes a liquid crystal display (LCD). The operation unit 15 includes a jog dial, a keypad, and the like. The operation unit 15 can input an input operation signal such as an input operation such as a telephone number or an e-mail text, or a setting operation of various modes.

  The storage unit 16 includes a nonvolatile memory such as a flash memory (semiconductor memory). The storage unit 16 stores various data and computer programs such as a phone book, a schedule, a mail message, a moving image, a still image, music, application software, a bookmark, and a web page.

  The clock unit 17 measures time and the occurrence time of a specified event, and an OS (Operating System) that manages the computer acquires date and time information from the clock unit 17. The voice processing unit 20 is a device unit that mainly converts voice data from analog to digital during a voice call.

  The wireless LAN communication circuit 21 performs predetermined modulation and demodulation based on the control of the control unit 12, and transmits wireless signals to and from an external access point device (not shown) via the wireless LAN antenna 22. Receive.

[Image recognition processing]
Next, processing for recognizing a face from a continuous image in the mobile communication terminal 1 according to the present embodiment will be described with reference to FIGS. FIG. 2 is a flowchart showing a procedure of image recognition processing according to the present embodiment. FIG. 3 is a flowchart showing in more detail the processing procedure in step S7 in FIG. FIGS. 4A to 4D are views showing the face search and tracking in steps S3, S5 and S7 in FIG. In the face recognition processing procedure described below, the process of starting the camera will be described. That is, a description will be given from the state where there is no image taken immediately before.

  First, the camera 24 is activated (step S1 in FIG. 2). Next, it is determined whether or not one or more faces have been detected in the immediately preceding image (hereinafter also referred to as a frame) (step S2 in FIG. 2). Note that, as described above, there is no previous image immediately after the camera 24 is activated, so the determination in step S2 is “No” at this stage.

  Next, when an image is taken, a face search is performed over the entire area of the frame (frame) (step S3 in FIG. 2). FIG. 4A shows the face search in the frame in step S3. The hatched portion in FIG. 4A is a face search area. At this time, for example, as shown in FIG. 4A, if the face of the person 51 (hereinafter also referred to as the first face) exists in the frame 30, the first face is detected in this step S3. .

  Next, if the first face is detected in step S3, the position of the first face in the frame 30 is detected, and the position information of the first face is output (step S4 in FIG. 2). In this step S4, for example, the position information of the first face is output by processing such as displaying a square frame around the detected face on the screen of the display unit 14. Next, the process returns to step S2. Even if the face is not recognized in step S3, the process returns to step S2 via step S4 after step S3, but in this case, the face position information is not output in step S4.

  Next, it is determined again in step S2 whether one or more faces have been detected in the immediately preceding frame. Here, when it becomes No determination in step S2 (when a face is not detected in the immediately preceding image), the above-described steps S3, S4, and S2 are repeated in this order.

  On the other hand, as shown in FIG. 4A, when a face is detected in the immediately preceding frame, a Yes determination is made in step S2. In this case, based on the face position information output in step S4, a face is searched in a predetermined area including the face position detected in the immediately preceding frame in the next frame (step S5 in FIG. 2).

  In step S5, first, the first face is searched at the position of the first face detected in the immediately preceding frame. If not detected at this position, the search range is expanded to search for the first face. If the first face is not detected in the expanded search range, the search range is further expanded (for example, the range is doubled) and the first face is searched. In step S5, this process is repeated until the first face is detected. Eventually, the entire region of the frame 30 may be searched in step S5.

  In step S5, the time required for the first face search process (hereinafter also referred to as search time) is also measured. The search time may be measured using a timer (included in the clock unit 17) built in the mobile communication terminal 1, or estimated from the number of repetitions of the search process in which the search range is expanded in step S5. May be.

  As described above, in step S5, the first face detected in the immediately preceding frame is substantially tracked.

  Next, the search time measured in step S5 is compared with a predetermined set time (for example, 100 msec) (step S6 in FIG. 2). The set time can be changed as appropriate according to the number of pixels per image (one frame), the frame rate of continuous images (number of frames per second), the processing capability of the apparatus, and the like.

  If the search time in step S5 is equal to or longer than the predetermined set time, a Yes determination is made in step S6. If the determination is Yes in step S6, the process of step S2 is performed through step S4 as shown in FIG. If the first face is not detected even after searching the entire area of the frame in step S5, a Yes determination is made in step S6, and face position information is not output in the next step S4.

  On the other hand, if the determination is No in step S6 (if the first face can be detected in a relatively short time in step S5), the process moves to step S7, where the frame is divided into a plurality of regions. The face is searched for in the one area (partial area of the image).

  Here, the processing operation of step S7 will be described in more detail with reference to FIG. 3 and FIGS. 4 (b) to 4 (d). In the present embodiment, as shown in FIGS. 4B to 4D, a form in which the frame 30 is equally divided into left and right is used as the form of dividing the frame 30. In the present embodiment, the two left search areas 32 and the right search area 33 are switched alternately for each frame, and a face other than the first face that appears in the frame is searched.

  In the example shown in FIGS. 4B to 4D, a case where the left search area 32 is searched first will be described. In the example shown in FIGS. 4B to 4D, a face of the person 52 newly appears in the left search area 32 at the timing shown in FIG. 4C (the frame shown in FIG. 4C). Will be explained.

  First, when the first face is detected in a relatively short time in Step S5 (when No determination is made in Step S6), the frame 30 is divided into left and right parts (Step S7A in FIG. 3). Next, the left search area 32 is searched (step S7B in FIG. 3: state shown in FIG. 4B). In the frame 30 of FIG. 4B, no new face image exists in the left search area 32, and therefore no new face is detected in this frame 30. An area 31 in FIG. 4B is an area searched when the first face is detected in step S 5, and the first face tracking process is also performed in this area 31.

  In this example, it is assumed that step S7 is started in the same frame as the frame in which the first face is detected in step S5. Depending on the set time set in step S6, after detecting the first face in step S5, a search time sufficient to search for another face in the same frame may not be obtained. In that case, step S7 is started from the next frame.

  Next, it is determined whether or not a new face has been detected in step S7B (step S7C in FIG. 3). Since no new face is detected in the frame 30 of FIG. 4B, the determination in step S7C is No, and the process proceeds to step S7D. If a new face is detected in step S7B, a Yes determination is made in step S7C, and the process proceeds to step S4.

  Next, in step S7D, the frame is switched and the search area is also switched. In the frame 30 of FIG. 4C, the search area is switched to the right search area 33, and the face is searched for in the right search area 33 (state of FIG. 4C).

  In the frame 30 of FIG. 4C, a new face of the person 52 appears in the left search area 32, but no new face exists in the right search area 33 under search. Therefore, a new face is not detected even at the timing of FIG. 4C, and the determination at step S7C is No again.

  Next, the process proceeds to step S7D again, the frame is switched, and the search area is also switched to the left search area 32 (state shown in FIG. 4D). In the frame 30 of FIG. 4D, since the face of the person 52 (hereinafter also referred to as a second face) exists in the left search area 32 being searched, the second face is newly detected at this timing. The In the face recognition process of the present embodiment, two faces can be detected at different frames (timing) as described above. Note that the search time in step S7 is set to a predetermined time in advance, and steps S7A to S7D described above are performed within the set predetermined search time. The search time can be changed as appropriate according to the number of pixels per image (one frame), the frame rate of continuous images (the number of frames per second), the processing capability of the apparatus, and the like.

  In the example of FIGS. 4B to 4D, the example in which the search is first performed from the left search area 32 has been described. However, the search in step S <b> 7 may be started from the right search area 33. In that case, the second face is detected at the timing shown in FIG.

  Next, returning to FIG. 2 again, the steps after step S7 will be described. If a new second face is detected in step S7 as described above, the position information of the second face is output in step S4, and the process returns to step S2. Note that a series of processing from step S2 to step S7 is performed by the control unit 12 of the mobile communication terminal 1 shown in FIG.

  As described above, in the present embodiment, a part of the image is searched when searching for a face other than the detected face while tracking a face that has already been detected in the frame. Therefore, by reducing the processing time for each frame, it is possible to reduce adverse effects on the user's feeling of use. In this embodiment, since the processing time for each frame can be reduced, an apparatus with a large number of captured pixels that increases the processing amount for each image, and a high frame rate that requires a shorter search time for each image. It is suitable for the image recognition processing of this device. Furthermore, the present embodiment is also suitable for an apparatus with a relatively low processing capability that requires more efficient recognition processing.

[Modification]
In the above-described embodiment, an example in which two faces appearing in images at different frames (different timings) in the continuous image has been described. However, similar recognition is also performed when three faces appear in the continuous images at different timings. It can be detected by processing. One example will be described with reference to FIGS. FIGS. 5A to 5D are diagrams showing face search and tracking in steps S3, S5 and S7 in FIG. 2 when detecting three faces.

  In this example, as an image division mode for searching for a newly appearing face in the frame, an example in which the frame 40 is equally divided into left and right as in FIGS. 4 (a) to 4 (d) will be described. To do. 5A to 5D, the new face of the person 53 appears in the left search area 43 when searching the timing shown in FIG. 5C, that is, the right search area 44. The case will be described.

  FIG. 5A shows a state in which two faces (faces of the persons 51 and 52) have already been detected in the frame 40. FIG. In this state, two faces are simultaneously detected in step S3 in FIG. 2, or the face of the person 51 (first face) and the face of the person 52 are obtained through the operations in steps S1 to S7 in FIG. It is obtained by detecting (second face) at another timing.

  Next, the first face and the second face are searched for in the predetermined area 41 including the position of the first face and the predetermined area 42 including the position of the second face, respectively (step S5 in FIG. 2). . At this time, if the first face and the second face are detected within a search time shorter than a predetermined set time, the search area in the frame 40 is divided into two equal parts (step S7A in FIG. 3). Then, the face search is started from the left search area 43 (step S7B in FIG. 3: state shown in FIG. 5B). In the frame 40 of FIG. 5B, no new face is detected in the left search area 43 being searched, and therefore no new face is detected. Therefore, the next step S7C is No.

  Next, the process proceeds to step S7D, the frame is switched, and the search area is also switched to the right search area 44 (state shown in FIG. 5C). In the frame 40 of FIG. 5C, a new face of the person 53 appears in the left search area 43, but no new face exists in the right search area 44 under search. Therefore, a new face is not detected even in the frame of FIG. 5C, and the determination in step S7C is No again.

  Next, the process proceeds to step S7D again, the frame is further switched, and the search area is switched to the left search area 43 again. In this frame, since the face of the person 53 (hereinafter, also referred to as a third face) exists in the left search area 43 during the search, a third face is newly detected in this frame (FIG. 5D ) State). When three faces appear at different timings in the continuous image, the three faces can be recognized as described above.

  In the embodiment described above, when searching for a face newly appearing in the continuous image other than the detected face (step S7 in FIG. 2 and steps S7A to S7D in FIG. 3), the frame is searched left and right. Although an example of equal division into areas has been described, the present invention is not limited to this. The left and right search areas may not be the same size, the search area may be divided up and down, or the frame may be divided into three or more search areas.

  In the above embodiment, when detecting a face newly appearing in the continuous image other than the detected face, search (tracking) of the detected face (for example, the face of the person 51 in FIG. 4). The example which searched the area | region (For example, right search area 33 in FIG.4 (c)) including the area | region was demonstrated. However, the present invention is not limited to this, and an area other than the detected face search area may be appropriately divided and a face newly appearing in the divided area may be searched. The search area division mode can be changed as appropriate according to the number of pixels per frame, the frame rate, the processing capability of the image processing apparatus, and the like.

  Here, with respect to an image in which the number of pixels per frame is 640 × 480 = 307200 pixels, the search area division form when the image recognition method of the present invention is applied, and the search time for each frame in step S7 The specific relationship is shown in Table 1 below. The “upper and lower halves” described in the division form in Table 1 is a division form in which the frame is divided into two equal parts in the vertical direction, and “left and right equal parts” means the frame is divided into two equal parts in the left and right direction It is a division form (the division form of FIGS. 4B to 4D). Further, the “upper and lower left and right equally” in Table 1 is a division form in which the frame is divided into two equal parts in the vertical direction and further divided into two equal parts in the left and right direction.

  As shown in Table 1, the search time for each frame in step S7 in FIG. 2 decreases in proportion to the increase in the number of divisions. When the image size changes, the search time for each frame in step S7 changes almost in proportion to the number of pixels in the frame. Further, the search time shown in Table 1 is an example, and changes depending on the processing capability of the imaging apparatus.

  In the above embodiment, a mobile communication terminal has been described as an example of the imaging device, but the present invention is not limited to this. The present invention can be applied to any device as long as it can capture images continuously. Further, the image recognition method of the present invention does not have a function of capturing a continuous image, but if it is an apparatus (for example, a personal computer) that performs processing for detecting (extracting) a predetermined object image from the continuous image, Applicable to any device.

  Moreover, in the said embodiment, although the face image was taken as an example as the target object image detected in an image, this invention is not limited to this, The target object (for example, car etc.) which can be extracted with a certain image pattern from an image Any image can be applied.

FIG. 1 is a block diagram of a mobile communication terminal according to an embodiment of the present invention. FIG. 2 is a flowchart of image recognition processing according to an embodiment of the present invention. FIG. 3 is a flowchart showing a processing procedure in step S7 in FIG. FIGS. 4A to 4D are views showing the face search and tracking process in steps S3, S5 and S7 in FIG. FIGS. 5A to 5D are views showing the face search and tracking process in steps S3, S5 and S7 in FIG. 2 in the modification.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Mobile communication terminal, 12 ... Control part, 17 ... Clock part, 24 ... Camera, 30 ... Frame, 31 ... Predetermined area, 32 ... Right search area, 33 ... Left search area

Claims (5)

An imaging unit having an imaging element and outputting an image signal;
An image processing unit that searches for and detects an object image that matches a specific condition from the image signal output by the imaging unit, and performs predetermined image processing based on the detection result;
When searching for a second object image in the image while searching and tracking the first object image as the object image in the image processing unit, a part of the image obtained by the image signal An imaging device comprising: a control unit that performs control to search for the second object image while limiting to a region. When the image processing unit searches for the first object image in the predetermined image, the first object image is detected in a predetermined area including the position of the first object image detected in the immediately preceding image. A first processing unit that searches for a first object image, a measurement unit that measures a search time in the first processing unit, and a second object image that searches for a second object image in a partial region of the image based on the measurement result of the search time The imaging device according to claim 1, further comprising: 2 processing units. The imaging device according to claim 2, wherein the second processing unit includes a switching unit that switches the partial area searched for each image to a different area. The imaging device according to claim 1, wherein the object image searched by the image processing unit is a human face image. A first step of searching for a first object image from an image signal obtained by imaging;
A second step of tracking the first object image searched in the first step;
And a third step of searching while limiting the second object image to a partial area of the image while tracking the first object image in the second step.

Images