JP5974148B2 - Image processing apparatus, image data processing method and program - Google Patents

Description

  The present invention relates to an image processing apparatus, a method for processing image data, and a program. In particular, the present invention relates to a technique for detecting a specific subject in an image.

  An image processing method for automatically detecting a specific subject pattern from an image is useful, and can be used, for example, for determination of a human face. Such methods can be used in many areas such as teleconferencing, man-machine interface, security, monitor systems for tracking human faces, image compression, and the like. As a technique for detecting a face from such an image, Non-Patent Document 1 lists various methods.

  In particular, due to its high execution speed and high detection rate, the AdaBoost-based method of Viola et al. Published in Non-Patent Document 2 is widely used in face detection research. The AdaBoost-based method connects different simple discriminant filters in series, and when the discriminant process is determined to be a non-face area, the subsequent discriminant process is terminated, thereby speeding up the discriminant process. Is going on. In order to further speed up the discrimination process, parallelization of the discrimination process and pipeline processing can be considered, but the process is interrupted depending on the contents of the image data and the like, and it is difficult to schedule the discrimination process. There are challenges. In particular, when the discrimination processing is realized by hardware, it is required to realize the discrimination processing by using limited resources with higher efficiency, and the difficulty of this schedule becomes a major issue. .

  In order to describe the problem more specifically, a face detection process using the Boosting algorithm will be described below as an example. FIG. 2 is a diagram for explaining an algorithm for face detection processing to which the Boosting algorithm is applied. In FIG. 2, reference numeral 201 denotes a window image that is a partial image constituting the input image data. The window image 201 includes an image region referred to by weak classifiers 210 to 250 described later.

  The weak discriminator 210 refers to a part or all of the pixel values of the window image 201, performs a predetermined calculation process on the pixel values, and the window image 201 is detected by the detection target object (for example, a human face). Region) is determined. Parameters such as coefficients used for the predetermined calculation are determined by machine learning before actual detection processing.

  The weak classifiers 211 to 250 can perform the same arithmetic processing as the weak classifier 210, but preferably the reference position and reference range of the window image 201 referred to by the weak classifier 210, and the coefficients used for the calculation of the weak classifier 210 The parameters such as are different from each other. The weak classifiers 210 to 250 can execute substantially the same processing as the processing of the weak classifier 210 except that the reference position and range of the window image 201 and parameters such as coefficients used for calculation are different. In FIG. 2, N weak classifiers (weak classifier 210 to weak classifier 250) are connected in series. Each of the weak discriminator 210 to the weak discriminator 250 determines PASS or NG according to the calculation result. Then, the process is performed sequentially from the 0th weak classifier 210, and if the determination result is PASS, the next weak classifier 211 performs the determination process. If the determination result of the weak classifier 250, which is the final weak classifier, is also PASS, it is determined that the window image 201 includes the detection target object (face). On the other hand, when it is determined as NG in the determination process in one of the N weak classifiers, the subsequent processing is aborted, and the detection target object (face) is included in the window image 201. Judge that there is no.

  FIG. 3 shows that the output of each weak classifier is not a binary value of PASS / NG, and the output of each weak classifier is OK (the partial image includes a detection target object (face)) / PASS / NG. This is an example when In FIG. 3, the same components as those in FIG. The weak classifiers 310 to 350 are different from the weak classifiers 210 to 250 in FIG. 2 in that the output is ternary as described above. If the determination process in any one of the N weak classifiers 310 to 350 determines that the partial image includes the detection target object (face) (determination result “OK”), the subsequent determination process Therefore, the determination process is aborted. This is different from the weak classifier described in FIG. The other processes in the N weak classifiers 310 to 350 are the same as those of the weak classifiers described with reference to FIG.

  The face detection process is realized by repeatedly performing the determination process in the weak classifier on a part of a given image. At this time, if the configuration is such that as many determination results as possible can be confirmed (OK or NG) at the beginning of the series of weak classifiers arranged in series, the expected value of the number of processing times of the weak classifiers as a whole can be kept small. it can. Moreover, it can be expected that the calculation amount of the weak classifier is reduced and the processing speed is improved as a whole.

M.H.Yang, D.J.Kriegman and N.Ahuja. “Detecting Faces in Images: A Survey,” IEEE Trans. On PAMI, vol.24, no.1, pp.34-58, January, 2002. P. Viola and M. Jones. “Robust Real-time Object Detection,” in Proc. Of IEEE Workshop SCTV, July, 2001.

  However, from an algorithmic point of view, while the average amount of computation can be reduced as described above, when pipeline processing is performed in order to further increase the processing speed, the processing amount for each window is not uniform, so the processing schedule is not uniform. There is a problem that it is difficult.

  The present invention provides a technique that makes it possible to efficiently perform a discrimination process configured by connecting a plurality of weak discriminators in series to input image data.

An image processing apparatus according to one aspect of the present invention specifies a plurality of partial images, and specifies a plurality of determination processes for determining whether or not a detection target is included in the specified partial image. Instruction means for designating one of the weak discrimination processes and instructing execution of the designated weak discrimination process;
Computing means for performing the designated weak discrimination processing for the designated partial image in accordance with the instruction of the instruction means;
Determination means for determining whether or not the determination processing for the specified partial image is completed based on the result of the specified weak determination processing by the calculation means;
The instructing means sequentially designates partial images other than the partial images determined by the determining means as having been completed in accordance with a predetermined order for the plurality of partial images, and the plurality of weak determination processes Designating one of the unexecuted weak discrimination processes for the designated partial image, sequentially instructing the execution of the designated weak discrimination process,
The computing means sequentially executes the designated weak discrimination processing for the partial images designated in the sequential instruction by the instruction means.

  According to the present invention, it is possible to efficiently perform discrimination processing for a plurality of partial images.

1 is a diagram illustrating a configuration example of an image processing apparatus according to an embodiment. The figure explaining the example of the detection process which applied the Boosting algorithm. The figure explaining the example of the detection process which applied the Boosting algorithm. The figure explaining the example of the detection process which applied the Boosting algorithm with a branch structure. The figure explaining the flow of a process of an instruction | indication part. The figure explaining the flow of the initialization process in the process of an instruction | indication part. The figure explaining the flow of the instruction | indication part state update process in the process of an instruction | indication part. The figure explaining the flow of the job input process in the process of an instruction | indication part. The figure explaining the flow of the memory state update process in the process of an instruction | indication part. The figure explaining the flow of a process of a determination part. The figure which shows the example of a window image. The figure which illustrates the process of a weak discriminator. The figure which illustrates the process of a weak discriminator. The figure which illustrates the process of a weak discriminator. The figure which illustrates the process of a weak discriminator. The figure which shows the flow of operation | movement among an instruction | indication part, a calculating part, a parameter supply part, memory, and a determination part. The figure explaining the flow of the determination part state update process in the process of a determination part. The figure which illustrates the execution order of the discrimination processing with respect to a window image.

(First embodiment)
Embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram illustrating a schematic configuration of an image processing apparatus that realizes an image processing method according to an embodiment. In FIG. 1, an instruction unit 101 performs image processing schedule management for each partial image. The instruction unit 101 receives an external control input (image processing start instruction) and starts image processing. The instruction unit 101 can input and output data with the memory 102. In addition, the instruction unit 101 is configured to be able to output data to the calculation unit 103 and to be able to input data from the determination unit 104.

  The memory 102 can hold, for example, a plurality of window images 201 in FIG. The memory 102 can transmit a request for image data to the outside, acquire image data, and hold the acquired image data. Further, the memory 102 is configured to receive an instruction from the instruction unit 101 and return a response. The memory 102 is configured to be able to receive a request for image data from the calculation unit 103 and output the requested image data to the calculation unit 103. The memory 102 may have a storage capacity capable of storing all of the detection target image data at the same time, but a storage capacity for holding the entire detection target image data is not necessarily required. The memory 102 only needs to hold a plurality of window images. For example, the memory 102 may have a storage capacity capable of storing partial images obtained by dividing image data to be detected into bands or blocks. Further, the memory 102 is configured so that part or all of the image data held in the memory 102 can be replaced in accordance with an instruction from the instruction unit 101. In this case, the memory 102 transmits a request for image data to the outside in response to an instruction from the instruction unit 101, and responds to the instruction unit 101 when reception (acquisition) of the image data is completed. It is configured.

  The calculation unit 103 is configured to be able to execute processing of at least one weak classifier. The calculation unit 103 is configured to transmit a request for image data to the memory 102 and to refer to the image data (a plurality of partial image data) transmitted from the memory 102. The parameter supply unit 105 holds parameters used by the weak classifier for the discrimination process, and is configured to transmit parameters in response to a request from the calculation unit 103. The calculation unit 103 is configured to transmit a parameter request to the parameter supply unit 105 and to refer to the parameter transmitted from the parameter supply unit 105. The determination unit 104 receives the output of the calculation unit 103, makes a determination, and outputs a determination result. At this time, the determination result is also transmitted to the instruction unit 101.

  A plurality of window images held in the memory 102 will be exemplarily described with reference to FIG. The memory area 1110 represents an area that can be referenced from the memory 102 by the arithmetic unit 103, and the smallest rectangular area 1111 inside the memory area 1110 represents each pixel data. Reference numerals 1101 to 1105 denote a plurality of window images set in the memory area 1110. For example, the window image 1101 is in an area having a width of 10 pixels and a height of 10 pixels from the position (0, 0) of the memory area 1110. Similarly, the window image 1102 is in an area of 10 pixels from the position (0, 2) of the memory area 1110 in both width and height. The window image 1103 is in an area of 10 pixels from the position (0, 4) of the memory area 1110 in both width and height. The window image 1104 is in an area of 10 pixels from the position (0, 6) of the memory area 1110 in both width and height. The window image 1105 is an area having 10 pixels in both width and height from the position (0, 8) of the memory area 1110. Window images 1101 to 1105 correspond to the window image 201 in FIG. 2, but are window images having different positions of the original images.

  FIG. 16 is a sequence diagram illustrating the processing flow of the instruction unit 101, the calculation unit 103, the parameter supply unit 105, the memory 102, and the determination unit 104 described in FIG. For example, as described with reference to FIG. 2, the target object detection process sequentially applies N weak classifiers to one window image and passes through all N weak classifiers (determined as PASS). The window image includes the target object. Alternatively, if any weak classifier determines NG, the process of the subsequent weak classifier is canceled at that time, and it is determined that the window area does not include the target object. Each of the N weak classifiers uses different parameters for processing, but the content of the processing is the same. The memory holds a plurality of window images at the same time.

  When the instruction unit 101 receives an instruction to start image processing from the outside (S1601), the instruction unit 101 transmits an image update request to the memory 102. When receiving an image update request from the instruction unit 101, the memory 102 requests image data from the outside, and stores image data transmitted from the outside in response to the image data request. A response (OK) is returned to the instruction unit 101 when a sufficient amount of data for composing a plurality of window images is accumulated.

  The instruction unit 101 instructs the calculation unit 103 to execute a discrimination process (job instruction). The instruction unit 101 gives the calculation unit 103 identification information (ID) to the window image or weak classifier that is the target of the discrimination process. The processing of one weak classifier for one window image is hereinafter referred to as a discrimination job or simply a job. The discrimination job is instructed by transmitting a pair of identification information (ID) of the window image and identification information (ID) of the weak discriminator to the calculation unit 103. In step S1602, S1603, and S1604, the instruction unit 101 determines a determination job to be processed next, and transmits a determination job instruction to the calculation unit 103. The generation of the discrimination job in steps S1602 to S1604 will be described later.

  The calculation unit 103 receives an instruction from the instruction unit 101 and transmits a parameter request to the parameter supply unit 105. The parameters include a coefficient used in the subsequent arithmetic processing step S1611, a threshold value for determining whether to continue (PASS) or abort (end: NG) the determination process, and the position and size of the reference area in the window image. Information about When the calculation unit 103 receives the parameter transmitted from the parameter supply unit 105, the calculation unit 103 transmits a request for image data for referring to the image data necessary for the calculation process to the memory 102 according to the parameter. The calculation unit 103 receives the image data transmitted from the memory 102, performs a predetermined calculation using the received image data and the previously received parameter (S1611), and the calculation result (PASS / NG) is transmitted to the determination unit 104.

  Steps S <b> 1612 and S <b> 1613 are arithmetic processing for a discrimination job different from step S <b> 1611.

  The discrimination jobs in steps S1612 and S1613 differ in at least one of the identification information (ID) of the window image and the identification information (ID) of the weak discriminator. In response to this, a parameter request and image data request are made respectively, but the contents of the processing are the same as in step S1611.

  The determination unit 104 receives the calculation result of step S1611 from the calculation unit 103, performs determination processing (S1621), and transmits the result to the instruction unit 101. The determination process will be described later. Note that step S1622 and step S1623 are the same processing except that the determination job to be determined is different. When the determination processing is completed, the determination result is transmitted to the instruction unit 101 as in step S1621.

  The timing of the determination job instruction of the instruction unit 101 or the transmission of the calculation result of the calculation unit 103 and the transmission of the determination result of the determination unit 104 are shifted from each other. It may be timing. For example, the determination job instruction for the result of job generation in step S1604 and the transmission of the determination result that is the result of the determination process in step S1621 can occur almost simultaneously.

  FIG. 16 exemplarily shows a case where the determination result is confirmed in the determination processing in step S1621. The instruction unit 101 receives the determination result in step S1621 from the determination unit 104. When the result indicating that the object (specific pattern) is included in the window image is included in the determination result, it is detected that the detection processing for the window image has ended (S1605). Thereafter, the instruction unit 101 stops generating the discrimination job.

  However, it is step S1606 after receiving the determination result transmission by the determination process of step S1623 that all the processes of each unit shown in FIG. In step S <b> 1606, the number of jobs instructed by the instruction unit 101 to the calculation unit 103 by job generation and the determination unit 104 transmitted to the instruction unit 101 in order to determine whether or not the entire processing has been completed. Count the number of results. After the window image processing end detection in step S1605, the instruction unit 101 determines that the entire processing has ended at the timing when the counted numbers match.

  If the determination processing in steps S1622 and S1623 is a determination job for the same window image as the window image for which the determination result has been confirmed in step S1621, the determination unit 104 determines that the determination processing for the window image has been completed. To do. The determination unit 104 can determine whether or not they are the same window image by comparing identification information (ID). The instruction unit 101 also detects that the detection process for the window image has ended in step S1605, and it is known that the window image determination process has ended. In such a case, the dummy data can be returned to the instructing unit 101 by omitting the determination processing in steps S <b> 1622 and S <b> 1623 in order to reduce the load of the overlapping arithmetic processing.

  In such a case, for example, the results of the calculation processes of the calculation processes S1612 and S1613 do not need to be accurate. Therefore, the instruction unit 101 may instruct the memory 102 to replace image data after the determination processing for all window images is confirmed in the determination processing in step S1621. The memory 102 can be configured to return a response to the image data request of the calculation unit 103 to the memory 102 even during the replacement of the image data. Then, the calculation unit 103 can advance the calculation, while reducing the calculation load of the overlapping determination process in the determination unit 104.

(Processing of the instruction unit 101)
FIG. 5 is a flowchart showing the processing flow of the instruction unit 101. The window images held by the memory 102 are W (W> 0), and are distinguished by giving identification information from 0 to (W-1). For example, in the example of FIG. 11, identification information (ID) of window image identification information (ID) = 0 to 4 is given to the window images 1101 to 1105 in order.

  Further, identification information (ID) of weak classifiers is assigned in order from the weak classifier arranged at the head, and identification information (ID) of weak classifiers is assigned. The determination process is sequentially executed in order. First, initialization processing is performed in step S501. Details of the initialization process will be described with reference to FIG.

  First, in step S601, current holding the identification information (ID) of the window image to be processed next is initialized with W-1. The W-1 is set to W-1 in order to start the processing from the window image identification information (ID) 0 in the discrimination job input process described later, but the value may be clearly initialized, You may initialize with the identification information (ID) of another window image. Also, init_flag indicating the first process is set to 1.

  In step S602, a counter i for counting the number of times of processing in steps S603 to S604 is initialized to zero. In step S603, the end of the loop process is determined. It is determined whether or not the process of S604 has been performed N times. If the process has not yet been completed (S603-Yes), the process of step S604 is repeated. If the process has been completed (S603-No), the process ends.

  In step S604, the value of array win_status [i] is initialized to 0, the value of counter i is counted up, and the process returns to step S603.

  In this embodiment, win_status is used as a variable for managing the state of each window image. win_status has W arrays and holds the value of the weak classifier ID to be processed next. Further, when this value is a negative value, it means that the discrimination processing for the window has been completed. At the initialization process stage, 0 is set as identification information (ID) of the weak classifier to be processed next.

  When the initialization process (S501, FIG. 6) described in FIG. 6 is completed, the process proceeds to step S502 in FIG. In step S502, a memory update process is performed on the memory 102. Details of the memory update processing will be described with reference to FIG. In step S <b> 901, the instruction unit 101 transmits an image update request to the memory 102. When receiving an image update request from the instruction unit 101, the memory 102 requests image data from the outside, and accumulates image data transmitted from the outside in response to the image data request. In step S <b> 902, the memory 102 returns a response (OK) to the instruction unit 101 when a data amount sufficient to form a plurality of window images is accumulated.

  When the memory update process (S502, FIG. 9) ends, the process proceeds to step S503 in FIG. In step S503, the presence / absence of an output from the determination unit 104 is confirmed. If there is an output from determination unit 104 (S503-Yes), the process proceeds to step S504. In step S504, the output (determination result) of the determination unit 104 is read. In step S505, the state of the instruction unit 101 is updated according to the output (determination result) of the determination unit 104. Details of the instruction unit state update processing will be described with reference to FIG.

  First, it is assumed that the output (determination result) of the determination unit 104 includes the determined window image identification information (ID) and the determination result. In step S <b> 701, the instruction unit 101 reads the window image identification information (ID) and the determination result from the output (determination result). Here, as a determination result, it is sufficient to know at least whether the processing is continued (PASS) or the processing is ended. For example, as the determination result returned by the determination unit 104, “0” is the processing continuation and “1” is the processing end. For example, in the case of FIG. 2, the process end corresponds to a case where the job determination result is NG or a case where the last weak classifier 250 is PASSed. Furthermore, in the case of FIG. 3, the process may be terminated when OK is determined by a weak classifier on the way. Also, as in steps S1622 and S1623 in FIG. 16, the result of a job for a window image for which the determination process has already been completed can also correspond to the end of the process.

  As for these determination results, meaning may be assigned to the signal in more detail, but it is only necessary to determine at least whether the window image is to be continued or terminated.

  In step S702, it is determined whether or not it is an instruction to continue processing for the window image whose window image identifier output by the determination unit 104 is represented by (win_ID). If the determination result (result) = 0, it is determined that the process is continued (S702-Yes), and the process ends. If it is determined in step S702 that the determination result (result) = 1, it is determined that the process has ended (S702-No), and the process proceeds to step S703.

  In step S703, the window image is invalidated. As described above, in order to invalidate win_status [win_ID], -1 is set as the value of the determination result, and the process ends.

  The description returns to step S503 in FIG. If it is determined in step S503 that there is no output from the determination unit 104 (S503-No), the process proceeds to step S506. In step S506, it is determined whether or not there is a vacancy in the calculation unit 103 to input the determination job. If there is no space (S506-No), the process proceeds to S508. On the other hand, if there is a vacancy in the calculation unit 103 to input the discrimination job, the process proceeds to the job input process in step S507. Details of the job submission process will be described with reference to FIG.

  First, in step S801, the instruction unit 101 updates the current value. In addition, the instruction unit 101 initializes a counter cnt that controls the processing of steps S802 to S804 to zero. In step S802, it is determined whether or not to exit the loop process. If cnt ≧ W (S802-No), the process ends. The determination of No in step S802 may correspond to a case where there is no window image to be input next as a determination job.

  If it is determined in step S802 that cnt <W (S802-Yes), the process proceeds to step S803. In step S803, it is determined whether the current window image is valid. If win_status [current] <0, it is determined that the processing of the current window image has been completed, that is, the window image is invalid (S803-Yes). If the window is invalid (S803-Yes), the process proceeds to step S804. In step S804, the current value and the cnt value are updated, the process returns to step S802, and the same process is repeated. If a valid window is found in the determination in step S803 (S803-No), the process proceeds to step S805.

  In step S805, the instruction unit 101 transmits current as the window image ID and win_status [current] as the weak classifier ID as a discrimination job to the calculation unit 103. The instruction unit 101 transmits a flag init_flag together with the determination job. If this process is the first transmission of a discrimination job for a window image, init_flag = 1 is set in the initialization process (S601 in FIG. 6), so init_flag = 1 is transmitted.

  In step S806, “1” is added to win_status [current], win_status [current] is updated, and init_flag is set to “0”. The job input process is completed by the above process. When the instruction unit state update process (S505, FIG. 7) or the job input process (S507, FIG. 8) ends, the process proceeds to step S508 in FIG. In step S508, it is determined whether there is a valid window image. Here, if all the elements of win_status are negative values, it is determined that there is no valid window image (S508-Yes). On the other hand, if it is determined in step S508 that there is a valid window image (S508-No), the process proceeds to step S503, and the same process is repeated thereafter.

  If it is determined in step S508 that there is no valid window image (S508-Yes), the process proceeds to step S509, and it is determined whether or not to end the process. For example, in the control input from the outside, information indicating how many times the window images are replaced and the determination process is completed is received, and the number of times the processes in steps S501 to S509 in FIG. 5 are performed is counted. Just keep it. Based on this comparison, the authenticity of step S509 may be determined, or in the determination of step S509, it is possible to receive an instruction from the outside whether to continue or to end, and to perform the end determination based on that It is. If it is determined in step S509 that the process is not terminated (S509-No), the process returns to step S501, and the same process is repeated.

  Step S510 is processing corresponding to step S1622 and step S1623 in FIG. 16, and is processing for waiting for a response from the determination unit 104.

(Processing of the determination unit 104)
The processing of the determination unit 104 will be described with reference to FIG. In step S <b> 1001, the determination unit 104 stands by in a state waiting for input from the calculation unit 103. In step S1002, win_ID is received from the calculation unit 103 as the window image ID, clf_ID as the weak classifier ID, val as the calculation value, and init_flag as the initialization flag. The window image ID, weak classifier ID, and init_flag are the same as those transmitted by the instruction unit 101 to the calculation unit 103 in step S805 of FIG.

  In the subsequent step S1003, the process is branched depending on the value of init_flag. If init_flag is 1 (S1003-Yes), the initialization process of step S1004 is performed. If not (S1003-No), step S1004 is skipped and the process proceeds to step S1005.

  In the initialization process in step S1004, a variable for managing whether or not each window image has been determined is initialized. For example, the information win_status2 [win_ID] (array: number of elements W) indicating the judgment status is provided, and if the value of the element is 0, the processing of the window image represents the judgment (state is confirmed), otherwise It is assumed that the window image processing has not been determined. In this case, a non-zero value (window is valid), for example, 1 may be set for each of those elements. Also, the value of the send_flag flag used in step S1008 (used to determine whether the result has been transmitted) is initialized to zero. Further, information win_detect [] array (number of elements W) indicating the detection result of each window image is provided, and 0 is set to each element. The win_detect [] array stores the detection result of each window, where 0 indicates no detection and 1 indicates detection.

  In step S1005, the internal state of the determination unit 104 is updated. This updates the value of win_status2 [win_ID] based on the result output from the calculation unit 103. For example, in the case of the example of FIG. 2, if the calculation result of the calculation unit 103 is NG or if the state is determined by PASSing the last weak discriminator (N-1) (250 in FIG. 2), win_status2 [ Set win_ID] to 0. Details of step S1005 will be described with reference to FIG.

  First, in step S1701, it is determined whether the specified window image has not been determined yet or has been completed. This may be determined based on the win_ID received from the calculation unit 103, referring to the corresponding element of the win_status2 [] array, that is, win_status2 [win_ID], and comparing the value with 0. If win_status2 [win_ID] is non-zero, the window has not been determined (S1701-Yes), and the process proceeds to step S1702. If win_status2 [win_ID] = 0 (S1701-No), the processing in FIG. 17 ends.

  In subsequent step S <b> 1702, the determination unit 104 determines whether or not the calculation value input from the calculation unit 103 represents PASS. The calculation value is determined using a threshold value corresponding to the weak classifier. If it is less than the threshold value, it is determined as NG, otherwise it is determined as PASS. When the threshold value varies depending on each weak classifier, the threshold value may be supplied from the parameter supply unit 105. In FIG. 1, the parameter supply unit 105 is not directly connected to the determination unit 104, but can be supplied with a threshold value via the calculation unit 103, for example. Alternatively, although not shown in FIG. 1, the parameter supply unit 105 and the determination unit 104 may be connected, and the threshold value may be supplied via the connection. The correspondence between the magnitude relationship with the threshold value and NG / PASS is an example, and conversely, for example, the calculation may be PASS if it is less than the threshold value, or NG otherwise.

  If the determination result in step S1702 is true, the process proceeds to step S1703; otherwise, the process proceeds to step S1705. The determination in step S1703 is a determination as to whether or not the weak classifier being processed is the final weak classifier (250 in FIG. 2). This may be compared with N-1 based on clf_ID received as the weak classifier ID from the arithmetic unit 103. In the case of step S1703—Yes, the weak discriminator to be determined is not the final discriminator, and the process ends. On the other hand, in the case of step S1703-No, since the weak classifier being processed is the final weak classifier, if it is PASS, the face that is the detection target is processed as “detected”. In step S1704, the final weak discriminator is PASSed to determine the determination state. The value of information win_status2 [win_ID] indicating the determination state is set to 0. Further, in order to store “detected”, the value of the information win_detect [win_ID] indicating the detection result is set to 1, and the process ends. As a detection result of each window image, for example, 0 represents no detection, and 1 represents detection.

  In step S1705, when the determination result of the weak discriminator is NG in the determination of S1702, determination that the target object (specific pattern) is not included in the partial image is confirmed, so information indicating the determination state win_status2 [win_ID ] Is set to 0, and the process ends. When the process of step S1005 (FIG. 17) ends, the process proceeds to step S1006 of FIG. 10, and the result of the determination unit state update process is notified to the instruction unit 101.

  In step S1007, the win_status2 [] array is referred to as to whether there is a valid window image. If there is no valid window image (S1007-N0), the process proceeds to step S1008; otherwise, the process proceeds to S1001. Return. In step S1008, the value of the flag send_flag used to determine whether or not the result has been transmitted is checked, and the process is branched. If the value of the flag send_flag is 1 (S1008-Yes), the process proceeds to step S1009; otherwise, the process returns to step S1001 (S1008-No).

  In step S1009, determination result output processing is executed. This is based on the value of each element of the win_detect [] array, and outputs the presence / absence of area detection of each window image. The output can be in any format.

  In step S1010, the value of send_flag is set to 0. This is because, as described with reference to FIG. 16, there is a possibility that the calculation result of the job may be input from the calculation unit 103 even after all the window image detection is completed. This process is for performing the process of step S1009 only once in the flowchart of FIG. 10 because there is a possibility that the path to step S1008 may be passed a plurality of times. After executing step S1010, the determination in step S1008 is false (S1008-No), and step S1009 is not executed. If 1 is given as init_flag, send_flag is set to 1 by the initialization processing in step S1004, and the determination in S1008 becomes true again.

  The processing of the calculation unit 103 may be arbitrary as long as it is weak discrimination processing in known detection processing. The outline is configured as follows. First, out of a plurality of window images set in the memory 102, a window image is specified based on the window image ID designated as a discrimination job from the instruction unit 101.

  Subsequently, a part or all of the pixel values in the window image are referred to, and the corresponding parameter is supplied from the parameter supply unit 105 based on the ID of the weak classifier of the discrimination job. Perform the operation used. Then, the calculation value is output to the determination unit 104 together with the information of the determination job.

  The memory 102 acquires and holds image data from the outside in response to an instruction from the instruction unit 101. For example, the memory 102 can hold a plurality of window images as shown in FIG. It is only necessary that a plurality of window images exist in the memory 102 at the same time, and the windows do not necessarily overlap with other windows as shown in FIG.

  The parameter supply unit 105 provides parameters for determination processing in response to a request from the calculation unit 103. The parameter supply unit 105 is configured to hold all parameters for discrimination processing, or a large-capacity memory is provided outside, and the parameter supply unit 105 is configured to operate like a cache for the large-capacity memory. May be. When the parameter supply unit 105 is configured to operate like a cache, the parameter supply unit 105 preferably receives the output from the instruction unit 101 directly. Further, the instructing unit 101 can also be configured to predict the progress of the processing and prefetch the parameters for determination processing required for the parameter supply unit 105 in advance. The instruction unit 101 predicts the progress of the process, for example, with respect to the process as shown in FIGS. 2 to 4, for example, N (N ≧ 0) ahead of the weak discriminator that instructs the operation unit 103. The parameter supply unit 105 is instructed to prefetch the discrimination parameter used in the weak discriminator. N may be determined in advance, or may be determined dynamically by predicting the progress of weak classifier processing by some method.

(Weak classifier processing)
Next, processing of the weak classifier will be described with reference to FIGS. The processing method of the weak discriminator is applicable to the discrimination processing of the weak discriminators connected in a straight line as shown in FIGS. 2 and 3, but is not limited to this example, and has the branch structure shown in FIG. It can also be applied to discrimination processing. Weak classifiers connected in a straight line (for example, weak classifier (1, 0) (1, P-1) or weak classifier (2, 0) (2, Q-1) Or the weak discriminator (3, 0) (3, R-1)). In addition, if the processing of the weak discriminator at the branch destination is treated virtually like the processing for another detection window image, it is obvious that the present invention can be applied even when there are multiple stages of branching.

  For the sake of simplicity, it is also said that the processing for the detection target window image is completed when it is determined whether the detection target window image includes or does not include the target object (specific pattern). 12 to 15 show time charts when the discrimination job is processed by the instruction unit 101, the calculation unit 103, and the determination unit 104. FIG. 12 to 15, the horizontal axis is a time axis. In FIG. 12, there are 15 and 18 time slots between times t0 and t15, and in FIGS. 13 to 15 between times t0 and t18, and each section processes one discrimination job in each time slot. To do. A square arranged in the time slot represents a discrimination job, Wx indicates that the window image ID is x, and Sy indicates that the weak classifier ID is y. For simplicity of explanation, it is assumed that there are three window images having IDs 0 to 2 (0 ≦ x ≦ 3) at the maximum in the group of window images. Further, there are six weak classifiers with IDs from 0 to 5 (0 ≦ y ≦ 5), and when the weak classifier S5 is PASSed, it is detected that there is an object (specific pattern). A discrimination job in which window image ID = Wx and weak discriminator ID = Sy is also expressed as Wx / Sy.

  In FIG. 12 to FIG. 15, the processing of the discrimination job represented by the solid line square is performed on the first group of window images. Then, when it is detected that the processing for all window images has been completed, the memory is replaced at the time point t13 of the instruction unit 101, and a discrimination job represented by a broken-line square is performed as the next window image group. To do. However, FIG. 13 is a time chart relating to one window image group, and a second window image group, that is, a broken line window image group discrimination job as shown in FIG. 12 is not written. In addition, the squares represented by shading indicate that the job is actually a discrimination job that has been processed by a weak discriminator before the weak discriminator corresponding to the discrimination job. 12 to 15 illustrate differences in processing by the three methods 1 to 3. System 1 to system 3 operate as follows.

(Method 1)
A discrimination job is speculatively instructed until the processing for a certain detection target window image is completed. When it is detected that the discrimination processing has been completed, the discrimination processing instruction for the window image is stopped, and a discrimination job for the next window image is instructed.

(Method 2)
A discrimination job for the same weak classifier for each window image is instructed, and the discrimination result is awaited. If the discrimination result is received and the processing of the window image is not completed, the discrimination job for the next weak discriminator is instructed.

(Method 3)
The image data processing according to the embodiment of the present invention is based on this method. The discriminating job for the discriminator to be performed next is instructed while circulating through the window image that has not been discriminated at the discriminating job.

  FIG. 12 shows an example in which three window images W0 to W2 exist in the first window image group, and the determination result is as shown in Table 1 below.

  When the case of Table 1 is processed by Method 1, Table 2 is obtained. Table 2 summarizes the time and the processing of the instruction unit and the determination unit for the processing by method 1. In Table 2, the shaded portion of the cell is a discrimination job that has been processed by the weak discriminator before the weak discriminator corresponding to the discrimination job. A blank cell indicates a state where the discrimination job is not processed.

  In the case of the example in Table 2, it is the timing at time t13 that the processing end of the group of window images is detected by the method according to method 1.

  When the case of Table 1 is processed by the method 2, Table 3 is obtained. Table 3 summarizes the time and the processing of the instruction unit and the determination unit for the processing by method 2. The notation in Table 3 is the same as the notation in Table 2.

  In the case of the example in Table 3, the end of processing of the window image group by the method 2 is detected at time t12.

  Next, method 3 in FIG. 12 will be described. In this method, the order in which window images are processed is defined. The instruction unit 101 can set the execution order of the discrimination processing for a plurality of window images by an external control input. The set execution order is included in the discrimination job and transmitted from the instruction unit 101 to the calculation unit 103. The calculation unit 103 performs a determination process for a plurality of window images (partial images) according to the execution order included in the determination job. FIGS. 18A, 18B, and 18C are diagrams illustrating the execution order of the discrimination processing for the window image. Examples of three window images (W0 to W2) are combined with the example of FIG. Show. It is assumed that window images 1801 to 1803 in FIG. 18 correspond to window images W0 to W2 in FIG. In FIGS. 18A, 18B, and 18C, the solid line rectangles indicate unprocessed window images, and the broken line rectangles indicate processed window images. It should be noted that the number of window images is not limited to three examples, and it is obvious that the order of circulation can be defined even when there are four or more window images.

  FIG. 18A illustrates a state in which all window images have not been processed. For example, a weak discriminator that performs discrimination processing is referred to as weak discriminator Si. The window image processed first by the weak classifier Si is defined as a window image W0 (1801). The window image W1 (1802) is the target of processing next to the window image W0 (1801). The window image W2 (1803) is to be processed next to the window image W1 (1802). When the window images W0 (1801) to W2 (1803) are not processed by the weak classifier Si, the window images W0 (1801) to W2 (1803) are processed by the next weak classifier Si + 1. The The object to be processed by the next weak discriminator Si + 1 is the window image W0 (1801) in order from the window image W0 (1801) in the order next to the window image W2 (1803). 1801), W1 (1802), and W2 (1803) are processed. FIG. 18B illustrates a state in which the window image W1 (1802) has been processed, and the processed window image W1 (1802) is excluded from the next determination processing target. The instruction unit 101 selects a window image (for example, W1) when at least one window image (partial image) determined to be aborted (terminated) of the discrimination process among a plurality of window images (partial images) is included. A new execution order (W0 → W2) is set. In this case, the window image W2 (1803) is to be processed next to the window image W0 (1801). FIG. 18C illustrates a state where there is one window image that has not been processed. For example, if the window image W0 (1801) is not processed by the weak discriminator Si as the weak discriminator Si that performs the discriminating process, the next weak discriminator Si + 1 is also subject to discrimination processing by the window. This is the image W0 (1801).

  When the case of Table 1 is processed by Method 3, Table 4 is obtained. Table 4 summarizes the time and the processing of the instruction unit and the determination unit for the processing by method 3. The notation in Table 4 is the same as the notation in Table 2.

  The window image W2 is the target of processing next to the window image W1 instructing the discrimination job by the discriminator S1 at time t4. Based on the W2 / S0 determination result (NG) at time t4, it is known in the instruction unit 101 at time t5 that the determination processing for the window image W2 has been completed. In this case, the instruction unit 101 further searches for the next window image W0 and instructs the W0 / S2 determination job again because the window image W0 has not been processed (t5).

  Similarly, at time t8, the window image W1 is subject to processing next to the window image W0 that has instructed the discrimination job by the weak classifier S3 again at time t7. Based on the W1 / S1 determination result (NG) at time t6, the determination processing for the window image W1 is completed. At times t7 and t8, discrimination processing for window images for which discrimination processing has not been completed is sequentially output (WO / S3, WO / S4). During this time, the window images W1 and W2 are excluded from the discrimination targets. At the timing of time t9, the determination unit 104 determines the determination result (NG) of WO / S3, and at the timing of time t10, the instruction unit 101 detects the end of the processing of the window image group. As described above, in the example of FIG. 12, the method 3 according to the embodiment of the present invention can process a group of window images earliest.

  As another example, FIG. 13 shows an example in which it is determined that all the window images W0 to W2 include a detection target (specific pattern) as shown in Table 5.

  In the case of FIG. 13, the instruction unit 101 detects that the processing of the group of window images is completed at the time t18 in all three methods, and there is no difference in processing speed between the three methods in this example.

  FIG. 14 shows an example in which all the window images W0 to W2 are determined to be NG by the first weak discriminator S0 as shown in Table 6 as yet another example.

  When the case of Table 6 is processed by Method 1, Table 7 is obtained. Table 7 summarizes the time and the processing of the instruction unit and the determination unit for the processing by method 1. The notation in Table 7 is the same as the notation in Table 2.

  As shown in Table 7, in method 1, the instruction unit 101 detects the end of the processing of the group of the window images W0 to W2 at time t9.

  When the case of Table 1 is processed by the method 2, Table 8 is obtained. Table 8 summarizes the time and the processing of the instruction unit and the determination unit for the processing by method 2. The notation in Table 8 is the same as the notation in Table 2.

  As shown in Table 8, in the method 2, the instruction unit 101 detects the end of the processing of the group of the window images W0 to W2 at time t5.

  When the case of Table 1 is processed by the method 3, Table 9 is obtained. Table 9 summarizes the time and the processing of the instruction unit and the determination unit for the processing by method 3. The notation in Table 9 is the same as the notation in Table 2.

  As shown in Table 9, in method 3, the instruction unit 101 detects the end of the processing of the group of the window images W0 to W2 at time t5. The method 3 can end the window image group processing at the same time t5 as the method 2. Also, method 3 can process a group of window images faster than method 1.

  FIG. 15 shows, as another example, as shown in Table 10, among the window images W0 to W2, the window images W0 and W1 are determined to be NG by the first weak discriminator S0, and the window that has not been processed is 1 An example of the case of only one is shown.

  When the case of Table 10 is processed by Method 1, Table 11 is obtained. Table 11 summarizes the time and the processing of the instruction unit and the determination unit for the processing by method 1. The notation in Table 11 is the same as the notation in Table 2.

  As shown in Table 11, in Method 1, the instruction unit 101 detects the end of the processing of the group of the window images W0 to W2 at time t10.

  When the case of Table 10 is processed by the method 2, Table 12 is obtained. Table 12 summarizes the time and the processing of the instruction unit and the determination unit for the processing by method 2. The notation in Table 12 is the same as the notation in Table 2.

  As shown in Table 12, in method 2, the instruction unit 101 detects the end of the processing of the group of the window images W0 to W2 at time t8.

  When the case of Table 10 is processed by method 3, it becomes as shown in Table 13. Table 13 summarizes the time and the processing of the instruction unit and the determination unit for the processing by the method 3. The notation in Table 13 is the same as the notation in Table 2.

  As shown in Table 13, in the method 3, the instruction unit 101 detects the end of the processing of the group of the window images W0 to W2 at time t7. Method 3 can process a group of window images faster than methods 1 and 2.

  In the examples shown in FIGS. 12 to 15, the method 3 can process a group of window images at the same or higher speed than the other methods 1 and 2.

  In the present embodiment, the description has been made assuming that the memory 102 stores still image data, but it may be moving image data. It is also possible to store a plurality of frames in the memory 102 and refer to them for processing. The memory 102 is not limited to a single memory, and may be a plurality of memories. For example, it may be configured by three memories, and each of the frames being noticed and the frames before and after the frames may be stored. In this case, when the calculation unit 103 executes the determination process for each of the plurality of memories, the association information for associating the window image of each memory with the parameter to be referred to in the determination process is the parameter supply unit 105. Has been granted. When the parameter supply unit 105 supplies parameters to the calculation unit 103, association information is also supplied to the calculation unit 103. The calculation unit 103 refers to the association information, determines the association with each memory, and specifies the parameter to be referred to in the window image discrimination process.

  As an embodiment, an example has been described in which a still image is two-dimensional data with a spatial axis and a moving image is three-dimensional data with a time axis added. In addition, the present invention can also be applied to multidimensional data or one-dimensional data such as voice (in the case of voice, the time axis may be set to two dimensions). In the present embodiment, the specific object is detected. However, in the case of general multidimensional data, this corresponds to detecting a specific pattern of multidimensional data.

  Further, in the present embodiment, what is held in the memory 102 is the image data itself. For example, a feature amount (for example, an edge amount, a difference between frames, etc.) is extracted as preprocessing, and is stored in the memory unit. It may be. Note that there is no need to limit the feature amount to one type. For example, it is possible to extract a plurality of feature amounts from one image and perform determination processing using the extracted feature amounts. In this case, as in the case of moving image data, information about which feature quantity is referred to is assigned to the parameter held by the parameter holding unit, and the calculation unit 103 specifies data to be referenced based on the information. Configure. Furthermore, in the present embodiment, the memory 102 has been described as holding normal image data, but it is also possible to hold in the form of an integral image.

  As described above, according to the present embodiment, it is possible to efficiently perform a discrimination process configured by connecting a plurality of weak discriminators in series to input image data.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

  101: Instruction unit, 102: Memory, 103: Calculation unit, 104: Determination unit, 105: Parameter supply unit

Claims (7)

Specify one of a plurality of partial images, specify one of a plurality of weak discrimination processes for discrimination processing to determine whether the designated partial image includes a detection target, and perform the designation Instruction means for instructing execution of the performed weak discrimination processing;
Computing means for performing the designated weak discrimination processing for the designated partial image in accordance with the instruction of the instruction means;
Determination means for determining whether or not the determination processing for the specified partial image is completed based on the result of the specified weak determination processing by the calculation means;
The instructing means sequentially designates partial images other than the partial images determined by the determining means as having been completed in accordance with a predetermined order for the plurality of partial images, and the plurality of weak determination processes Designating one of the unexecuted weak discrimination processes for the designated partial image, sequentially instructing the execution of the designated weak discrimination process,
The image processing apparatus according to claim 1, wherein the arithmetic means sequentially executes designated weak discrimination processing for the partial images designated in the sequential instruction by the instruction means.   The calculation means does not wait for the result of the determination even if the determination by the determination means based on the result of the weak determination process previously specified for the partial image specified by the instruction means is incomplete. The image processing apparatus according to claim 1, wherein newly designated weak discrimination processing is executed on the partial image.   The instruction means designates a next unexecuted weak discrimination process for the designated partial image among the plurality of weak discrimination processes according to a predetermined order for the plurality of weak discrimination processes. The image processing apparatus according to claim 1.   The determination unit determines that the determination process for the specified partial image is completed when it is determined that the specific pattern is not included in the specified weak determination process by the calculation unit. Item 4. The image processing device according to any one of Items 1 to 3.   The determination unit determines that the determination process for the specified partial image is completed when it is determined that the specific pattern is included in all of the plurality of weak determination processes for the specified partial image. The image processing apparatus according to claim 1, wherein: An image processing method executed by an image processing apparatus,
Instructing means of the image processing apparatus designates one of a plurality of partial images, and a plurality of weak discrimination processes for discrimination processing for discriminating whether or not a detection target is included in the designated partial image. An instruction process for designating one and instructing execution of the designated weak discrimination process;
A computing step in which the computing means of the image processing device performs the designated weak discrimination processing for the designated partial image in accordance with the instruction of the instruction step;
A determination step for determining whether or not the determination processing for the specified partial image is completed based on a result of the specified weak determination processing by the calculation step. And
The instructing step sequentially designates partial images other than the partial images determined by the determination step to be completed according to a predetermined order with respect to the plurality of partial images, and the plurality of weak determination processes Designating one of the unexecuted weak discrimination processes for the designated partial image, sequentially instructing the execution of the designated weak discrimination process,
The image processing method characterized in that the calculation step sequentially executes designated weak discrimination processing for the partial images designated in the sequential instruction in the instruction step.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 5.