JP2018147240A - Image processing device, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To hasten the timing to start detecting while preventing the accuracy from decreasing in object detection processing in which learning model data having a plurality of stages is applied.SOLUTION: In an image processing device 10 that detects feature points of an object by correcting, in stages, the initial feature points within an input image by applying learning model data of N (N is an integer of 2 or more) stages each of which contribute to the derivation of correction vectors of the feature points, at least at the time of start-up, a correction vector derivation unit and a feature point correction unit start the correction processing of the feature points before it completes loading the learning model data of the N stages into a work area unit 15, and the correction vector derivation unit derives, when it derives the correction vector of a stage t(1≤t≤N) and if the learning model data of the stage t has not been loaded into the work area unit 15, the correction vector by substituting the learning model data with that of another stage.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program that detect feature points of a predetermined object.

  In recent years, methods have been proposed for detecting feature points of an object (for example, a human face) in an input image using a shape regression model. The shape regression model is a model in which initial feature points are sequentially approximated to correct feature points by a correction process of a plurality of stages, and requires learning model data for a plurality of stages.

JP-A-2006-66114

  In the feature point detection using the conventional shape regression model, when loading large-capacity learning model data to the work area at startup, the object detection process can be started if all the learning data has not been loaded. It took a long time from startup to detection start.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a technique for accelerating detection start timing while suppressing deterioration in accuracy in object detection processing using learning model data of a plurality of stages. .

  In order to solve the above problems, an image processing apparatus according to an aspect of the present invention uses N (N is an integer of 2 or more) stages of learning model data, each of which contributes to the derivation of a feature point correction vector. An image processing apparatus for detecting feature points of an object by correcting initial feature points in an image stepwise, a work area unit for loading the N-stage learning model data, and an initial stage in an input image Refer to the learning model data of the corresponding stage held in the work area based on the feature quantity around each feature point or the feature quantity around each feature point modified in the previous stage. Then, a correction vector deriving unit for deriving a correction vector of each feature point, and a feature point correcting unit for correcting each feature point in the input image based on each derived correction vector. At least when the image processing apparatus is activated, the correction vector derivation is performed before the loading of the N stage learning model data from the recording unit recording the N stage learning model data to the work area unit is completed. And the feature point correction unit start the feature point correction process, and the correction vector deriving unit determines the stage t in the work area unit when deriving the correction vector of the stage t (1 ≦ t ≦ N). If the learning model data is not loaded, the correction vector is derived by substituting the learning model data of the other stages.

  Note that any combination of the above-described constituent elements and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a recording medium recording the computer program, etc. are also effective as an aspect of the present invention. .

  According to the present invention, in object detection processing using learning model data of a plurality of stages, it is possible to advance detection start timing while suppressing a decrease in accuracy.

It is a block diagram which shows the structure of the imaging system which concerns on embodiment of this invention. FIGS. 2A to 2C are diagrams for explaining an algorithm for generating learning model data for N stages used in the feature point correction process by the feature point detection unit. FIG. 2 is a block diagram illustrating in detail a configuration of an object detection unit and a feature point detection unit in FIG. 1. It is a figure which shows the relationship between each detection stage and learning model data to be used based on Example 1. FIG. It is a figure which shows the relationship between each detection stage and learning model data to be used based on Example 2. FIG. FIG. 10 is a diagram illustrating an example of stage selection rule determination processing according to the second embodiment. FIGS. 7A and 7B are flowcharts illustrating an operation example of the image processing apparatus according to the comparative example. FIGS. 8A to 8C are diagrams showing specific examples of feature points detected by the operation examples of FIGS. 7A and 7B. FIGS. 9A and 9B are flowcharts illustrating an operation example of the image processing apparatus according to the second embodiment.

  FIG. 1 is a block diagram showing a configuration of an imaging system 1 according to an embodiment of the present invention. Hereinafter, in this specification, a camera system for photographing a driver's face in a vehicle is assumed as the imaging system 1. The imaging system 1 includes an imaging unit 20 and an image processing device 10. The imaging unit 20 is installed in the upper front part of the driver's seat in the vehicle so that the driver's face is within the angle of view. The substrate on which the imaging unit 20 is mounted and the substrate on which the image processing apparatus 10 is mounted may be integrally stored in one housing, or may be stored separately in separate housings. In the latter case, the imaging unit 20 and the image processing apparatus 10 are connected by wire or wireless.

  The imaging unit 20 includes a solid-state imaging device and a signal processing circuit. For example, a CMOS image sensor or a CCD image sensor can be used for the solid-state imaging device. The solid-state imaging device converts incident light into an electrical image signal and outputs it to a signal processing circuit. The signal processing circuit performs signal processing such as A / D conversion and noise removal on the image signal input from the solid-state imaging device, and outputs the signal to the image processing apparatus 10. Note that the signal processing circuit may execute predetermined preprocessing such as distortion correction and frame / pixel thinning.

  The image processing apparatus 10 includes a processing unit 11, a work area unit 15, and a nonvolatile recording unit 16. The work area unit 15 includes a storage area composed of a volatile memory such as DRAM or SDRAM. The non-volatile recording unit 16 includes a large-capacity non-volatile memory including a NAND / NOR flash memory or the like. The nonvolatile recording unit 16 may be configured to include a detachable recording medium, or may be configured to be mounted with a semiconductor memory card, an optical disk, a removable hard disk, and the like. The nonvolatile recording unit 16 stores learning model data for N (N is an integer of 2 or more) stages used in the feature point correction process by the feature point detection unit 13.

  The processing unit 11 includes an object detection unit 12, a feature point detection unit 13, and an application processing unit 14. The configuration of the processing unit 11 can be realized by cooperation of hardware resources and software resources, or only by hardware resources. As a hardware resource, a CPU, GPU, DSP, FPGA, or other LSI can be used. Programs such as an operating system, firmware, and applications can be used as software resources.

  The object detection unit 12 detects an approximate position and size of a predetermined object from the input image using a predetermined object identifier. Hereinafter, in this specification, a driver's face is assumed as a predetermined object. The feature point detection unit 13 sequentially corrects the initial feature point coordinates set in the input image including the object to be detected at a plurality of stages, and detects the feature point estimated as the true feature point of the object. . The application processing unit 14 executes predetermined application processing based on the feature points of the object detected by the feature point detection unit 13. In the present embodiment, for example, application processing such as driver personal authentication, face orientation determination, side-view determination, and the like is executed.

  FIGS. 2A to 2C are diagrams for explaining an algorithm for generating learning model data for N stages used in the feature point correction process by the feature point detection unit 13. In the example shown in FIGS. 2A to 2C, a total of nine points are set as the facial feature points between the left and right eyebrows, the left and right eyes, the right and left eye corners, the nostrils, and both ends of the mouth. . In stage 1, an average facial feature point (dark point) is set as an initial feature point in the input image (see FIG. 2A).

  The learning algorithm calculates the feature amount around the initial feature point in stage 1 for each feature point, and classifies the pattern around each feature point. As the feature amount, a luminance gradient, a HOG (Histgrams of Oriented Gradients) feature amount, or any other feature amount can be used. Note that the extracted feature value may be further subjected to clustering by the K-means method or the like or dimensional compression by principal component analysis or the like as the feature value. The learning algorithm calculates a difference between the set initial feature point (dark point) and the correct feature point (light circle) as a correction vector. Here, the correct feature point is defined by, for example, a human eye in advance to determine the coordinates. The learning algorithm associates the pattern around the feature point with the correction vector and stores it as learning model data of stage 1. For example, a lookup table is prepared for each feature point classification pattern, and the lookup table is represented by an average value or median value of correction vectors derived from a plurality of learning images classified into the same pattern. The converted representative value is written as a correction vector. Alternatively, when the relationship between the feature quantity and the correction vector can be approximated by a mathematical expression, coefficient parameters included in the mathematical expression may be stored as learning model data.

  In addition, the learning algorithm uses the learning model data constructed in stage 1 to correct the initial feature point using the learned correction vector corresponding to the pattern around the feature point with respect to the initial feature point of stage 1. Thus, the detected feature point in stage 1 is derived and set as the initial feature point in stage 2. For example, the learning algorithm reads nine correction vectors for nine feature points from the lookup table, adds the nine correction vectors to the nine feature points, respectively, and determines the nine initial feature points of stage 2. Derived (see FIGS. 2A and 2B).

  The learning algorithm calculates the feature amount around the initial feature point of stage 2 for each feature point in stage 2 as in stage 1, and classifies the pattern around each feature point. The learning algorithm calculates a difference between the set initial feature point (dark point) and the correct feature point (light circle) as a correction vector. The learning algorithm associates the pattern around the feature point with the correction vector and stores it as learning model data of stage 2. Thereafter, the same processing is repeated up to stage N. As the stage progresses, the correction amount of the detected feature point becomes smaller, and the deviation between the detected feature point and the correct feature point becomes smaller accordingly (see FIG. 2C). By performing this learning on a large number of face images, learning model data for N stages is generated.

  FIG. 3 is a block diagram showing in detail the configuration of the object detection unit 12 and the feature point detection unit 13 of FIG. The object detection unit 12 includes an object region detection unit 121 and an initial feature point output unit 122. The feature point detection unit 13 includes a load order determination unit 131, a learning model data load unit 132, a use stage determination unit 133, a correction vector derivation unit 134, a feature point correction unit 135, a reliability determination unit 136, and a feature point output unit 137. Including.

  The object area detection unit 121 performs a raster scan of the object search window in the input image to search for an area in which the object is shown. When one raster scan is completed, the size of the search window is changed and the raster scan is executed again. For this search, for example, a discriminator having a Cascade structure using Haar-Like features can be used. In the detection process by the object area detection unit 121, the approximate position and size of the object can be detected regardless of the size of the object and the degree of blur of the image. Based on the object detection window detected by the object region detection unit 121, the initial feature point output unit 122 outputs the approximate feature point coordinates in the detection window to the correction vector deriving unit 134 as initial coordinates.

  The load order determination unit 131 determines the load order of the learning model data for N stages stored in the nonvolatile recording unit 16 to the work area unit 15. The learning model data loading unit 132 sequentially loads the learning model data from the nonvolatile recording unit 16 to the work area unit 15 in units of stages based on the loading order determined by the loading order determination unit 131.

  The use stage determination unit 133 determines which stage of learning model data is used in each detection stage based on which stage of learning model data for N stages is loaded in the work area unit 15. The correction vector deriving unit 134 derives each correction vector of a plurality of feature points in the input image from the learning model data of the stage determined by the use stage determining unit 133 at each detection stage. The feature point correction unit 135 corrects a plurality of feature points in the input image using the correction vectors derived by the use stage determination unit 133 at each detection stage.

  The reliability determination unit 136 determines the reliability of feature point correction for each detection stage. When the reliability of the correction vector derived at a certain detection stage is less than the reference value, the reliability determination unit 136 determines that the reliability of the correction vector derived at the detection stage is low.

  The feature point output unit 137 outputs the corrected feature points over the N stages to the application processing unit 14 as the detection result of the feature point detection unit 13. In the process of correcting a feature point, the feature point output unit 137 cancels the correction of the feature point at the detection stage when the reliability determination unit 136 determines that the reliability is low, and immediately preceding the detection stage. Return to the state of the feature point. When a detection stage that is determined to have low reliability by the reliability determination unit 136 occurs, feature point detection for the frame may be invalidated. Further, when a detection stage whose reliability is determined to be low by the reliability determination unit 136 occurs a predetermined number of times or more, the feature point detection of the frame may be invalidated.

  In the above configuration, it takes a long time to load learning model data for N stages from the nonvolatile recording unit 16 to the work area unit 15 when the imaging system 1 is activated. As the number of stages, the number of feature points, and the number of pattern classifications around the feature points increase, the amount of learning model data increases. In an inexpensive built-in system, the bandwidth of the bus is narrow, and it may take several seconds or more to load all the learning model data from the nonvolatile recording unit 16 to the work area unit 15. For example, when the imaging system 1 is used for personal authentication at the start of driving, it takes several seconds or more until the vehicle can be actually moved after the ignition is turned on. In the following, we will introduce a mechanism to reduce the load waiting time.

  Example 1 will be described first. In the first embodiment, when learning model data for N stages is loaded into the work area unit 15 halfway, detection is started using the learning model data loaded in the work area unit 15. That is, detection is started when loading of some stages is completed, and detection is performed instead of the loaded learning model data in the detection stage where the learning model data is not loaded.

  FIG. 4 is a diagram illustrating a relationship between each detection stage and learning model data to be used according to the first embodiment. The comparative example is an example in which detection is started after the loading of learning model data for N stages to the work area unit 15 is completed. In the comparative example, each detection stage always matches the learning model data stage used in each detection stage. On the other hand, in the first embodiment, the detection is started when the loading up to the stage k is completed. In the first embodiment, in the stages k to N after the stage k, the feature points are corrected using the learning model data of the stage k.

  Next, Example 2 will be described. In the second embodiment, data closer to the actual stage can be used by changing the loading order of the learning model data, and the correction accuracy is improved as compared with the first embodiment. For example, the loading order is to sequentially load a stage group including stage 1 but not including some stages in ascending order and then loading the remaining stages. Note that the load order of the remaining stages may be set to an order in which all or a part of the remaining stages are sequentially loaded in ascending order.

  FIG. 5 is a diagram illustrating a relationship between each detection stage and learning model data to be used according to the second embodiment. The example shown in FIG. 5 shows an example in which learning model data for seven stages is used, and the loading order of the learning model data is set to 1, 2, 4, 6, 3, 5, 7 . If detection is started at the stage where only the learning model data of stage 1 is loaded, the learning model data of stage 1 is used at all detection stages, and the error increases. Therefore, it is preferable to start the detection from the stage where the learning model data of about half of the number of detection stages (7 in FIG. 5) (3 or 4 in FIG. 5) has been loaded. . If detection is started at the stage when the learning model data of stages 1, 2, 4, and 6 is completely loaded, the learning model data of the own stage or the immediately preceding stage can be used at each detection stage, and errors are minimized. Can be suppressed.

  As described above, the learning model data is basically loaded in the order of “1, 2, 4, 6, 3, 5, 7” or “1, 3, 5, 7, 2, 4” which skips one and loops twice. , 6 "is preferred. Hereinafter, an optimal learning model data loading order and a method for deriving an optimal stage selection rule will be described.

  A stage selection rule that minimizes the average error between the corrected feature point and the correct feature point is determined using a plurality of test images. First, a plurality of candidates for stage selection rules are prepared. In stage 7, which is the final stage, the error between the corrected feature point and the correct feature point by the correction vector of the selected learning model data is calculated for each feature point, and the average of these errors is calculated. At this time, the number of pixels normalized by the distance between the right eye and the left eye is used as a value indicating the error between the corrected feature point and the correct feature point, instead of the raw pixel number. Thereby, it is possible to prevent the average error value from being affected by the difference in the size of the face in the test image. In addition, you may perform arbitrary normalizations other than the above.

  Next, the average error in each test image is averaged over all test images, and the average error for the entire test image is calculated. Finally, a stage selection rule that minimizes the average error is adopted. Instead of the average error of all test images, for example, the average error of the worst 20% test images may be used. In this case, the evaluation of stage selection candidates that have undergone significant corrections is reduced, and it becomes easier to employ stage selection candidates that have been stably corrected for any image. Note that the error representative value may be derived by a method other than the above.

  FIG. 6 is a diagram illustrating an example of a stage selection rule determination process according to the second embodiment. The example shown in FIG. 6 is an example for determining an optimum learning model data selection rule to be used in each detection stage at the stage where learning model data of four stages is loaded in the work area unit 15. is there. FIG. 6 shows that the test is performed on four stage selection rule candidates, and candidate 3 shows the smallest average error. In order to realize the stage 3 selection rule, the learning model data of stages 1, 2, 4, and 6 is loaded at the stage where the learning model data of four stages is loaded in the work area unit 15. Necessary. Accordingly, it is possible to determine that “1, 2, 4, 6, 3, 5, 7” is appropriate as the loading order of the learning model data.

  In the first and second embodiments, a state occurs in which the detection stage does not match the stage of the learning model data to be used. A reliability determination unit 136 is provided in order to reduce the adverse effects caused by this stage mismatch. The reliability determination unit 136 determines that the reliability of the correction vector is low when the reliability of the correction vector derived at a certain detection stage is less than the reference value. For example, when the average length of the plurality of correction vectors derived at stage t (t is an integer from 1 to N) is greater than the average length of the plurality of correction vectors derived at stage (t−1) by a predetermined value or more, It is determined that the reliability of the correction vector derived at stage t is low. The length of the correction vector can be obtained from the square root of the sum of squares of the X and Y components of the correction vector.

  In addition, the reliability determination unit 136 performs a plurality of detections on a plurality of initial feature points generated by, for example, preparing a plurality of initial feature points in advance or adding a random number to one initial feature point, and When a plurality of correction vectors are derived, the distribution of the plurality of correction vectors or the distribution of the plurality of feature points after correction is the distribution of the plurality of correction vectors derived at stage (t-1) or the plurality of correction points after correction. May be determined to be low in reliability of the correction vector derived in stage t. If feature points are corrected correctly for the correct answer, the amount of correction should decrease and converge as the stage progresses, and the length and variance of the correction vector and variance between feature points from the previous stage If it increases, it indicates that the correction was not successful.

  FIGS. 7A and 7B are flowcharts illustrating an operation example of the image processing apparatus 10 according to the comparative example. FIG. 7A shows a main routine, and FIG. 7B shows a face feature point detection subroutine. FIGS. 8A to 8C are diagrams showing specific examples of feature points detected by the operation examples of FIGS. 7A and 7B.

  In FIG. 7A, the object detection unit 12 detects a face region Rf in the input image input from the imaging unit 20 (S10, see FIG. 8A). The learning model data loading unit 132 loads the learning model data for N stages recorded in the nonvolatile recording unit 16 into the work area unit 15 in the order of stages (S12). When all of the N stages of learning model data have been loaded, the facial feature point detection process starts (S13).

  In FIG. 7B, the initial feature point output unit 122 sets initial feature points in the face region Rf (S130). The use stage determination unit 133 selects learning model data of the stage t (S132). The correction vector deriving unit 134 derives the correction vector of the feature point of the detection stage t using the learning model data of the stage t (S133). The feature point correction unit 135 corrects the feature point of the detection stage t using the derived correction vector (S134). The processing from step S132 to step S134 is repeatedly performed N times. FIG. 8B shows the position of the initial feature point, and FIG. 8B shows the position of the feature point after completion of the correction.

  In FIG. 7A, the application processing unit 14 estimates the face direction based on the feature points detected in the face feature point detection process (S14), and determines the presence or absence of side effects (S15). If the application processing unit 14 determines that the driver is looking aside, the application processing unit 14 outputs an alert signal. The process shown in FIG. 7A is executed for each frame. The application processing unit 14 may determine the presence or absence of a sidewalk by detecting the movement amount and movement direction of each feature point between a plurality of frames.

  FIGS. 9A and 9B are flowcharts illustrating an operation example of the image processing apparatus 10 according to the second embodiment. In FIG. 9A, the object detection unit 12 detects a face region Rf in the input image input from the imaging unit 20 (S10). The load order determination unit 131 determines the load order of learning model data for N stages from the nonvolatile recording unit 16 to the work area unit 15 (S11). The learning model data loading unit 132 loads the learning model data into the work area unit 15 according to the determined loading order (S12a). In the second embodiment, the face feature point detection process starts before all the learning model data for N stages are loaded (S13).

  In FIG. 9B, the initial feature point output unit 122 sets initial feature points in the face region Rf (S130). The use stage determination unit 133 determines the stage m of the learning model data to be used at the detection stage t according to a preset stage selection rule (S131). The use stage determination unit 133 selects the learning model data of the determined stage m (S132a). The correction vector deriving unit 134 derives a correction vector of the feature point of the detection stage t using the learning model data of the stage m (S133). The feature point correction unit 135 corrects the feature point of the detection stage t using the derived correction vector (S134).

  The reliability determination unit 136 calculates the reliability of the correction vector of the detection stage t (S135). When the calculated reliability is less than the reference value (N in S136), the feature point output unit 137 cancels the correction of the detection stage t, and the feature point of the detection stage (t−1) is changed to the detection stage (t + 1). It outputs (S137). When the calculated reliability is equal to or higher than the reference value (Y in S136), the process in step S137 is skipped. The processing from step S131 to step S137 is repeatedly executed N times.

  In FIG. 9A, the application processing unit 14 estimates the face direction based on the feature points detected in the face feature point detection process (S14), and determines the presence or absence of side effects (S15).

  As described above, according to the present embodiment, when learning model data is loaded from the nonvolatile recording unit 16 to the work area unit 15, the learning model data for N stages are sequentially loaded in units in a predetermined order. To do. Detection starts when loading of some stages is completed, and when detecting an unloaded stage, the learning model data of the loaded stage closest to the current detection stage is substituted according to a predetermined selection rule And use it. By determining the load order so as to minimize the degradation in accuracy at the time of substitution, it is possible to perform detection while suppressing the degradation in accuracy while shortening the time to start detection.

  If the reliability is calculated at each detection stage and it is determined that the reliability is low, the feature point of the previous stage is output as the detection result. As a result, detection with reduced accuracy can be performed.

  The present invention has been described based on the embodiments. The embodiments are exemplifications, and it is understood by those skilled in the art that various modifications can be made to the respective components or combinations of the respective treatment processes, and such modifications are also within the scope of the present invention. is there.

  In the above-described embodiment, the example in which the object to be detected is single has been described. In this regard, when there are a plurality of objects to be detected, learning model data for N stages is stored in the nonvolatile recording unit 16 for each object. For example, learning model data A for normal face detection, learning model data B for face detection using a mask, and learning model data C for face detection using sunglasses are stored in the nonvolatile recording unit 16.

  The object detection unit 12 outputs an object ID indicating the object detected in the input image to the feature point detection unit 13. The learning model data loading unit 132 of the feature point detection unit 13 loads the learning model data corresponding to the object ID from the nonvolatile recording unit 16 to the work area unit 15 by the method according to the first embodiment or the second embodiment described above. .

  Hereinafter, learning model data before switching objects (learning model data for normal faces) is referred to as learning model data A, and learning model data after switching objects (masked face learning model data) is referred to as learning model data B. When the reliability of the detection stage using the learning model data B is less than the reference value and the learning model data A remains in the work area unit 15, the reliability when the learning model data A is used is calculated. In the case of the reference value or more, the detection using the learning model data A may be returned. In this case, it is possible to prevent the learning model data A in the work area unit 15 from being overwritten, and to avoid reloading the learning model data A.

  The application processing unit 14 may determine the start timing of each of the plurality of application processes according to the number of stages that have been loaded with the learning model data to the work area unit 15. For example, in the case of an application that estimates emotions from facial expressions, it is necessary to detect feature points that are sensible, and therefore, the processing is started after the learning model data of all stages has been loaded. On the other hand, since the face direction estimation application can be detected from roughly feature points, the processing is started when the learning model data of some stages is completely loaded. As the number of stages loaded in the work area unit 15 increases, the application processing unit 14 sequentially activates applications that can be processed among a plurality of applications.

  In the above-described embodiment, the example in which the initial feature point coordinates in the feature point detection unit 13 are set from the approximate feature point coordinates of the object detected by the object detection unit 12 has been described. In this regard, in the case of an application in which the size and position of an object in an image are generally determined (for example, an ID photo), an average face feature point may be set as an initial feature point at a predetermined position.

  In the above-described embodiment, the image processing apparatus 10 incorporated in the imaging system 1 installed in the vehicle is assumed as the image processing apparatus 10. In this regard, the image processing apparatus 10 may be mounted on any device of information processing devices such as a smartphone and a PC. For example, even if a PC is a low-spec model, it may take a long time to load learning model data in the hard disk into the RAM. In such a case, the method described in the above embodiment functions effectively.

  The embodiment may be specified by the following items.

[Item 1]
Using the learning model data of N (N is an integer of 2 or more) stages, each of which contributes to the derivation of the correction vector of the feature point, the initial feature point in the input image is corrected step by step to obtain the feature point of the object. An image processing device (10) to detect,
A work area section (15) for loading the N-stage learning model data;
Correspondence stored in the work area unit (15) based on the feature amount around each initial feature point in the input image or the feature amount around each feature point modified in the previous stage A correction vector deriving unit (134) for deriving a correction vector of each feature point with reference to the learning model data of the stage to perform;
A feature point correction unit (135) that corrects each feature point in the input image based on each derived correction vector;
At least when the image processing apparatus (10) is activated, the N stage learning model data is loaded from the recording unit (16) recording the N stage learning model data to the work area unit (15). Before completion, the correction vector deriving unit (134) and the feature point correcting unit (135) start the feature point correcting process,
When the correction vector deriving unit (134) derives the correction vector of the stage t (1 ≦ t ≦ N), if the learning model data of the stage t is not loaded in the work area unit (15), the other An image processing apparatus (10), wherein a correction vector is derived by substituting the learning model data of the stage.
According to this, the start timing of the feature point correction processing can be advanced.
[Item 2]
When the correction vector deriving unit (134) derives the correction vector of the stage t (1 ≦ t ≦ N) and the learning model data of the stage t is not loaded in the work area unit (15), the correction vector deriving unit (134) Item 10. The image processing apparatus (10) according to item 1, wherein a correction vector is derived by substituting the learning model data at the nearest stage among the learning model data already loaded in the work area (15). .
According to this, the start timing of the correction process can be advanced while suppressing a decrease in the correction accuracy of the feature points.
[Item 3]
The learning order data of the N stages from the recording unit (16) to the work area unit (15) is loaded after the stage group including stage 1 is skipped in order of ascending order and then skipped. 3. The image processing apparatus (10) according to item 1 or 2, wherein the stages are sequentially loaded.
According to this, in the stage where learning model data needs to be substituted, the learning model data of the near stage can be substituted, and the reduction in the correction accuracy of the feature points can be suppressed.
[Item 4]
A load order determination unit (131) for determining a load order of the N-stage learning model data to the work area unit (15);
A learning model data loading unit (132) that sequentially loads the learning model data of the N stages in units of stages based on the loading order determined by the loading order determination unit (131);
The load order determination unit (131) selects a load order candidate that has the smallest average error from the correct feature point in the correction process of feature points for test data from among a plurality of load order candidates as a load order to use. The image processing apparatus (10) according to any one of items 1 to 3, wherein the image processing apparatus (10) is determined.
According to this, it is possible to further suppress a decrease in the correction accuracy of the feature points.
[Item 5]
A reliability determination unit (136) for determining the reliability of the correction vector of the feature point derived at each stage;
The reliability determination unit (136) invalidates the correction of the feature point of the stage t when the reliability of the correction vector derived at the stage t is less than a reference value. The image processing device (10) according to any one of the above.
According to this, it is possible to suppress a reduction in correction accuracy by invalidating correction of feature points with low reliability.
[Item 6]
The reliability determination unit (136) determines the average value of the lengths of the plurality of correction vectors derived at the stage t from the average value of the lengths of the plurality of correction vectors derived at the stage (t-1). The image processing apparatus (10) according to item 5, wherein the correction of a plurality of feature points of the stage t is invalidated when the value is larger than the value.
According to this, the correction accuracy can be accurately evaluated by comparing the correction of the immediately preceding stage with the correction of the current stage.
[Item 7]
In the recording unit (16), the learning model data of the N stage is recorded for each object to be detected,
When the detection target is switched from the first object to the second object, the loading of the learning model data of the N stage of the second object from the recording unit (16) to the work area unit (15) is completed. The correction vector deriving unit (134) and the feature point correcting unit (135) start correction processing of the feature points of the second object,
When the correction vector deriving unit (134) derives the correction vector of the stage t (1 ≦ t ≦ N) in the correction process of the feature point of the second object, the correction vector deriving unit (134) outputs the stage t to the work area unit (15). 5. The image processing device (10) according to any one of items 1 to 4, wherein when the learning model data is not loaded, a correction vector is derived by substituting the learning model data of another stage.
According to this, at the time of switching from the first object to the second object, it is possible to advance the start timing of the correction process of the feature point of the second object.
[Item 8]
A reliability determination unit (136) for determining the reliability of the correction vector of the feature point derived at each stage;
The reliability determination unit (136) is a case where the reliability of the correction vector using the learning model data of the second object is less than a reference value, and remains in the work area unit (15) 8. The image processing apparatus (10) according to item 7, wherein when the reliability of the correction vector using the learning model data of the first object satisfies a reference value, the process is switched to the first object detection process.
According to this, it is possible to effectively utilize the learning model data of the first object remaining in the work area part (15).
[Item 9]
An object area detection unit (121) for searching for a specific object in the input image and setting a detection frame of the detected specific object;
An initial feature point setting unit (122) for setting a feature point of an object in the detection frame as the initial feature point;
The image processing apparatus (10) according to any one of items 1 to 8, further comprising:
According to this, the initial feature point can be set at a position relatively close to the correct feature point.
[Item 10]
An application processing unit (14) for executing a plurality of application processes based on the feature points after the stepwise correction of the initial feature points is completed;
The application processing unit (14) determines the start timing of each of the plurality of application processes according to the number of stages in which the learning model data has been loaded into the work area unit (15). The image processing apparatus (10) according to any one of items 1 to 9.
According to this, the start timing of application processing can be optimized.
[Item 11]
The input image is an image input from an imaging unit (20) for photographing a driver's face installed in a vehicle,
The image processing device (10) according to any one of items 1 to 10, wherein the object to be detected is a driver's face.
According to this, in the driver's face detection system, the start timing of the feature point correction process can be advanced.
[Item 12]
Using the learning model data of N (N is an integer of 2 or more) stages, each of which contributes to the derivation of the correction vector of the feature point, the initial feature point in the input image is corrected step by step to obtain the feature point of the object. An image processing method to detect,
Loading the N stage learning model data into the work area section (15);
Correspondence stored in the work area unit (15) based on the feature amount around each initial feature point in the input image or the feature amount around each feature point modified in the previous stage Deriving a correction vector of each feature point with reference to learning model data of the stage to be performed;
Correcting each feature point in the input image based on each derived correction vector,
At least at the time of activation, the correction vector is stored before the learning model data of the N stage is completely loaded from the recording unit (16) in which the learning model data of the N stage is recorded to the work area unit (15). The process of deriving and the process of correcting the feature point starts the process of correcting the feature point,
In the step of deriving the correction vector, if the learning model data of stage t is not loaded in the work area part (15) when deriving the correction vector of stage t (1 ≦ t ≦ N), An image processing method, wherein a correction vector is derived by substituting learning model data of a stage.
According to this, the start timing of the feature point correction processing can be advanced.
[Item 13]
Using the learning model data of N (N is an integer of 2 or more) stages, each of which contributes to the derivation of the correction vector of the feature point, the initial feature point in the input image is corrected step by step to obtain the feature point of the object. An image processing program to detect,
A process of loading the N-stage learning model data into the work area unit (15);
Correspondence stored in the work area unit (15) based on the feature amount around each initial feature point in the input image or the feature amount around each feature point modified in the previous stage A process for deriving a correction vector of each feature point with reference to the learning model data of the stage to be performed,
Processing for correcting each feature point in the input image based on each derived correction vector,
At least at the time of activation, the correction vector is stored before the learning model data of the N stage is completely loaded from the recording unit (16) in which the learning model data of the N stage is recorded to the work area unit (15). The process of deriving and the process of correcting the feature point starts the process of correcting the feature point,
In the process of deriving the correction vector, when deriving the correction vector of stage t (1 ≦ t ≦ N), if learning model data of stage t is not loaded in the work area part (15), An image processing program that derives a correction vector by substituting learning model data of a stage.
According to this, the start timing of the feature point correction processing can be advanced.

  DESCRIPTION OF SYMBOLS 1 Imaging system, 10 Image processing apparatus, 11 Processing part, 12 Object detection part, 13 Feature point detection part, 14 Application processing part, 15 Work area part, 16 Non-volatile recording part, 20 Imaging part, 121 Object area detection part, 122 initial feature point output unit, 131 load order determination unit, 132 learning model data load unit, 133 use stage determination unit, 134 correction vector derivation unit, 135 feature point correction unit, 136 reliability determination unit, 137 feature point output unit

Claims (13)

Using the learning model data of N (N is an integer of 2 or more) stages, each of which contributes to the derivation of the correction vector of the feature point, the initial feature point in the input image is corrected step by step to obtain the feature point of the object. An image processing device to detect,
A work area section for loading the N-stage learning model data;
Based on the feature amount around each initial feature point in the input image or the feature amount around each feature point modified in the previous stage, the corresponding stage held in the work area unit A correction vector deriving unit for deriving a correction vector of each feature point with reference to the learning model data;
A feature point correction unit that corrects each feature point in the input image based on each derived correction vector;
At least when the image processing apparatus is activated, the correction vector derivation is performed before the loading of the N stage learning model data from the recording unit recording the N stage learning model data to the work area unit is completed. And the feature point correction unit start a feature point correction process,
When the correction vector deriving unit derives the correction vector of the stage t (1 ≦ t ≦ N), if the learning model data of the stage t is not loaded in the work area unit, the learning model data of another stage An image processing apparatus that derives a correction vector by substituting.   When the correction vector deriving unit derives the correction vector of the stage t (1 ≦ t ≦ N) and the learning model data of the stage t is not loaded in the work area unit, the correction vector deriving unit is already loaded in the work area unit. The image processing apparatus according to claim 1, wherein the correction vector is derived by substituting the learning model data of the closest stage among the learning model data being processed.   The learning model data of the N stage from the recording unit to the work area unit is loaded in order of ascending order of stages including stage 1 and skipping some stages, and then sequentially loading the skipped stages. The image processing apparatus according to claim 1, wherein the order is an order. A load order determination unit for determining a load order of the learning model data of the N stages to the work area unit;
A learning model data loading unit that sequentially loads the learning model data of the N stages in units of stages based on the loading order determined by the loading order determination unit;
The load order determination unit determines, from among a plurality of load order candidates, a load order candidate that minimizes an average error from the correct feature point in the correction process of the feature points for the test data as a load order to be used. The image processing apparatus according to claim 1, wherein: A reliability determination unit that determines the reliability of the correction vector of the feature point derived at each stage;
The reliability determination unit invalidates the correction of the feature point of the stage t when the reliability of the correction vector derived at the stage t is less than a reference value. An image processing apparatus according to 1.   In the reliability determination unit, the average value of the lengths of the plurality of correction vectors derived in the stage t is larger than the average value of the lengths of the plurality of correction vectors derived in the stage (t-1) by a predetermined value or more. 6. The image processing apparatus according to claim 5, wherein correction of a plurality of feature points of stage t is invalidated. In the recording unit, the learning model data of the N stage is recorded for each object to be detected,
When the detection target is switched from the first object to the second object, before the loading of the learning model data of the N stage of the second object from the recording unit to the work area unit is completed, the correction vector deriving unit and the The feature point correction unit starts correction processing of the feature points of the second object,
When the correction vector deriving unit derives a correction vector of the stage t (1 ≦ t ≦ N) in the correction process of the feature point of the second object, the learning model data of the stage t is loaded into the work area unit 5. If not, the image processing apparatus according to claim 1, wherein the correction vector is derived by substituting the learning model data of another stage. A reliability determination unit that determines the reliability of the correction vector of the feature point derived at each stage;
The reliability determination unit is a case where the reliability of the correction vector using the learning model data of the second object is less than a reference value, and the learning model of the first object remaining in the work area unit The image processing apparatus according to claim 7, wherein when the reliability of the correction vector using data satisfies a reference value, the processing is switched to the first object detection process. Searching for a specific object in the input image, an object area detection unit for setting a detection frame of the detected specific object;
An initial feature point setting unit for setting a feature point of the object in the detection frame as the initial feature point;
The image processing apparatus according to claim 1, further comprising: Based on the feature point after completion of the stepwise correction of the initial feature point, further comprising an application processing unit that executes a plurality of application processes,
The said application process part determines the start timing of each of these application processes according to the number of stages by which the said learning model data was loaded to the said work area part. The image processing apparatus according to any one of the above. The input image is an image input from an imaging unit for photographing a driver's face installed in a vehicle,
The image processing apparatus according to claim 1, wherein the object to be detected is a driver's face. Using the learning model data of N (N is an integer of 2 or more) stages, each of which contributes to the derivation of the correction vector of the feature point, the initial feature point in the input image is corrected step by step to obtain the feature point of the object. An image processing method to detect,
Loading the N stage learning model data into a work area;
Based on the feature amount around each initial feature point in the input image or the feature amount around each feature point modified in the previous stage, the corresponding stage held in the work area unit Deriving a correction vector of each feature point with reference to the learning model data;
Correcting each feature point in the input image based on each derived correction vector,
The process of deriving the correction vector before the completion of the loading of the learning model data of the N stage from the recording unit recording the learning model data of the N stage to the work area at least at the time of start-up The process of correcting the feature point starts the correction process of the feature point,
In the step of deriving the correction vector, when the learning model data of the stage t is not loaded in the work area part when deriving the correction vector of the stage t (1 ≦ t ≦ N), learning of another stage is performed. An image processing method characterized by deriving a correction vector by substituting model data. Using the learning model data of N (N is an integer of 2 or more) stages, each of which contributes to the derivation of the correction vector of the feature point, the initial feature point in the input image is corrected step by step to obtain the feature point of the object. An image processing program to detect,
A process of loading the N-stage learning model data into a work area part;
Based on the feature amount around each initial feature point in the input image or the feature amount around each feature point modified in the previous stage, the corresponding stage held in the work area unit A process for deriving a correction vector of each feature point with reference to the learning model data;
Processing for correcting each feature point in the input image based on each derived correction vector,
The process of deriving the correction vector before the completion of the loading of the learning model data of the N stage from the recording unit recording the learning model data of the N stage to the work area at least at the time of start-up The process of correcting the feature point starts the correction process of the feature point,
In the process of deriving the correction vector, when the learning model data of stage t is not loaded in the work area when deriving the correction vector of stage t (1 ≦ t ≦ N), learning of another stage is performed. An image processing program that derives a correction vector by substituting model data.

Images