JP2018128930A - Image processing device and control method thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image division technique that enables more efficient processing to be performed by a plurality of processing units.SOLUTION: An image processing device performing processing using a plurality of processing units includes: division means for regionally dividing an input image into a plurality of subregions; determination means for, on the basis of the number of the plurality of processing units, a first division number which is the number of the plurality of subregions, and an influence in the case of adjusting the number of the plurality of subregions from the first division number to a second division number, determining the second division number; adjustment means for adjusting a part of the subregions of the first division number such that the input image is regionally divided by the second division number; and control means for exerting control such that predetermined processing on subregions of the second division number is executed by the plurality of processing units.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a technique for dividing an image into a plurality of partial regions.

  In recent years, technological development for recognizing a person or an object in a photograph or a scene of the photograph has progressed. Among them, there is a method of dividing an image into partial regions, extracting feature amounts of each region, and performing recognition using the feature amounts. In such a method, after dividing into partial areas, processing is often performed for each partial area. In this case, processing can be performed in parallel with one partial area as a processing unit. When a large number of processing cores are mounted in a processing system that performs these processes, the processing can be performed more efficiently if the processing for each partial region is assigned to each processing core.

  In order to perform parallel processing more efficiently, a technique has been proposed for adjusting a processing unit by dividing a certain process into a plurality of processes. Patent Document 1 discloses a technique for detecting the number of CPUs that execute the system and dividing the shape object according to the number of CPUs when a three-dimensional shape expressed in a certain format is read into the shape processing system. ing.

Japanese Patent No. 3847020

  In general, when the number of divisions changes, the feature quantity of the partial region may change, resulting in an influence on the image recognition result. For this reason, the method of determining the number of divisions based only on the number of CPUs as in the technique disclosed in Patent Document 1 may not be desirable in image recognition processing.

  The present invention has been made in view of such a problem, and an object thereof is to provide a technique for performing image division that enables more efficient processing.

  In order to solve the above-described problems, an image processing apparatus according to the present invention has the following configuration. That is, an image processing apparatus that performs processing using a plurality of processing units includes a dividing unit that divides an input image into a plurality of partial regions, a number of the plurality of processing units, and a number of the plurality of partial regions. Determining means for determining the second division number on the basis of the division number of 1 and the effect of adjusting the number of the plurality of partial areas from the first division number to the second division number; Adjustment means for adjusting a part of the partial region of the first division number so that the input image is divided into regions by the second division number, and the portion of the second division number by the plurality of processing units And control means for controlling to execute a predetermined process for the area.

  According to the present invention, it is possible to provide a technique for performing image division that enables more efficient processing.

1 is a block diagram of an information processing apparatus according to a first embodiment. It is a flowchart which shows the recognition process in 1st Embodiment. It is a figure which shows a mode that an input image is processed. It is a figure which shows the list of the partial area | region pairs of an integration plan. It is a flowchart which shows the recognition process in 2nd Embodiment. It is a figure explaining division and integration of a partial field. It is a flowchart which shows the recognition process in 3rd Embodiment. It is a figure which shows the table which matched dictionary number and the number of classes.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. The following embodiments are merely examples, and are not intended to limit the scope of the present invention.

(First embodiment)
As a first embodiment of an image processing apparatus according to the present invention, a smartphone that performs image recognition processing using a GPU having a plurality of processing units (processing cores (PE)) will be described below as an example. Note that GPU is an abbreviation for Graphics Processing Unit.

  Specifically, processing of an image processing method in which a user performs recognition processing in real time on an image captured by a camera using a smartphone will be described. More specifically, a mode will be described in which a photograph is divided into partial regions, feature amounts of each region are extracted, and image recognition is performed using the feature amounts. In particular, a mode in which the number of divisions is adjusted based on the number of processing systems that execute recognition processing and the recognition accuracy of recognition processing as an influence on the entire processing will be described.

<Prerequisites>
First, preconditions in the first embodiment will be described.

  The recognition result represents the category ratio of the area. Specifically, it can be divided into four categories of organisms, nature, things, and others, and the recognition result is, for example, “organisms: nature: things: other = 0.8: 0.1: 0.1: 0”. It is assumed that it is represented by the likelihood of each category. When presenting the recognition result to the user, the category with the highest likelihood is selected and displayed for each category.

  Generally used PA (Pixel Accuracy) is used for the recognition accuracy of the recognition process. Note that PA indicates a rate at which the recognition result matches the correct answer data when the obtained recognition result is compared with the correct answer data in pixel units.

  The recognition process is performed by a GPU (Graphics Processing Unit) mounted on the user's smartphone. Here, the number of processing cores (PE) mounted on the GPU is “16”. The input of the recognition process is an image and a dictionary described later, and the output is a category likelihood map of each region in the image.

  The number of cores on the GPU is used as the number of executable processes of the processing system that executes the recognition process. In this embodiment, the recognition accuracy of the recognition process is used as an influence on the entire process. That is, from the viewpoint of processing efficiency, the number of divisions is selected to be close to an integer multiple of the number of cores of the GPU, and from the viewpoint of recognition accuracy, the degree of deterioration in recognition accuracy is selected. A specific method will be described in the description of the flowchart.

  Assume that a commonly used k-means algorithm is used to divide the input image into partial regions. Since the clustering by the k-means algorithm is known, the description thereof is omitted. In the present embodiment, using k-means, first, RGB color information is extracted as a feature amount from pixels in the input image, and each pixel is clustered into 50 clusters on the feature space. Thereafter, each cluster is mapped on the two-dimensional image. Then, for example, two pixels belonging to the same cluster on the feature space may be separated on the two-dimensional image. Therefore, a labeling process is performed on the clusters mapped on the two-dimensional image, and the clusters that are spatially separated are divided to obtain a final cluster, that is, a partial region. From this, the number of partial regions finally obtained is expected to be 50 or more.

  Assume that a commonly used SVM (Support Vector Machine) is used for learning / recognition processing used when recognizing a category of a partial region. Since this learning / identification method is publicly known, description thereof is omitted. In addition, since the parameters used for recognition (hereinafter referred to as a dictionary) are considered to depend on the number of divisions of the input image at the time of learning, generally the closer the number of divisions at the time of learning, the better the recognition accuracy is expected. . In this embodiment, the number of divisions during learning is “50”.

  It is assumed that a histogram of color distribution of pixels in the partial area is used as a feature amount input to the SVM for recognizing the partial area. Specifically, the luminance of each of RGB (0 to 255) of the pixels in the region is equally divided into four stages (0 to 63, 64 to 127, 128 to 191 and 192 to 255) to create a histogram. That is, each of RGB has 4 dimensions and 12 dimensions in total. The histogram is normalized so that the value is 0-1.

<Device configuration>
FIG. 1 is a block diagram of an information processing apparatus according to the first embodiment. The memory 101 stores data necessary for each process. For example, a program and a dictionary that realize recognition processing are stored. The camera 102 is a camera that captures a two-dimensional image, and outputs the captured image to the memory 101.

  The input device 103 receives various inputs from the user. For example, an input of a command for taking a picture with the camera 102 is received from the user. The display 104 displays the input image and the recognition processing result.

  The CPU 105 is a functional unit that executes a program stored in the memory 101 and realizes various processes. It also provides a function to control each device. For example, a partial area dividing process for dividing a captured image is performed, or an image stored in the memory 101 is transferred to the display 104.

  The GPU 106 executes processes instructed by the CPU 105 in parallel. For example, partial region feature generation processing or partial region recognition processing is performed. Further, the GPU 106 includes a plurality of PEs (processing cores) 107, and processing is distributed to each processing core, and processing is executed. The code (kernel) of each process is loaded from the memory 101 to the GPU 106 and executed before execution.

  Note that the configuration of the image processing apparatus in the present embodiment is not limited to the above-described configuration. For example, although the CPU 105 has a function of controlling each functional unit, a configuration using a dedicated processing processor for controlling each device may be used. In addition, other parallel processing accelerators such as a DSP may be mounted instead of the GPU.

<Operation of the device>
FIG. 2 is a flowchart showing the recognition processing in the first embodiment. The following processing is realized by the CPU 105 executing a predetermined control program.

  In step S <b> 201, the CPU 105 acquires the number of processing cores mounted on the GPU and records it in the memory 101. The number of processing cores can be obtained using, for example, clGetDeviceInfo in the case of the OpenCL API. Here, a value of processing core number = 16 is stored in the memory 101.

  In step S202, the CPU 105 divides the input image into partial regions using the k-means algorithm presented in the precondition. Here, the input image is an image photographed in accordance with a photography command input by the input device 103.

  That is, clustering is performed by setting the value of “50” presented as the precondition to the value of k, which is the number of k-means classes. Then, as shown in the preconditions, after mapping onto a two-dimensional image, a labeling process is performed. That is, a number is assigned to each partial area.

  FIG. 3 is a diagram exemplarily showing how an input image is processed. FIG. 3A shows an example of the input image before division, and FIG. 3B shows an example of the input image after division. It is assumed that the number of the partial area is held by each pixel of the input image. Further, it is assumed that the number of partial regions finally obtained is 54. That is, any number from 1 to 54 is assigned to each partial region shown in FIG.

  In step S <b> 203, the CPU 105 derives the 12-dimensional feature amount presented as the precondition for each partial region. That is, the RGB value of each pixel in the partial area is added to each 12-dimensional bin for each partial area, and finally normalization processing is performed.

  In step S204, the CPU 105 determines a new division number (x) using the number of processing cores (p) and the division number (y) divided in S202, stored in the memory 101. . Here, a new division number is determined according to the following conditions.

x = max (n × p) (n: natural number) and x ≦ y
Here, max (X) means the maximum value among the possible values of X.

  By determining in this way, the processing is close to the number of divisions used at the time of creating the dictionary (here, “50”) and more efficiently using a plurality of processing cores than when the number of divisions is not adjusted. Is possible. That is, the number of divisions that can improve the processing speed while maintaining the same level of accuracy can be achieved. Here, since p = 16 and y = 54, x = 48.

  In step S205, the CPU 105 determines which partial area is to be integrated in the partial area integration process, which is an adjustment process necessary to obtain the new division number determined in step S204 from the current division number. That is, in order to change the current number of divisions “54” to the new number of divisions “48”, it is necessary to reduce six partial areas (partial partial areas), and processing for selecting the six is performed. Here, it is assumed that six pairs of adjacent partial regions having similar feature quantities are selected from the closest order.

  Specifically, first, a partial area adjacent to the partial area is listed. As a list-up method, for example, the partial area is shifted by one pixel vertically and horizontally, the partial area number is checked for each pixel, and if a number other than the partial area number appears, it is adjacent to that number. It can be said that After that, on the 12-dimensional feature space extracted in S203, the Euclidean distance between two adjacent partial areas on the image is calculated, and six of the Euclidean distances that are short are selected, and a pair of the partial area numbers is selected. Is stored in the memory 101. FIG. 4 is a diagram showing a list of six partial region pairs to be integrated determined in this way.

  In step S206, the CPU 105 integrates the partial area pairs determined in S205. Here, the number of the partial area registered in each pixel in the two partial areas is updated with the younger number. For example, from FIG. 4, when the partial area numbers “4” and “8” are a pair, the number of the partial area of each pixel included in the partial area “8” is rewritten to “4”. FIG. 3C shows an example after integration. At this stage, the input image is divided by the new division number “48”.

  In step S207, the CPU 105 causes the 16 processing cores to execute the recognition processing of 48 partial areas. That is, each processing core outputs the category estimated from the feature using the SVM presented in the preconditions and the parameters of the SVM dictionary in the memory 101. Here, a plurality of partial regions obtained by the division are distributed equally to each processing core. Therefore, each processing core performs recognition processing of 3 (= 48/16) partial areas.

  The category likelihood map as an output result is output through the display 104, for example, with the category having the highest value as the category of the partial area. FIG. 3D shows the output recognition result.

  In step S208, the CPU 105 determines whether evaluation for all input images has been completed. That is, if it is determined that there are still unrecognized input images, the process proceeds to S202. If it is determined that the recognition has been completed for all input images, the process is terminated.

  Although the number of cores on the GPU is used as the number of executable processes of the processing system that executes the recognition process, for example, the number of threads or the maximum number of executable processes at that time may be used. In addition to the GPU, it is of course possible to select a processing system capable of parallel processing such as a general multi-core CPU or SIMD as the processing system.

  In the above description, the number of divisions is an integral multiple of the number of processing cores of the GPU. However, for example, a certain threshold may be provided, and a slight deviation may occur as long as it falls within a certain range. It is good also as. For example, when the threshold value is 80% of the number of processing cores (p) and the division number before the division number adjustment is y, the division number is adjusted when ymodp <0.8p, otherwise It is also effective not to make adjustments. Here, YmodX is a remainder when Y is divided by X. In this way, it is possible to select the number of divisions while balancing recognition accuracy and processing speed.

  In addition, in the above description, the recognition accuracy is improved as the number of divisions close to that at the time of creating the dictionary. However, even in an algorithm that does not use a dictionary, an appropriate number of divisions is designated in advance. It is also possible to select a division number close to the division number.

  Furthermore, the number of divisions can be determined in consideration of the request processing time. For example, if recognition processing is performed with a predetermined number of divisions, and it is found that the number of divisions is too large to satisfy the requested processing time, the maximum number of divisions that can be processed within the range that satisfies the requested processing time is set. be able to. For example, if the required processing time is not satisfied when trying to perform the pixel-by-pixel recognition processing, a process of integrating and reducing pixels until the number of divisions satisfying the required processing time can be considered.

  In the above description, the k-means algorithm is used as a method of dividing an image into partial regions. However, other clustering methods can be used. For example, the Mean-shift algorithm can be used. In addition, by using a division algorithm that can specify the number of image divisions in advance, it is possible to divide the image into the optimum number of divisions for the processing system at the stage of input image division, and from the division number setting step, adjacent partial region integration You can also skip to steps.

  In addition, in the above description, the partial region feature extraction processing is performed before the determination processing of the partial region candidates to be integrated, but may be performed after the determination processing. In this case, a feature different from the feature extracted by the feature extraction process may be used to determine the partial region candidate. For example, even when a color histogram is used, it is conceivable to use 6-dimensional features in which the number of bins is reduced by half in order to reduce the amount of calculation. This is particularly effective when the feature extraction process of the partial region is heavy and takes time.

  Furthermore, in the above description, “the number of divisions” has been considered. However, for example, when using a division algorithm that divides an image with substantially the same size, “area of a partial region” may be considered in association with the number of divisions. it can. For example, when the area of the partial region is 10 × 10 for a 100 × 100 image, the number of divisions corresponds to 100.

  As described above, according to the first embodiment, the number of divisions is determined based on the number of executable processes of the processing system that executes the recognition process and the recognition accuracy of the recognition process as an influence on the entire process. With this configuration, it is possible to divide an image into a number of divisions that can efficiently use a plurality of processing cores while reducing the possibility of a reduction in recognition accuracy.

  In the above description, the image recognition process has been described as an example, but the present invention can also be applied to other processes using the characteristics of the partial area. For example, it can be applied to various processing such as image quality enhancement processing and main subject detection processing.

(Second Embodiment)
In the second embodiment, a mode in which the number of divisions is adjusted based on the number of executable processes of the processing system that executes the recognition process and the change in processing time that occurs when the number of divisions is adjusted will be described. In the following description, description of processes similar to those described in the first embodiment will be omitted, and only processes different from those described in the first embodiment will be described.

<Prerequisites>
First, preconditions in the second embodiment will be described. Other preconditions not described below are the same as those in the first embodiment.

  As with the first embodiment, the number of cores on the GPU is used as the number of executable processes of the processing system that executes the recognition process. In the present embodiment, as an influence on the entire process, an increase in the processing time that occurs when the number of divisions is adjusted and an increase in the processing time that occurs when the number of divisions is not adjusted are used. That is, from the viewpoint of processing efficiency, the number of divisions is selected to be an integral multiple of the number of cores of the GPU, and from the viewpoint of processing time, the entire processing time is selected.

  As an image to be input to generate a partial region, an image obtained by applying a Gaussian filter to the input image and an image obtained by applying a sharpness filter to the input image are used in addition to the original input image. That is, a partial area is generated for each of the three images. It is assumed that the conditions for dividing into partial areas are the same for all images. In addition, the feature extraction process and the recognition process of each partial area are performed under the same conditions for any image-derived partial area.

  The average processing time for adjusting the number of divisions is measured in advance, and the average processing time required to integrate one set of partial areas is 0.1 (milliseconds / 1 GHz · set). In this embodiment, the division number adjustment is performed by the CPU, and the CPU clock number is 1 GHz.

  The average processing time required for the recognition processing is measured in advance, and the average processing time required for recognizing one partial area is 1 (millisecond / 1 GHz · piece). At this time, since there are 16 GPU cores, if the number of partial areas is 16 or less, processing can be performed at a time by parallel processing. That is, as long as the number of partial areas is 16 or less, it is possible to perform processing in the same processing time. Note that the number of clocks of the GPU that performs recognition processing is 1 GHz.

<Operation of the device>
FIG. 5 is a flowchart showing recognition processing in the second embodiment. Note that steps S501 and S513 are the same as S201 and S208 in the first embodiment, and a description thereof will be omitted.

  In step S502, the CPU 105 applies a Gaussian filter and a sharpness filter to the input image, and outputs two images obtained as a result of the application.

  In step S503, the CPU 105 divides the original input image and the two images output in step S502 into partial areas in the same manner as in S202 in the first embodiment. Here, it is assumed that the number of partial regions finally obtained is “54” in the original input image, “51” in the Gaussian filter applied image, and “58” in the sharpness filter applied image. In step S504, the CPU 105 extracts features from the respective areas obtained in S503 in the same manner as in S203 in the first embodiment.

  In step S505, the CPU 105 adds up the numbers of all partial areas divided in S503. That is, since the number of partial areas currently obtained is 54, 51, and 58, respectively, the total is 163, and this value is recorded in the memory 101.

  In step S506, the CPU 105 determines candidates for the number of divisions in the same manner as in S204 in the first embodiment. Since the processing core number p = 16 and the division number y = 163, the new division number x = 160. In the first embodiment, this value is used as it is to adjust the number of divisions. However, in the second embodiment, the processing load (particularly the processing time) is compared, and the processing when the number of divisions obtained in this step is used. Only when the time is shortened, the number of divisions is adjusted.

  In step S507, the CPU 105 increases an increase in processing time that occurs when the number of divisions is adjusted (hereinafter referred to as processing time during adjustment) and an increase in processing time that occurs when the number of divisions is not adjusted (hereinafter referred to as unadjusted processing time). Is calculated (predicted).

First, regarding the adjustment processing time, it is necessary to reduce three partial areas from 163 to 160 this time, that is, to integrate three sets of partial areas. When the adjustment processing time t1 is obtained based on the preconditions,
t1 = 0.1 (milliseconds / 1 GHz · set) × 1 (GHz) × 3 (set) = 0.3 (ms)
It becomes. On the other hand, regarding the unadjusted processing time, this time, it is necessary to additionally process the processing for three on the GPU. When the unadjusted processing time t2 is obtained based on the preconditions,
t2 = 1 (milliseconds / 1 GHz / piece) × 1 (GHz) × ceil (3 (pieces) / 16) = 1 (ms)
It becomes. Note that ceil (X) represents a process (round-up process) that rounds the element of X to the nearest integer in the positive direction.

  In step S508, the CPU 105 compares the adjustment processing time with the non-adjustment processing time. If it is determined that the processing time during adjustment <the processing time during non-adjustment, it means that the total processing time becomes shorter when the number of divisions is adjusted, and the process proceeds to step S509. On the other hand, if it is determined that the adjustment processing time> the non-adjustment processing time, it means that the total processing time is shorter when the number of divisions is not adjusted, and the process proceeds to step S512. Here, t1 <t2, and the process proceeds to step S509.

  In step S509, the CPU 105 determines which partial area to be integrated in the integration process of partial areas necessary for obtaining the new division number determined in step S506 from the current division number. That is, a process of selecting three partial areas to be reduced and integrated in order to change the current division number “163” to the new division number “160” is performed. Here, three partial areas are selected in ascending order, and the numbers of the partial areas are recorded in the memory 101.

  In step S510, the CPU 105 divides the partial area selected in step S509 by the number of partial areas surrounding the periphery, and absorbs and integrates the partial areas in each partial area. In order to identify the partial region surrounding the periphery, the technique for listing adjacent partial regions described in S205 in the first embodiment can be used. If the adjacent number is known, the partial area is divided by using, for example, a k-means algorithm as the number of clusters.

  FIG. 6 is a diagram for explaining division and integration of partial areas. FIG. 6A shows an example before division, and FIG. 6B shows an example after division. Here, if a partial region that is not adjacent to any adjacent partial region after division is generated like an enclave, it is integrated into the partial region having the closest feature amount as in the first embodiment.

  In step S511, the CPU 105 integrates the areas divided in S510 into the adjacent partial areas having the closest features. That is, the number of the partial area is updated in the same manner as S206 in the first embodiment. FIG. 6C shows an example of the updated partial area.

  In step S512, the CPU 105 causes the 16 processing cores to execute the recognition processing of 160 partial areas. That is, each processing core outputs the category estimated from the feature using the SVM presented in the preconditions and the parameters of the SVM dictionary in the memory 101. Therefore, each processing core performs recognition processing of 10 (= 160/16) partial areas. The recognition results of 160 partial areas are totaled for each pixel, and for pixels having two to three recognition results (category likelihoods), the average of the category likelihoods is used as the category likelihood of the pixel. .

  In the above description, the processing time is considered as the processing load on the entire processing. However, other indexes may be used. For example, the power consumption required for adjusting the number of divisions and the power consumption of the recognition process may be calculated in advance, and the number of divisions may be determined so that the overall power consumption is the lowest. Further, the data transfer time may be used as an index.

  As described above, according to the second embodiment, the number of divisions, the number of executable processes of the processing system that executes the recognition process, and the change in the processing time that occurs when the number of divisions is adjusted as an influence on the entire process, Determine based on. With this configuration, it is possible to divide an image into a number of divisions that can efficiently use a plurality of processing cores while reducing the possibility of an increase in processing time.

(Third embodiment)
In the third embodiment, another mode of adjusting the number of divisions based on the number of executable processes of the processing system that executes the recognition process and the change in the processing time that occurs when the number of divisions is adjusted will be described. In the following description, description of processes similar to those described in the first and second embodiments will be omitted, and only processes different from those described in the first and second embodiments will be described.

<Prerequisites>
First, preconditions in the third embodiment will be described. Other preconditions not described below are the same as those in the first and second embodiments.

  It is assumed that a dictionary is created for each division number in advance. Here, in particular, it is assumed that a dictionary learned in a partial region obtained by dividing the number of k-means classes in increments of 25, 50, 75,.

  FIG. 8 is a diagram illustrating a table in which dictionary numbers and class numbers are associated with each other. Here, as the dictionary number, the class number “25” is the dictionary number “1”, the class number “50” is the dictionary number “2”.

  As with the first embodiment, the number of cores on the GPU is used as the number of executable processes of the processing system that executes the recognition process. In this embodiment, an increase in processing time that occurs when the number of divisions is adjusted and an increase in processing time that occurs when adjustment is not performed are used as the influence on the entire processing. That is, from the viewpoint of processing efficiency, the number of divisions is selected to be an integral multiple of the number of GPU cores, and from the viewpoint of processing time, the entire processing time is selected to be short.

The contents of the recognition process are changed depending on whether the number of divisions is adjusted or not. Specifically, when the number of divisions is not adjusted, recognition processing is performed using a dictionary learned with the same number of classes. When the number of divisions is adjusted, in addition to the recognition result by the dictionary described above, a recognition result by a dictionary having a class number one level smaller than that is used to perform a process of integrating both results. In other words, there is a possibility that the number of classes may be reduced compared to before the integration due to the process of integrating the divided areas. Therefore, by using a dictionary with a small number of classes and outputting the results, the recognition accuracy can be improved. Aim for improvement. For example, when the class number is divided by “50” and finally the number of divisions becomes “48”, in addition to the recognition processing by the dictionary divided by the class number “50” (dictionary with dictionary number “1”), Recognition processing is also performed using a dictionary divided by class number “25” (dictionary with dictionary number “2”). Both results will be integrated as appropriate. That is, in the above case,
Recognition result of class number “25”: Recognition result of class number “50” = (50−48) :( 48−25) = 2: 23
Integrate the results with a weight of. When the number of divisions is adjusted, it is necessary to perform recognition processing with two types of dictionaries. Therefore, when recognition processing is performed with one type of dictionary (similar to the second embodiment, 1 (millisecond / 1 GHz · piece) is assumed. ), The processing time is extended. Here, the processing time required to recognize one partial area using two dictionaries is assumed to be 1.1 (millisecond / 1 GHz · piece). Furthermore, since there are 16 GPU cores, if the number of partial regions is 16 or less, processing can be performed at once by parallel processing. In other words, all can be processed in the same processing time. Note that the number of clocks of the GPU that performs the recognition process is 1 GHz as in the second embodiment.

  Unlike the second embodiment, the processing for adjusting the number of divisions is performed by the GPU. The processing time is 0.1 (millisecond / 1 GHz · set). At this time, since there are 16 GPU cores, if the number of sets of partial areas is 16 or less, processing can be performed at a time by parallel processing. That is, it is possible to perform processing in the same processing time.

<Operation of the device>
FIG. 7 is a flowchart showing recognition processing in the third embodiment. Steps S701 to S704, S706 to S707, and S711 are the same as S201 to S204, S205 to S206, and S208 in the first embodiment, and a description thereof will be omitted. Here, it is assumed that the number of partial areas finally obtained in S702 is “54” and the number of divisions obtained in S704 is “48”.

  In step S705, the CPU 105 calculates and compares the processing time when the number of divisions is adjusted (hereinafter referred to as adjustment time processing time) and the processing time when the number of divisions is not adjusted (hereinafter referred to as unadjusted processing time). .

  As a result of the comparison, if it is determined that the adjustment processing time is smaller than the non-adjustment processing time, it means that the total processing time is reduced when the number of divisions is adjusted, and the process proceeds to step S706. On the other hand, if it is determined that the adjustment processing time> the non-adjustment processing time, it means that the entire processing time is reduced when the number of divisions is not adjusted, and the process proceeds to step S708.

  More specifically, regarding the increase in processing time that occurs when the number of divisions is adjusted, for the process of adjusting the number of divisions, the number of partial areas is reduced from 54 to 48 this time. Six sets of partial areas need to be integrated. The increase t11 in processing time required for adjusting the number of divisions based on the preconditions is obtained as follows.

t11 = 0.1 (millisecond / 1 GHz · set) × 1 (GHz) × ceil (6 (set) / 16) = 0.1 (ms)
Furthermore, in this embodiment, when the number of divisions is adjusted, recognition processing is performed using two dictionaries. The increase t12 in the recognition processing time is obtained based on the preconditions as follows.

t12 = (recognition processing time for two types of dictionaries—recognition processing time for one type of dictionaries) = (1.1-1) (milliseconds / 1 GHz · piece) × 1 (GHz) × ceil (48 (pieces) ) / 16) = 0.3 (milliseconds)
Therefore, the total adjustment processing time is as follows.

t1 = t11 + t12 = 0.1 (ms) +0.3 (ms) = 0.4 (ms)
On the other hand, with respect to the increase in processing time that occurs when the number of divisions is not adjusted, it is necessary to process six processes this time on the GPU. When the increase t2 in the processing time is obtained based on the preconditions, it is as follows.

t2 = 1 (milliseconds / 1 GHz / piece) × 1 (GHz) × ceil (6 (pieces) / 16) = 1 (ms)
t1 <t2, which indicates that the overall processing time is reduced by adjusting the number of divisions. Therefore, in this case, the process proceeds to step S706. In addition, a flag for adjusting the number of divisions is set and recorded in the memory 101.

  In step S708, the CPU 105 determines whether the division number adjustment process has been performed. That is, it is checked whether a flag for adjusting the number of divisions is set in the memory 101. If it is determined that the flag is not set, the process proceeds to step S709. If it is determined that the flag is set, the process proceeds to step S710.

  Step S709 is a process of recognizing with one type of dictionary (a dictionary divided by class number “50”, that is, a dictionary with dictionary number “1”), and is the same as S512 in the second embodiment.

  In step S710, recognition processing is performed using two types of dictionaries. Since the number of divisions finally becomes “48”, a dictionary divided by class number “50” (dictionary with dictionary number “2”) and a dictionary divided by class number 25 (dictionary with dictionary number “1”) are used. Then, each recognition result will be divided and integrated.

  That is, in the above case, the recognition result of “class number“ 25 ”: recognition result of class number“ 50 ”= 2: 23” is weighted and the results are integrated. For example, in a certain partial area, the result of the class number “25” is “organism: nature: thing: others = 0.7: 0.2: 0.1: 0”, and the result of the class number “50” is “ Biology: Nature: Object: Others = 0.4: 0.4: 0.2: 0 ”. In this case, when the results are appropriately divided and integrated, the result is “organism: nature: thing: other = 0.424: 0.384: 0.192: 0”.

  As described above, according to the third embodiment, the number of divisions, the number of executable processes of the processing system that executes the recognition process, the change in the processing time that occurs when the number of divisions is adjusted as an influence on the entire process, Determine based on. With this configuration, it is possible to efficiently perform more accurate image recognition.

(Fourth embodiment)
The fourth embodiment is a modification of the second embodiment, and describes a mode in which the number of divisions is further determined based on deterioration in accuracy of the recognition process as an influence on the entire process. In the following description, description of processes similar to those described in the second embodiment will be omitted, and only processes different from those in the second embodiment will be described.

  In the present embodiment, first, the accuracy that deteriorates due to the division number adjustment is defined as the accuracy degradation cost e. It is assumed that the accuracy degradation cost e has the same dimension (ms) as the processing time, and that the accuracy degradation cost e is proportional to the number m of sets of partial areas integrated by the division number adjustment.

e = a (ms / piece) × m (piece) = am (ms)
In this embodiment, the proportionality constant a is set so that the accuracy degradation cost e when the accuracy obtained as a result of adjusting the number of divisions is outside the allowable range is the same as the processing time when the number of divisions is not adjusted. Select in advance. For example, when it is statistically known that the average processing time is 3 (ms) when the accuracy required when m = 10 on average cannot be satisfied and the number of divisions at that time is not adjusted It becomes as follows.

am = a × 10 = 3 Therefore, a = 0.3 (ms / piece)
Then, the total cost f is defined, and the final division number is adjusted based on the total cost f. The dimension of the total cost f is the same as the processing time (ms).

f = t (increase in processing time) (ms) + e (ms)
It is assumed that the larger the total cost f, the greater the influence on the entire process.

  When the above formula is applied to the second embodiment, when the number of divisions is adjusted, three sets of partial areas are integrated, so the accuracy degradation cost e is as follows.

e = 0.3 × 3 = 0.9 (ms)
On the other hand, since the increase in processing time is 0.3 (ms) from the second embodiment, the total cost f is as follows.

f1 = 0.9 + 0.3 = 1.2 (ms)
Next, when the number of divisions is not adjusted, the accuracy degradation cost e is 0 (ms). On the other hand, since the increase in processing time is 1 (ms) from the second embodiment, the total cost f is as follows.

f2 = 1 + 0 = 1 (ms)
Therefore, since f1> f2, the process proceeds without adjusting the number of divisions (that is, with the number of divisions set to “163”).

  In addition, although the formula of the above-mentioned accuracy degradation cost and total cost was defined, this is an example and there may be another definition. For example, it is possible to consider additional terms such as data transfer cost and power consumption in addition to the total cost.

  As described above, according to the fourth embodiment, the number of divisions, the number of executable processes of the processing system that executes the recognition process, and the change in the processing time that occurs when the number of divisions is adjusted as an influence on the entire process, Determine based on. With this configuration, it is possible to determine the number of divisions that enable efficient recognition processing in consideration of the balance between processing time and accuracy degradation.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  101 memory; 102 camera; 103 input device; 104 display; 105 CPU; 106 GPU; 107 processing core

Claims (12)

An image processing apparatus that performs processing using a plurality of processing units,
A dividing means for dividing the input image into a plurality of partial areas;
The effect of adjusting the number of the plurality of processing units, the first division number that is the number of the plurality of partial areas, and the number of the plurality of partial areas from the first division number to the second division number; Determining means for determining the second number of divisions based on:
Adjusting means for adjusting a part of the partial area of the first division number so that the input image is divided into areas by the second division number;
Control means for controlling the plurality of processing units to execute predetermined processing on the second divided number of partial regions;
An image processing apparatus comprising: The image processing apparatus according to claim 1, wherein the determination unit determines the second division number so as to be an integral multiple of the number of the plurality of processing units. The image processing apparatus according to claim 2, wherein the control unit equally distributes the second divided number of partial areas to the plurality of processing units. A derivation unit for deriving a feature amount of each partial region;
4. The image according to claim 1, wherein the adjustment unit integrates adjacent partial regions having the similar feature amount among the first divided partial regions. 5. Processing equipment. The image processing apparatus according to claim 4, wherein the predetermined process is an image recognition process based on a feature amount of a partial region. The image processing apparatus according to claim 5, wherein the image recognition process includes a plurality of processes under different conditions and a process of dividing and integrating the results of the plurality of processes. The image processing apparatus according to claim 1, wherein the influence is a degree of deterioration of accuracy in the predetermined processing by the plurality of processing units. 7. The apparatus according to claim 1, wherein the influence is an increase in a sum of a time required for adjustment by the adjusting unit and a time required for the predetermined processing by the plurality of processing units. The image processing apparatus described. First predicting means for predicting a first processing load that is a processing load when the predetermined processing is executed on the first partial number of partial areas by the plurality of processing units;
Predicting a second processing load that is the sum of the processing load due to the adjusting means and the processing load when the predetermined processing is executed on the second divided number of partial areas by the plurality of processing units. Prediction means,
Further comprising
When the first processing load is smaller than the second processing load, the adjustment unit does not perform the adjustment, and the control unit performs the predetermined processing on the partial area of the first division number by the plurality of processing units. The image processing apparatus according to claim 1, wherein control is performed so as to execute the process. The image processing apparatus according to claim 9, wherein the processing load includes any one of processing time, power consumption, and data transfer time. A control method of an image processing apparatus that performs processing using a plurality of processing units,
A division step of dividing the input image into a plurality of partial areas;
The effect of adjusting the number of the plurality of processing units, the first division number that is the number of the plurality of partial areas, and the number of the plurality of partial areas from the first division number to the second division number; A determination step of determining the second number of divisions based on:
An adjustment step of adjusting a part of the partial region of the first division number so that the input image is divided into regions by the second division number;
A control step of controlling the plurality of processing units to execute predetermined processing on the second divided number of partial regions;
A control method for an image processing apparatus.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 10.