CN112771566A - Image processing apparatus, image processing method, and program - Google Patents
Image processing apparatus, image processing method, and program Download PDFInfo
- Publication number
- CN112771566A CN112771566A CN201880098076.9A CN201880098076A CN112771566A CN 112771566 A CN112771566 A CN 112771566A CN 201880098076 A CN201880098076 A CN 201880098076A CN 112771566 A CN112771566 A CN 112771566A
- Authority
- CN
- China
- Prior art keywords
- image
- filtering
- filter
- parameter set
- scheduler
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 135
- 238000003672 processing method Methods 0.000 title claims abstract description 11
- 238000001914 filtration Methods 0.000 claims abstract description 294
- 230000015654 memory Effects 0.000 claims abstract description 54
- 238000010191 image analysis Methods 0.000 claims description 61
- 230000000694 effects Effects 0.000 claims description 54
- 238000000034 method Methods 0.000 claims description 45
- 238000013528 artificial neural network Methods 0.000 claims description 14
- 238000002372 labelling Methods 0.000 claims description 10
- 238000013473 artificial intelligence Methods 0.000 abstract description 85
- 230000000875 corresponding effect Effects 0.000 description 22
- 238000010586 diagram Methods 0.000 description 15
- 239000011159 matrix material Substances 0.000 description 12
- 230000008569 process Effects 0.000 description 11
- 238000012360 testing method Methods 0.000 description 11
- 238000007781 pre-processing Methods 0.000 description 9
- 238000004364 calculation method Methods 0.000 description 6
- 101150041570 TOP1 gene Proteins 0.000 description 5
- 101100153581 Bacillus anthracis topX gene Proteins 0.000 description 4
- MHABMANUFPZXEB-UHFFFAOYSA-N O-demethyl-aloesaponarin I Natural products O=C1C2=CC=CC(O)=C2C(=O)C2=C1C=C(O)C(C(O)=O)=C2C MHABMANUFPZXEB-UHFFFAOYSA-N 0.000 description 4
- 239000000872 buffer Substances 0.000 description 4
- 238000013507 mapping Methods 0.000 description 4
- 238000010606 normalization Methods 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000004913 activation Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000009499 grossing Methods 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000011664 signaling Effects 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000012886 linear function Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the application discloses an image processing device and an image processing method applied to the field of artificial intelligence AI, wherein the image processing device comprises: a scheduler for obtaining a parameter set selection instruction input by an input device, the parameter set selection instruction being an instruction input by a user through the input device and being used for instructing to select N filtering parameter set files from F filtering parameter set files stored in a memory, where F is a positive integer greater than 1 and N is an integer greater than 0; a hardware filter coupled to the scheduler for filtering the reference picture using a set of filtering parameters included in one of the N filtering parameter set files. In the embodiment of the application, a user can configure the filtering parameter file of the hardware filter according to the user self, the requirements of different scenes can be met, and the implementation is simple.
Description
The present disclosure relates to the field of electronic technologies, and in particular, to an image processing apparatus and an image processing method.
Artificial Intelligence (AI) is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human Intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision and reasoning, human-computer interaction, recommendation and search, AI basic theory, and the like. With the development of AI technology, AI networks, such as Neural Networks (NN), recurrent Neural networks, convolutional Neural networks, deep Neural networks, etc., are increasingly applied in various fields. In real life, AI networks have been widely used in the fields of image processing, image recognition, image classification, and the like.
Before image analysis or image processing is performed on an image to be processed by using an AI network, the image to be processed is usually preprocessed. The main purposes of image preprocessing are to eliminate irrelevant information in an image, recover useful real information, enhance the detectability of relevant information and simplify data to the maximum extent, so that the reliability and the precision of image processing operations such as feature extraction, image segmentation, image matching, image identification and the like of an AI network on the image are improved. The image filtering is an indispensable operation in image preprocessing, and is to suppress the noise of an image under the condition of keeping the detail features of the image as much as possible, and the effectiveness and reliability of subsequent image processing and analysis are directly affected by the quality of the processing effect.
The image filtering in the image preprocessing mainly comprises adjusting the image size and carrying out denoising and smoothing processing on noise in the zoomed image. For example, image denoising may be performed using a median hardware filter, a gaussian hardware filter, or the like. The size adjustment is mainly to cut and scale the image to be processed of the AI network according to the size requirement of the AI network on the input image so as to adapt to the standard size required by the specific AI network. Currently, a hardware filter is usually adopted to implement image filtering in the image preprocessing process, such as common RC filtering, LC filtering, and the like. Because the filtering parameters in the hardware filters are pre-configured, each hardware filter can at most meet the image filtering requirements of a limited number of image filtering scenes, and a better image filtering effect can not be obtained in any image filtering scene. Therefore, the problem that the filtering parameters in the hardware filter cannot be adjusted needs to be solved, so that a better image filtering effect can be obtained in any image filtering scene.
Disclosure of Invention
The embodiment of the application provides an image processing device and an image processing method, a hardware filter utilizes a filtering parameter configured by a user to filter an image, and the problem that the hardware filter can only utilize a set of fixed filtering parameters to filter can be solved.
In a first aspect, an embodiment of the present application provides an image processing apparatus, including:
a scheduler for obtaining a parameter set selection instruction input by an input device, the parameter set selection instruction being an instruction input by a user through the input device and being used for instructing to select N filtering parameter set files from F filtering parameter set files stored in a memory, where F is a positive integer greater than 1 and N is an integer greater than 0;
a hardware filter coupled to the scheduler for filtering the reference picture using a set of filtering parameters included in one of the N filtering parameter set files.
In the embodiment of the application, the user configures the filtering parameter file in the hardware filter by himself, so that the hardware filter can filter the image by using the filtering parameter file configured by the user, the requirements of different scenes can be met, and the implementation is simple.
In an optional implementation manner, an image processor respectively coupled to the scheduler and the hardware filter is configured to parse the filter parameter set file to obtain a reference filter parameter set; and determining the group of filter parameters in the reference filter parameter set as parameters to be adopted by the hardware filter for filtering the reference image by using the reference filter parameter set according to a first scaling coefficient of the reference image in the horizontal direction and a second scaling coefficient of the reference image in the vertical direction.
The reference filter parameter set file includes at least two sets of filter parameters.
In the implementation mode, the image processor determines parameters to be adopted by the hardware filter for filtering the reference image by using a reference filtering parameter set according to a first scaling coefficient of the reference image in the horizontal direction and a second scaling coefficient of the reference image in the vertical direction; so that the size of the reference picture filtered picture meets the requirements.
In an optional implementation manner, the scheduler is further configured to send, according to the parameter set selection instruction, N file names corresponding to the N filter parameter set files to the image processor, where the N file names correspond to the N filter parameter set files one to one;
the image processor is further configured to acquire the reference filter parameter set file from the memory according to a reference file name, where the reference file name is a file name of the reference filter parameter set file and is included in the N file names.
Optionally, the scheduler is further configured to read the reference image from the memory and send the reference image to the image processor.
In this implementation, the scheduler reads the reference filter parameter set from the memory by sending a reference file name to the image processor, so that the image processor reads the reference filter parameter set from the memory using the reference file name; the realization is simple, and the consumed signaling resource is less.
In an alternative implementation, the image processor is specifically configured to select a set of filtering parameters in a horizontal direction from the reference filtering parameter set according to the first scaling factor, and to select a set of filtering parameters in a vertical direction from the reference filtering parameter set according to the second scaling factor; taking the set of filtering parameters in the horizontal direction and the set of filtering parameters in the vertical direction as the set of filtering parameters; configuring the set of filtering parameters to registers of the hardware filter;
the hardware filter is specifically configured to read the set of filtering parameters in the register, and filter the reference image by using the set of filtering parameters.
The reference filter parameter set file includes at least two sets of filter parameters in a horizontal direction and at least two sets of filter parameters in a vertical direction. The image processor can determine a set of horizontal filtering parameters corresponding to any horizontal scaling factor by using the mapping relation between the horizontal scaling factor of the image and the horizontal filtering parameters; the mapping relationship between the scaling factor in the vertical direction of the image and the filtering parameter in the vertical direction can be utilized to determine a set of filtering parameters in the vertical direction corresponding to the scaling factor in any vertical direction. Optionally, the scaling factor in the horizontal direction of the image corresponds to H parameter intervals, the reference filter parameter set file includes H groups of filter parameters in the horizontal direction, the H groups of filter parameters in the horizontal direction correspond to the H parameter intervals one to one, and H is an integer greater than 1. The image processor may select a set of horizontal-direction filter parameters corresponding to a parameter interval in which the first scaling coefficient is located, from the H sets of horizontal-direction filter parameters. Optionally, the scaling factor in the vertical direction of the image corresponds to G parameter intervals, the reference filter parameter set file includes G sets of filter parameters in the vertical direction, the G sets of filter parameters in the vertical direction correspond to the G parameter intervals one to one, and G is an integer greater than 1. The image processor may select a set of vertical-direction filter parameters corresponding to a parameter interval in which the second scaling factor is located, from the G sets of vertical-direction filter parameters.
In this implementation, the image processor may configure the filtering parameters for the hardware filter according to the scaling factor of the reference image in the horizontal direction and the scaling factor in the vertical direction, so that the size of the image after filtering the reference image meets the requirement.
In an optional implementation manner, the hardware filter is further configured to send the image obtained by filtering the reference image to the image processor;
the image processor is further configured to send the reference image filtered image to the scheduler; the image processing apparatus further includes:
a neural network processor NPU coupled to the scheduler;
the scheduler is further configured to input the reference picture filtered picture to the NPU;
and the NPU is used for carrying out image analysis on the image after the reference image is filtered by utilizing a target AI network and sending an obtained image analysis result to the scheduler.
In the implementation mode, the AI network is used for carrying out image analysis on the image after the reference image is filtered, and the image processing efficiency is high.
In an optional implementation manner, the image processing apparatus further includes:
a result comparator coupled to the scheduler;
the scheduler is further configured to send the annotation information of the reference image and the image analysis result to the result comparator;
and the result comparator is used for comparing the image analysis result with the labeling information of the reference image to obtain a comparison result of the parameter image.
In the implementation mode, the result comparer is used for comparing the image analysis result with the labeling information of the image so as to determine the filtering effect of the hardware filter.
In an optional implementation manner, the result comparator is further configured to send the comparison result of the reference image and the comparison result of each image in the image set except the reference image to the scheduler; the mode of obtaining the comparison result of each image except the reference image in the image set by the result comparator is the same as the mode of obtaining the comparison result of the reference image, and the image set comprises at least two images;
the scheduler is further configured to determine a filtering effect of the reference filtering parameter set according to a comparison result of each image in the image set; respectively determining the filtering effects of the filtering parameter sets except the reference filtering parameter set in the N filtering parameter sets, wherein the N filtering parameter sets are N parameter sets obtained by analyzing the N filtering parameter set files by the image processor; configuring the target filter parameter set with the best filter effect in the N filter parameter sets as the filter parameter set of the hardware filter; wherein the scheduler determines a filtering effect of the filtering parameter sets of the N filtering parameter sets other than the reference filtering parameter set in the same manner as the filtering effect of the reference filtering parameter set.
In this implementation, the scheduler configures the target filter parameter set with the best filtering effect among the plurality of filter parameter sets as the filter parameter set of the hardware filter, so as to improve the performance of image processing.
In an optional implementation manner, the NPU is further configured to obtain an AI network selection instruction input by the input device, where the AI network selection instruction is used to select the target AI network of at least two AI networks to be selected; configuring the network adopted by the NPU as the target AI network according to the AI network selection instruction;
alternatively, the first and second electrodes may be,
the NPU is also used for obtaining AI network programming codes input by the input equipment; executing the AI network programming code to implement the target AI network.
In this implementation, the user configures the AI network in the image processing apparatus by himself, so as to process different types of images or perform different image processing operations, which can satisfy different requirements.
Second aspect an embodiment of the present application provides an image processing method applied to an image processing apparatus, where the image processing apparatus includes a scheduler and a hardware filter, the method including:
the scheduler obtains a parameter set selection instruction input by an input device, wherein the parameter set selection instruction is an instruction input by a user through the input device and is used for instructing to select N filtering parameter set files from F filtering parameter set files stored in a memory, wherein F is a positive integer larger than 1, and N is an integer larger than 0;
the hardware filter filters the reference image using a set of filter parameters included in one of the N filter parameter set files.
In the embodiment of the application, a user can configure the filtering parameter file of the hardware filter according to the user, so that the hardware filter can filter the image by using the filtering parameter file configured by the user, the requirements of different scenes can be met, and the implementation is simple.
In an alternative implementation, the image processor further comprises an image processor coupled to the scheduler and the hardware filter, respectively; before the hardware filter filters the reference image by using a set of filtering parameters contained in one of the N filtering parameter set files, the method further includes:
the image processor analyzes the filtering parameter set file to obtain a reference filtering parameter set; and determining the group of filter parameters in the reference filter parameter set as parameters to be adopted by the hardware filter for filtering the reference image by using the reference filter parameter set according to a first scaling coefficient of the reference image in the horizontal direction and a second scaling coefficient of the reference image in the vertical direction.
In an optional implementation manner, before the image processor configures a set of filtering parameters in a reference filtering parameter set file as filtering parameters to be used by the hardware filter for filtering a reference image, the method further includes:
the scheduler sends N file names corresponding to the N filtering parameter set files to the image processor according to the parameter set selection instruction, wherein the N file names are in one-to-one correspondence with the N filtering parameter set files;
the image processor acquires the reference filter parameter set file from the first memory according to a reference file name, which is a file name of the reference filter parameter set file and is included in the N file names.
In an optional implementation manner, the determining, according to the first scaling coefficient of the reference image in the horizontal direction and the second scaling coefficient of the reference image in the vertical direction, the set of filter parameters in the reference filter parameter set as parameters to be adopted by the hardware filter to filter the reference image by using the reference filter parameter set includes:
selecting, by the image processor, a set of horizontal-direction filter parameters from the reference filter parameter set according to the first scaling factor and a set of vertical-direction filter parameters from the reference filter parameter set according to the second scaling factor; taking the set of filtering parameters in the horizontal direction and the set of filtering parameters in the vertical direction as the set of filtering parameters; configuring the set of filtering parameters to registers of the hardware filter;
the hardware filter filtering the reference image by using a set of filtering parameters included in one of the N filtering parameter set files comprises:
the hardware filter reads the set of filtering parameters in the register and filters the reference image using the set of filtering parameters.
In an optional implementation, the image processing apparatus further comprises a neural network processor NPU coupled with the scheduler; after the hardware filter filters the reference image by using a set of filtering parameters contained in one of the N filtering parameter set files, the method further includes:
the hardware filter sends the image after the reference image is filtered to the image processor;
the image processor sends the reference image filtered image to the scheduler;
the scheduler inputs the reference picture filtered picture to the NPU;
and the NPU utilizes a target AI network to perform image analysis on the image after the reference image filtering, and sends the obtained image analysis result to the scheduler.
In an optional implementation, the image processing apparatus further comprises a result comparator coupled to the scheduler; the NPU performs image analysis on the image after the reference image filtering by using a target AI network, and after sending an obtained image analysis result to the scheduler, the method further includes:
the scheduler sends the labeling information of the reference image and the image analysis result to the result comparator;
and the result comparer compares the image analysis result with the labeling information of the reference image to obtain a comparison result of the parameter image.
In an optional implementation manner, after the result comparing device compares the image analysis result with the labeling information of the reference image to obtain a comparison result of the parameter image, the method further includes:
the result comparer sends the comparison result of the reference image and the comparison result of each image except the reference image in the image set to the scheduler; the mode of obtaining the comparison result of each image except the reference image in the image set by the result comparator is the same as the mode of obtaining the comparison result of the reference image, and the image set comprises at least two images;
the scheduler determines the filtering effect of the reference filtering parameter set according to the comparison result of each image in the image set; respectively determining the filtering effects of the filtering parameter sets except the reference filtering parameter set in the N filtering parameter sets, wherein the N filtering parameter sets are N parameter sets obtained by analyzing the N filtering parameter set files by the image processor; configuring the target filter parameter set with the best filter effect in the N filter parameter sets as the filter parameter set of the hardware filter; wherein the scheduler determines a filtering effect of the filtering parameter sets of the N filtering parameter sets other than the reference filtering parameter set in the same manner as the filtering effect of the reference filtering parameter set.
In an optional implementation manner, before the NPU performs image analysis on the image filtered by the reference image by using a target AI network and sends an obtained image analysis result to the scheduler, the method further includes:
the NPU obtains an AI network selection instruction input by the input equipment, wherein the AI network selection instruction is used for selecting the target AI network in at least two AI networks to be selected; configuring a network adopted by the NPU as the target AI network according to the AI network selection instruction;
alternatively, the first and second electrodes may be,
the NPU obtains an AI network programming code input by the input equipment; executing the AI network programming code to implement the target AI network.
Fig. 1 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present disclosure;
fig. 2 is a schematic diagram of a filter parameter set selection interface according to an embodiment of the present disclosure;
FIG. 3 is a schematic structural diagram of another image processing apparatus provided in the present application;
FIG. 4 is a schematic diagram illustrating a method for calculating an overlap between two bounding boxes according to an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of another image processing apparatus according to an embodiment of the present application;
fig. 6 is a schematic diagram of an AI network selection interface according to an embodiment of the present disclosure;
fig. 7 is a schematic structural diagram of a neural network processor according to an embodiment of the present disclosure;
fig. 8 is a schematic flowchart of an image processing method according to an embodiment of the present application;
fig. 9 is a flowchart of a method for comparing filtering effects of multiple filter parameter set files according to an embodiment of the present disclosure.
In order to make the embodiments of the present application better understood, the technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, but not all embodiments.
The terms "first," "second," and "third," etc. in the description and claims of the present application and the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. Furthermore, the terms "comprises" and "comprising," as well as any variations thereof, are intended to cover a non-exclusive inclusion, such as a list of steps or elements. A method, system, article, or apparatus is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, system, article, or apparatus.
Fig. 1 is a schematic structural diagram of another image processing apparatus according to an embodiment of the present application. As shown in fig. 1, the image processing apparatus 10 includes:
a scheduler 101, configured to obtain a parameter set selection instruction input by the input device 102, where the parameter set selection instruction is an instruction input by a user through the input device 102 and is used to instruct to select N filtering parameter set files from F filtering parameter set files stored in the memory 103, where F is a positive integer greater than 1 and N is an integer greater than 0;
a hardware filter 104 coupled to the scheduler 101 for filtering the reference picture by using a set of filtering parameters included in one of the N filtering parameter set files.
The input device 102 may be a keyboard, mouse, touch screen, etc. The F filter parameter set files stored in the memory 103 may be configured in advance when the image processing apparatus is shipped from the factory, or may be configured by the user. A method of generating a filter parameter set file is presented below: firstly, parameter strategies such as iteration step length (Kaiser step length and Gauss step length), a filtering parameter interval and the like are formulated, so that the filtering parameter interval is traversed as quickly and comprehensively as possible; and then, according to the established parameter strategy, taking the iteration step length and the filtering parameter interval as input parameters, and generating a script by utilizing the compiled coefficient to generate a plurality of filtering parameter set files. The number of the filtering parameter set files generated by different parameter strategies is different. The user or the research and development personnel can make different parameter strategies according to the requirement to generate the required filtering parameter file. For example, 200 filter parameter set files are generated, and each filter parameter set file has 1000 sets of filter parameters in the horizontal direction and 1000 sets of filter parameters in the vertical direction. Alternatively, the image processing apparatus exposes the file name of each filter parameter set file to the user, and the user selects the filter parameter set file name through the input device 102 to configure the corresponding filter parameter set. Because the hardware filter is used for image filtering, a group of filtering parameters in any filtering parameter set file is a group of filtering parameters of the hardware filter in a spatial domain.
In the embodiment of the application, a user can configure the filtering parameter file of the hardware filter according to the user, so that the hardware filter can filter the image by using the filtering parameter file configured by the user, the requirements of different scenes can be met, and the implementation is simple.
In an alternative implementation, the image processing apparatus further comprises an image processor 105 coupled to the scheduler 101 and the hardware filter 104, respectively:
an image processor 105, configured to parse the filter parameter set file to obtain a reference filter parameter set; and determining the group of filter parameters in the reference filter parameter set as parameters to be adopted by the hardware filter for filtering the reference image by using the reference filter parameter set according to a first scaling coefficient of the reference image in the horizontal direction and a second scaling coefficient of the reference image in the vertical direction.
Optionally, the image processor 105 is specifically configured to select a set of filtering parameters in a horizontal direction from the reference filtering parameter set according to the first scaling factor, and select a set of filtering parameters in a vertical direction from the reference filtering parameter set according to the second scaling factor; taking the set of filtering parameters in the horizontal direction and the set of filtering parameters in the vertical direction as the set of filtering parameters; configure the set of filtering parameters to registers of hardware filter 101;
the hardware filter 104 is specifically configured to read the set of filtering parameters in the register, and filter the reference image by using the set of filtering parameters.
The first scaling factor of the reference image in the horizontal direction is a scaling factor of the reference image in the horizontal direction corresponding to the standard size of the input image required by the image processing apparatus. The second scaling factor of the reference image in the vertical direction is a scaling factor of the reference image in the vertical direction corresponding to the standard size of the input image required by the image processing apparatus. The reference filter parameter set file includes at least two sets of filter parameters in a horizontal direction and at least two sets of filter parameters in a vertical direction. The image processor may determine a set of horizontal filtering parameters corresponding to any horizontal scaling factor by using a mapping relationship between the horizontal scaling factor of the image and the horizontal filtering parameters; the mapping relationship between the scaling factor in the vertical direction of the image and the filtering parameter in the vertical direction can be utilized to determine a set of filtering parameters in the vertical direction corresponding to the scaling factor in any vertical direction. Optionally, the scaling factor in the horizontal direction of the image corresponds to H parameter intervals, the reference filter parameter set file includes H groups of filter parameters in the horizontal direction, the H groups of filter parameters in the horizontal direction correspond to the H parameter intervals one to one, and H is an integer greater than 1. The image processor may select a set of horizontal filtering parameters corresponding to the parameter interval in which the first scaling factor is located from the H sets of horizontal filtering parameters. Optionally, the scaling factor in the vertical direction of the image corresponds to G parameter intervals, the reference filter parameter set file includes G groups of filter parameters in the vertical direction, the G groups of filter parameters in the vertical direction correspond to the G parameter intervals one to one, and G is an integer greater than 1. The image processor may select a set of vertical filter parameters corresponding to a parameter interval in which the second scaling factor is located from the G sets of vertical filter parameters.
The reference image is an image to be currently processed by the data processing device and is one image in the image set. The memory 103 stores the set of images. Each image in the image set is labeled, that is, each image has a label information (label or real value). It is understood that each image in the image set is a test image for testing the image processing effect of the image processing apparatus. The labeled label or true value (labeling information) of the image is called group Truth, and the group Truth of each image is used for comparing with an inference value (image analysis result) output by the image through AI network processing.
In practical applications, the image processor may configure the hardware filter with the filter parameter set file for the filter parameters it employs to process each image in the image set. That is, the image processor may configure the filtering parameters for the hardware filter using one or more filtering parameter set files selected by the user, and the filtering parameter set file used by the hardware filter is not fixed but is selected by the user as needed. In addition, the user can select a plurality of filtering parameter set files, and the image processor configures filtering parameters for the hardware filter by using each parameter set text in the filtering parameter set files, so that the user can determine one filtering parameter set file with the best filtering effect in the filtering parameter set files. It can be understood that the user may test the filtering effect of each filtering parameter set file, and then select a filtering parameter set file with the best filtering effect to be used in the subsequent image processing.
In this implementation, the image processor may configure the filtering parameters for the hardware filter according to the scaling factor of the reference image in the horizontal direction and the scaling factor in the vertical direction, so that the size of the image after filtering the reference image meets the requirement.
In an optional implementation manner, the scheduler 101 is further configured to send, according to the parameter set selection instruction, N file names corresponding to the N filter parameter set files to the image processor 105, where the N file names are in one-to-one correspondence with the N filter parameter set files;
the image processor 105 is further configured to obtain the reference filter parameter set file from the memory 103 according to a reference file name, where the reference file name is a file name of the reference filter parameter set file and is included in the N file names.
Fig. 2 is a schematic diagram of a filter parameter set selection interface according to an embodiment of the present disclosure. As shown in fig. 2, the filtering parameter set selection interface includes file names of F filtering parameter set files, i.e., first to fth file names; the square in front of each file name is a selection interface of the filter parameter set file corresponding to the file name, and after a user selects the square through a keyboard or a mouse, the square is changed into black (indicating that the selection interface corresponding to the square is selected). The parameter set selection instruction is an operation of selecting a file name by a user through an input device. The filter parameter set selection interface is an interface of an image processing program. As shown in fig. 2, the second file name, the third file name, and the fifth file name are file names selected by the user, the 3 filtering parameter files corresponding to the three file names are filtering parameter files configured by the user, and the image processing apparatus performs image processing using the three filtering parameter files.
Optionally, the scheduler 101 is further configured to read the reference image from the memory 103 and send the reference image to the image processor. In actual practice, the scheduler 101 reads one image at a time from the image set and passes the image to the image processor 105.
In this implementation, the scheduler sends the file name of the filter parameter set file to the image processor, so that the image processor reads the corresponding filter parameter set file from the memory according to the received file name; the realization is simple, and the consumed signaling resource is less.
Fig. 3 is a schematic structural diagram of another image processing apparatus provided in the present application, which is a further refinement of the apparatus shown in fig. 1. As shown in fig. 3, the image processing apparatus in fig. 3 adds an NPU320 and a result comparator 340, both coupled to the scheduler 101, to the image processing apparatus in fig. 1. The NPU320 may implement a variety of AI networks, and the NPU320 may perform image analysis or image processing on the filtered image from the scheduler 101 using the AI networks. The result comparing unit 340 is used for comparing the image analysis result (image processing result) with the real result (labeled information of the test image) so as to determine the correctness of the image analysis or the image processing. The image processing apparatus in fig. 3 can realize the following operations:
301. the scheduler 101 obtains a parameter set selection instruction input by the input device 102.
The input device 102 may be a keyboard, mouse, touch screen, etc. In practical applications, the operation of the user to select a file name in the filtering parameter set selection interface in fig. 3 through the input device 102 may be understood as a parameter set selection instruction input by the input device 102.
302. The scheduler 101 reads a reference image from the memory 103.
The reference image is any image in the image set stored in the memory 103. In practical applications, the scheduler 101 reads each image in the image set in turn and sends the image to the image processor 105 for processing.
303. The scheduler 101 sends the N file names selected by the parameter set selection instruction to the image processor 105.
The N file names are file names corresponding to the N filter parameter set files among the F filter parameter set files stored in the memory 103. Alternatively, the scheduler 101 transmits one file name out of the above N file names to the image processor 105 at a time. Optionally, the scheduler sends the reference image to the image processor 105 together with at least one file name. The memory storing the F filter parameter set files and the memory storing the image sets may be the same or different.
304. The image processor 105 acquires the reference filter parameter set file from the memory 103 using the reference file name.
The reference file name is a file name of the reference filter parameter set file and is included in the N file names.
305. The image processor 105 configures the set of filtering parameters in the reference filtering parameter set file as the filtering parameters to be used by the hardware filter 104 to filter the reference image.
The foregoing embodiments describe implementations of how the image processor 105 configures the filtering parameters for the hardware filter 104 by using the reference filtering parameter set file, and are not described in detail herein.
306. The hardware filter 104 filters the reference picture using the set of filtering parameters and sends the reference picture filtered picture to the image processor 105.
307. The image processor 105 transmits the above-described reference image-filtered image to the scheduler 101.
308. The scheduler 101 transmits the reference picture filtered picture to the NPU 320.
309. The NPU320 performs image analysis on the image after the reference image filtering by using the target AI network, and sends the obtained image analysis result to the scheduler 101.
310. The scheduler 101 sends the image analysis result and the annotation information of the reference image to the result comparing unit 430.
Optionally, the result comparing device 340 stores the annotation information of the reference image, and the scheduler 101 only needs to send the image analysis result to the result comparing device 340.
311. The result comparing unit 430 compares the image analysis result with the labeled information of the reference image to obtain a comparison result of the parameter image.
The NPU320 uses different AI networks or outputs different image analysis results in different forms, and the result comparing unit 340 uses different comparison algorithms. For example, the AI network utilized by the NPU320 is a Resnet18 network or a Resnet50 network, and the image analysis result output by the NPU320 is a label (classification) obtained by identifying an image; the result comparator 340 compares the image analysis result output by the NPU320 with the ground route to obtain top 1-top 5. top1 refers to the probability that the classification that the NPU320 predicts the maximum probability using the AI network is correct. top5 refers to the probability that the NPU320 has the correct classification out of the 5 classifications (labels) that the AI network predicts the highest probability. Top 2-top 4 have similar meanings to top 5. For another example, the AI network utilized by the NPU320 is a fasterncn network, and the image analysis result output by the NPU320 is a bounding box (bounding box) obtained by performing object detection on an image; the result comparing unit 340 uses an overlap over Union (IoU) comparison algorithm, that is, calculates the overlap between the bounding box output by the NPU320 and the bounding box (labeled information) labeled in the image. IoU, the degree of overlap of the two bounding boxes is defined, the degree of overlap IoU of the first bounding box and the second bounding box is calculated as: IoU ═ A ≡ B)/(A ≡ B). Where A denotes an area occupied by the first bounding box, B denotes an area occupied by the second bounding box, A ^ B denotes an area where the first bounding box and the second bounding box overlap, and AomebB denotes an area where the first bounding box and the second bounding box are merged. Fig. 4 is a schematic diagram illustrating a calculation of an overlap degree between two bounding boxes according to an embodiment of the present disclosure. As shown in fig. 4, the black area is an overlapping area (a ∞ B) of two boundary frames, and the entire area is an area (a ≡ B) where two boundary frames are merged.
Repeating 302 to 311 until the scheduler 101 obtains the comparison result of processing each image in the image set by using the reference filter parameter set file; 301 to 311 are repeated until the scheduler 101 obtains the comparison result for each image in the image set processed by the N filter parameter set files.
312. The scheduler 101 determines the filtering effect of the reference filtering parameter set according to the comparison result of each image in the image set; respectively determining the filtering effect of the filtering parameter sets except the reference filtering parameter set in the N filtering parameter sets; and configuring the target filter parameter set with the best filter effect in the N filter parameter sets as the filter parameter set of the hardware filter.
The N filter parameter sets are N parameter sets obtained by the image processor parsing the N filter parameter set files. The scheduler 101 determines the filtering effect of the filtering parameter sets other than the reference filtering parameter set among the N filtering parameter sets in the same manner as the filtering effect of the reference filtering parameter set. It can be understood that the better the image filtering effect of the image processing device, the better the image processing or image analysis effect. Therefore, the quality of the image analysis result obtained by the image processing apparatus processing the image set using each filter parameter set can reflect the quality of the effect of the filter processing using the filter parameter set. The filtering effect of the reference filtering parameter set may be obtained according to a plurality of comparison results (overlapping degrees) output by the result comparator. Optionally, the filtering effect of each filtering parameter set in the N filtering parameter sets is top1 obtained by the image processing apparatus performing image recognition on each image in the image set by using the filtering parameter set. That is, the accuracy of image recognition is positively correlated with the filtering effect. Optionally, the filtering effect of each filtering parameter set in the N filtering parameter sets is the number of overlapping degrees exceeding a target threshold, which are obtained by the image processing apparatus using the filtering parameter set to perform object detection on each image in the image set. The target threshold may be 0.5, 0.6, 0.75, 0.8, etc. For example, the image processing apparatus processes the images in the image set using the first filtering parameter set to obtain 600 overlapping degrees exceeding a target threshold, and processes the images in the image set using the second filtering parameter set to obtain 800 overlapping degrees exceeding the target threshold; the filtering effect of the second set of filter parameters is better than the filtering effect of the first set of filter parameters. In practical applications, the filtering effects of different sets of filtering parameters may be compared in a variety of ways, which is not limited in this application.
In the embodiment of the application, the image processing device compares the filtering effects of the plurality of filtering parameter set files selected by the user, and selects one filtering parameter set file with the best filtering effect, so that the filtering parameter set file with the best filtering effect can be quickly determined, and the image analysis efficiency is improved.
Fig. 5 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application. The image processing apparatus may be a device without a hardware filter and an NPU, such as a desktop computer, a notebook computer, and the like, and may be applied to a developer scenario. As shown in fig. 1, the image processing apparatus includes a main device 501 and an external device 500. External device 500 is coupled to host device 501 through IO interface 511. Alternatively, the external device 500 may be coupled to the main device 501 by other means or the external device 100 may be integrated with the main device (i.e., the external device 500 may be integrated into the main device 501). The processor 503 and the memory 103 are both coupled to the system bus 505. Processor 503 may be one or more processors, each of which may include one or more processor cores. A display adapter (video adapter)507, the display adapter 507 driving a display 509, the display 509 coupled to the system bus 505. The display 509 may be used to display information entered by or provided to the user as well as various program interfaces. The system bus 505 is coupled to the I/O interface 511. The I/O interface 511 communicates with various I/O devices, such as an input device 102 (e.g., keyboard, mouse, touch screen, etc.), a multimedia disc (media tray)515 (e.g., CD-ROM, multimedia interface, etc.), and a transceiver 517 (which may send and/or receive radio communication signals).
The processor 503 may be a general-purpose Central Processing Unit (CPU), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the embodiments of the present Application.
The memory 103 stores various software programs and/or sets of instructions. In particular implementations, memory 103 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 103 may store an operating system (hereinafter referred to simply as a system), such as an embedded operating system like ANDROID, IOS, WINDOWS, or LINUX. The memory 103 may also store a network communication program that may be used to communicate with one or more additional devices, one or more terminal devices, one or more network devices. The memory 103 may further store a user interface program, which may vividly display the content of the application program through a graphical operation interface, and receive a control operation of the application program by a user through input controls such as menus, dialog boxes, and buttons.
The image processing apparatus can communicate with other devices through the network interface 521. Optionally, the network interface 521 communicates with other devices via the internet, ethernet, virtual private network, WiFi network, cellular network, etc.
Hard disk drive interface 523 is coupled to system bus 505. Hardware drive interface 523 is coupled to hard disk drive 525. A system memory 527 is coupled to the system bus 505. The data running in the system memory 527 may include the operating system 529 and the application 531 of the image processing apparatus. The applications 531 may include social applications, image management applications (e.g., photo albums), map-type applications (e.g., google maps), browsers, gaming applications, programming software, and so forth.
The operating system includes a shell (shell)533 and a kernel (kernel) 535. Shell 533 is an interface between the user and the kernel of the operating system. The shell is the outermost layer of the operating system. The shell manages the interaction between users and the operating system: waits for user input, interprets the user input to the operating system, and processes the output results of the various operating systems.
The kernel 535 is made up of those parts of the operating system that are used to manage memory, files, peripherals, and system resources. Interacting directly with the hardware, the operating system kernel typically runs processes and provides inter-process communication, CPU slot management, interrupts, memory management, IO management, and the like.
The image processor 105 is coupled to the processor 503, and the image processor 105 is used for processing the image, and is a hardware dedicated to image processing and configured to configure the filtering parameters of the hardware filter 104. The result comparing device 340 is coupled to the processor 503, and the result comparing device 340 is used for comparing the image analysis result (image processing result) with the real result (labeled information of the test image) so as to determine the correctness of the image analysis or the image processing. The image analysis results may come from a Neural-Network Processing Unit (NPU) 320. The NPU320 may be replaced with a Tensor Processor (TPU) and other machine learning capable processors. External device 500 may include hardware filter 104, NPU320, and scheduler 101. The hardware filter 104 is coupled to an image processor 105, and the image processor 105 can configure the filtering parameters of the hardware filter 104. Optionally, the processor 503 allocates image processing tasks to the scheduler 101, and the scheduler 101 completes the tasks allocated by the processor 503 by calling the hardware filter 104 and the NPU 320. Alternatively, the processor 503 directly calls the hardware filter 104 and the NPU320 to complete the image processing task, i.e. the scheduler 101 is optional. The NPU320 may implement a variety of AI networks for image analysis and image processing, and the user may configure the desired network through instructions input by the input device 102. The memory 103 stores at least two filter parameter set files usable by the hardware filter 104, and the user can configure the filter parameter set files used by the hardware filter 104 through instructions input by the input device 102.
One objective of the present application is to configure filtering parameters for the hardware filter in the external device 500, that is, complete the development of the hardware filter 104, and then directly provide the hardware filter 104 configured with the filtering parameters to other devices. For example, the configured hardware filter is connected to a computer to provide the computer with a hardware filtering function. The external device 500 and the main device 501 may be independent devices. For example, the host device 501 is a desktop computer, the external device 500 is a mobile device, and the external device 500 may be connected to the host device 501 through a USB interface or other interfaces. It is another object of the present application to enhance the image processing capabilities of a host device without hardware filters and NPUs. Currently, devices such as desktop computers and notebook computers do not have NPUs and hardware filters, and after the external device 500 is connected to these devices (the main device 501), these devices can enhance their image processing performance by using the external device 500. Therefore, the image processing performance of the devices can be enhanced without changing the hardware structure of the devices, and the cost is low.
In the embodiment of the present application, the user may configure not only the filter parameter set used by the image processing apparatus, but also the AI network used by the image processing apparatus. The foregoing embodiment describes an implementation of user selection of a filter parameter set, and the following describes an implementation of user configuration of an AI network employed by an image processing apparatus.
In an alternative implementation, the NPU320 obtains an AI network selection instruction input by the input device 102; and configuring the network adopted by the AI network selection instruction as a target AI network according to the AI network selection instruction.
The AI network selection instruction is used to select the target AI network of at least two AI networks to be selected. The at least two AI networks to be selected may include a resnet50 network, an ssd (single Shot multi box detector) network, a fast RCNN network, a resnet18 network, and the like. Fig. 6 is a schematic diagram of an AI network selection interface according to an embodiment of the present disclosure. As shown in fig. 6, the AI network selection interface includes selection interfaces of R AI networks; the circle in front of each AI network name is a selection interface of the AI network, and after a user selects the circle through a keyboard or a mouse, the circle becomes black (indicating that the corresponding AI network is selected). The AI network selection instruction is an operation by which a user selects an AI network through an input device. The AI network selection interface is an interface of an image processing program. The user may initiate an image processing program to select an AI network and configure a filter parameter set file. As shown in fig. 6, the fifth AI network is a target AI network selected by the user, and the image processing apparatus performs image processing using the fifth AI network.
In the implementation mode, the user selects the AI network adopted by the image processing device according to the needs of the user, the implementation is simple, and different image processing tasks can be met.
In an alternative implementation, the NPU320 obtains the AI network programming code input by the input device 102; and executing the AI network programming code to realize the target AI network.
In practical application, a user can write the AI network required by the user in the AI network programming interface. When the AI network desired by the user is not included in the AI networks preset by the image processing apparatus, the user can write the desired AI network.
In this implementation, the user can write code by himself to implement a desired AI network, instead of being limited to the AI network preset by the image processing apparatus, in order to develop a better-performing AI network.
Fig. 7 is a schematic structural diagram of a neural network processor according to an embodiment of the present invention, as shown in fig. 7, an NPU320 is mounted on a scheduler 101, a core portion of the NPU320 is an arithmetic circuit 703, and the controller 704 controls the arithmetic circuit 703 to extract matrix data in a memory and perform multiplication. Optionally, the processor 503 allocates tasks to the scheduler 101, and the scheduler 101 schedules the NPU320, the hardware filter 104, and other components to execute the corresponding tasks.
In some implementations, the arithmetic circuit 703 includes a plurality of processing units (PEs) therein. In some implementations, the operational circuit 703 is a two-dimensional systolic array. The arithmetic circuit 703 may also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry 703 is a general-purpose matrix processor.
For example, assume that there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit fetches the data corresponding to matrix B from the weight memory 702 and buffers it on each PE in the arithmetic circuit. The arithmetic circuit takes the matrix a data from the input memory 701 and performs matrix operation with the matrix B, and partial results or final results of the obtained matrix are stored in the accumulator 708.
The unified memory 706 is used to store input data as well as output data. The weight data is directly transferred to the weight Memory 702 through a Memory Access Controller (DMAC) 705. The input data is also carried through the DMAC into the unified memory 706.
A Bus Interface Unit (BIU) 710 for the interaction of the AXI Bus with the DMAC and an Instruction Fetch memory (Instruction Fetch Buffer) 709.
The bus interface unit 710 is used for the instruction fetch memory 709 to fetch instructions, and is also used for the memory unit access controller 705 to fetch the original data of the input matrix a or the weight matrix B.
The DMAC is primarily used to carry input data to the unified memory 706 or into the weight memory 702 or into the input memory 701.
The vector calculation unit 707 has a plurality of operation processing units, and further processes the output of the operation circuit such as vector multiplication, vector addition, exponential operation, logarithmic operation, magnitude comparison, and the like, if necessary. The method is mainly used for non-convolution/FC layer network calculation in the neural network, such as Pooling (Pooling), Batch Normalization (Batch Normalization), Local Response Normalization (Local Response Normalization) and the like.
In some implementations, the vector calculation unit 707 can store the processed output vector to the unified buffer 706. For example, the vector calculation unit 707 may apply a non-linear function to the output of the arithmetic circuit 703, such as a vector of accumulated values, to generate the activation value. In some implementations, the vector calculation unit 707 generates normalized values, combined values, or both. In some implementations, the vector of processed outputs can be used as activation inputs to the arithmetic circuitry 703, for example for use in subsequent layers in a neural network.
The controller 704 is connected to an instruction fetch buffer 709 for storing instructions used by the controller 704. The unified memory 706, the input memory 701, the weight memory 702, and the instruction fetch memory 709 are all On-Chip memories.
It should be understood that fig. 7 is only a schematic hardware structure diagram of an NPU provided in an embodiment of the present application, and the present application does not limit the structure of the NPU.
Fig. 8 is a flowchart illustrating an image processing method according to an embodiment of the present application, where the image processing method is applied to the image processing apparatuses in fig. 1, 3, and 5. As shown in fig. 8, the method may include:
801. the scheduler 101 obtains a parameter set selection instruction input by the input device 102.
The parameter set selection instruction is used to select N filter parameter set files among F filter parameter set files stored in the memory 103, where F is an integer greater than 1 and N is an integer greater than 0.
802. Image processor 105 configures a set of filter parameters in the reference filter parameter set file as filter parameters to be employed by the hardware filter to filter the reference image.
The reference filter parameter set file is any one of the N filter parameter set files, and the reference image is any one of the images in the image set stored in the memory 103.
803. The hardware filter 105 filters the reference image using the set of filter parameters.
In the embodiment of the application, a user can configure the filtering parameter file of the hardware filter according to the user, so that the hardware filter can filter the image by using the filtering parameter file configured by the user, the requirements of different scenes can be met, and the implementation is simple.
The image processing method in fig. 8 describes a process in which the image processing apparatus filters images in an image set using a filter parameter set file selected by a user. The following describes how to determine the parameter set file with the best filtering effect among the plurality of filtering parameter set files selected by the user.
Fig. 9 is a flowchart of a method for comparing filtering effects of multiple filter parameter set files according to an embodiment of the present application, where the method is applied to the image processing apparatuses in fig. 1, 3, and 5. As shown in fig. 9, the method may include:
901. the image processing device performs image preprocessing on the images in the image set.
The general flow of image pre-processing is as follows: uniform format, graying, filtering, sample enhancement. Unified format: i.e. the format of each image in the set of images is unified, e.g. all unified in jpg format. Graying: different image processing apparatuses have different requirements for image sets, some requiring input of a grayscale image and some requiring input of an RGB color image. The image processing apparatus needs to convert the images in the image set into satisfactory images, for example, into RGB color images, before processing the images. Filtering: the image filtering in the image preprocessing mainly comprises adjusting the image size and denoising and smoothing the noise in the zoomed image. Sample enhancement: the neural network is fed by data, and if the sample size of training data can be increased and massive data can be provided for training, the quality of an image analysis algorithm can be effectively improved. Common sample enhancement methods are: horizontal flipping of images, random cropping, translation transformation, color, illumination transformation, etc. Image pre-processing is a commonly used technique in the art and is not described in detail here.
In the process of image preprocessing, the image processing device filters the images in the image set by respectively adopting a plurality of filter parameter set files selected by a user. As shown in fig. 9, first, the image processing apparatus filters images in an image set using a filter parameter set file 1 and performs subsequent image processing; then, filtering the images in the image set by using the filtering parameter set file 2 and performing subsequent image processing; and finally, filtering the images in the image set by using the filtering parameter set file F and executing subsequent image processing. The implementation of the filtering operation can be seen in 301 to 306, which are not described in detail here.
902. The image processing device performs image analysis on each image in the image set to obtain an image analysis result of each image.
The image processing apparatus may perform image analysis, such as object detection and image classification, on the preprocessed image using an AI network. The image analysis result may be a label (classification) obtained by the image processing apparatus recognizing the images in the image set, or may be a bounding box obtained by the image processing apparatus detecting an object in the images in the image set. The implementation is the same as 309 in fig. 3.
903. And the image processing device compares the image analysis result of each image with the labeling information to obtain a comparison result of each image.
The way of comparing the image analysis result of each image with the annotation information by the image processing apparatus can be referred to 311 in fig. 3.
904. The image processing device determines a target filtering parameter set file with the best filtering effect in the N filtering parameter set files; the target filter parameter set file is configured as the filter parameter set file it employs.
In practical applications, the image processing apparatus may respectively test an effect of performing image analysis on images in an image set using each filter parameter set file selected by a user, and then select a filter parameter set file with a best filter effect as a filter parameter set file to be used.
In the embodiment of the present application, the image processing apparatus may determine the filter parameter set file with the best filtering effect among the plurality of filter parameter set files selected by the user, and may further improve the image analysis performance of the image processing apparatus.
The technical scheme of the application is introduced in the foregoing embodiment, and the following describes beneficial effects achieved by adopting the technical scheme of the application.
An image processing scheme (first scheme) provided by the embodiment of the application is as follows: the image processing device configures an optimal filter parameter set for 1000 test images (samples) of a resnet50 network, inputs the results into a resnet50 network after filtering processing of a hardware filter, and compares the obtained output (image analysis result) with a ground route to obtain top1 to top 5.
The image processing scheme (second scheme) used for comparison is as follows: 1000 test images of the resnet50 network are input into the balloon, the images are input into the resnet50 network after filtering processing, and the obtained output (image analysis result) is compared with ground route to obtain top 1-top 5. The PIL (Python Imaging library) is a commonly used image processing library of Python, and the pink is a friendly Fork of the PIL, which provides wide file format support and strong image processing capability, and mainly includes image storage, image display, format conversion, basic image processing operation and the like.
TABLE 1
Table 1 is a comparison table of image recognition rates. As can be seen from table 1, the performance of the first scheme is better than the performance of the second scheme. For top1, the first scheme exceeds the second scheme (pilot) by 0.3%. Top1 is taken as a main evaluation index in five statistical results of top 1-top 5.
Another image processing scheme (third scheme) provided in the embodiment of the present application is as follows: the image processing device configures an optimal filtering parameter set for 2000 test images of the lane line identification network, inputs the images into the lane line identification network after filtering processing of a hardware filter, and compares the obtained output with a ground route to obtain the accuracy rate and the recall rate.
The image processing scheme used for comparison (fourth scheme) is as follows: inputting 2000 test images of the lane line identification network into OpenCV, inputting the test images into the lane line identification network after filtering processing, and comparing the obtained output with a ground route to obtain the accuracy rate and the recall rate.
Precision ratio (precision) refers to the number of samples identified/the number of samples identified. Recall (recall) refers to identifying the correct number of samples/the correct number in all samples.
TABLE 2
| Image processing scheme | Overall rate of accuracy | Overall recall rate | Current rate of accuracy | Current recall rate |
| Third scheme | 0.9446 | 0.6682 | 0.9318 | 0.9058 |
| Fourth embodiment | 0.9536 | 0.6467 | 0.9428 | 0.9061 |
Table 2 is an accuracy and recall table for image analysis. The overall accuracy rate in table 2 refers to the accuracy rate of identifying each lane line in the image, and the current accuracy rate refers to the accuracy rate of identifying the lane where the vehicle is currently driving; the overall recall rate refers to the recall rate of each lane line in the identification image, and the current recall rate refers to the recall rate of the lane where the identification vehicle is currently running. As can be seen from table 2, the performance of the third scheme is superior to that of the fourth scheme. The third scheme exceeds the fourth scheme by 1.1 percent in the accuracy rate of recognizing the current lane and exceeds the fourth scheme by 0.3 percent in the recall rate of recognizing the current lane.
According to the technical scheme, the accuracy of image recognition can be effectively improved.
The above embodiments may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, the above-described embodiments may be implemented in whole or in part in the form of a computer program product.
While the invention has been described with reference to specific embodiments, the scope of the invention is not limited thereto, and those skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the invention. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (14)
- An image processing apparatus characterized by comprising:a scheduler for obtaining a parameter set selection instruction input by an input device, the parameter set selection instruction being an instruction input by a user through the input device and being used for instructing to select N filtering parameter set files from F filtering parameter set files stored in a memory, where F is a positive integer greater than 1 and N is an integer greater than 0;a hardware filter coupled to the scheduler for filtering the reference picture using a set of filtering parameters included in one of the N filtering parameter set files.
- The image processing apparatus according to claim 1,the image processor is respectively coupled with the scheduler and the hardware filter and used for analyzing the filtering parameter set file to obtain a reference filtering parameter set; and determining the group of filter parameters in the reference filter parameter set as parameters to be adopted by the hardware filter for filtering the reference image by using the reference filter parameter set according to a first scaling coefficient of the reference image in the horizontal direction and a second scaling coefficient of the reference image in the vertical direction.
- The image processing apparatus according to claim 2,the image processor is specifically configured to select a set of filter parameters in a horizontal direction from the reference filter parameter set according to the first scaling factor, and to select a set of filter parameters in a vertical direction from the reference filter parameter set according to the second scaling factor; taking the set of filtering parameters in the horizontal direction and the set of filtering parameters in the vertical direction as the set of filtering parameters; configuring the set of filtering parameters to registers of the hardware filter;the hardware filter is specifically configured to read the set of filtering parameters in the register, and filter the reference image by using the set of filtering parameters.
- The image processing apparatus according to claim 3,the hardware filter is further configured to send the image after the reference image is filtered to the image processor;the image processor is further configured to send the reference image filtered image to the scheduler; the image processing apparatus further includes:a neural network processor NPU coupled to the scheduler;the scheduler is further configured to input the reference picture filtered picture to the NPU;and the NPU is used for carrying out image analysis on the image after the reference image is filtered by utilizing a target AI network and sending an obtained image analysis result to the scheduler.
- The image processing apparatus according to claim 4, characterized in that the image processing apparatus further comprises:a result comparator coupled to the scheduler;the scheduler is further configured to send the annotation information of the reference image and the image analysis result to the result comparator;and the result comparator is used for comparing the image analysis result with the labeling information of the reference image to obtain a comparison result of the parameter image.
- The image processing apparatus according to claim 5,the result comparator is further configured to send the comparison result of the reference image and the comparison result of each image in the image set except the reference image to the scheduler; the mode of obtaining the comparison result of each image except the reference image in the image set by the result comparator is the same as the mode of obtaining the comparison result of the reference image, and the image set comprises at least two images;the scheduler is further configured to determine a filtering effect of the reference filtering parameter set according to a comparison result of each image in the image set; respectively determining the filtering effects of the filtering parameter sets except the reference filtering parameter set in the N filtering parameter sets, wherein the N filtering parameter sets are N parameter sets obtained by analyzing the N filtering parameter set files by the image processor; configuring the target filter parameter set with the best filter effect in the N filter parameter sets as the filter parameter set of the hardware filter; wherein the scheduler determines a filtering effect of the filtering parameter sets of the N filtering parameter sets other than the reference filtering parameter set in the same manner as the filtering effect of the reference filtering parameter set.
- The image processing apparatus according to any one of claims 4 to 6,the NPU is further configured to obtain an AI network selection instruction input by the input device, where the AI network selection instruction is used to select the target AI network of at least two AI networks to be selected; configuring a network adopted by the NPU as the target AI network according to the AI network selection instruction;alternatively, the first and second electrodes may be,the NPU is also used for obtaining AI network programming codes input by the input equipment; executing the AI network programming code to implement the target AI network.
- An image processing method applied to an image processing apparatus, wherein the image processing apparatus includes a scheduler and a hardware filter, the method comprising:the scheduler obtains a parameter set selection instruction input by an input device, wherein the parameter set selection instruction is an instruction input by a user through the input device and is used for instructing to select N filtering parameter set files from F filtering parameter set files stored in a memory, wherein F is a positive integer larger than 1, and N is an integer larger than 0;the hardware filter filters the reference image using a set of filter parameters included in one of the N filter parameter set files.
- The method of claim 8, wherein the image processor further comprises an image processor coupled to the scheduler and the hardware filter, respectively; before the hardware filter filters the reference image by using a set of filtering parameters contained in one of the N filtering parameter set files, the method further includes:the image processor analyzes the filtering parameter set file to obtain a reference filtering parameter set; and determining the group of filter parameters in the reference filter parameter set as parameters to be adopted by the hardware filter for filtering the reference image by using the reference filter parameter set according to a first scaling coefficient of the reference image in the horizontal direction and a second scaling coefficient of the reference image in the vertical direction.
- The method of claim 9, wherein determining the set of filter parameters in the reference filter parameter set as parameters to be used by the hardware filter to filter the reference image using the reference filter parameter set according to a first scaling coefficient of the reference image in a horizontal direction and a second scaling coefficient of the reference image in a vertical direction comprises:selecting, by the image processor, a set of horizontal-direction filter parameters from the reference filter parameter set according to the first scaling factor and a set of vertical-direction filter parameters from the reference filter parameter set according to the second scaling factor; taking the set of filtering parameters in the horizontal direction and the set of filtering parameters in the vertical direction as the set of filtering parameters; configuring the set of filtering parameters to registers of the hardware filter;the hardware filter filtering the reference image by using a set of filtering parameters included in one of the N filtering parameter set files comprises:the hardware filter reads the set of filtering parameters in the register and filters the reference image using the set of filtering parameters.
- The method of claim 10, wherein the image processing apparatus further comprises a neural network processor NPU coupled with the scheduler; after the hardware filter filters the reference image by using a set of filtering parameters contained in one of the N filtering parameter set files, the method further includes:the hardware filter sends the image after the reference image is filtered to the image processor;the image processor sends the reference image filtered image to the scheduler;the scheduler inputs the reference picture filtered picture to the NPU;and the NPU utilizes a target AI network to perform image analysis on the image after the reference image filtering, and sends the obtained image analysis result to the scheduler.
- The method of claim 11, wherein the image processing apparatus further comprises a result comparator coupled to the scheduler; the NPU performs image analysis on the image after the reference image filtering by using a target AI network, and after sending an obtained image analysis result to the scheduler, the method further includes:the scheduler sends the labeling information of the reference image and the image analysis result to the result comparator;and the result comparer compares the image analysis result with the labeling information of the reference image to obtain a comparison result of the parameter image.
- The method according to claim 12, wherein after the comparing of the image analysis result and the labeled information of the reference image by the result comparing device to obtain the comparison result of the parameter image, the method further comprises:the result comparer sends the comparison result of the reference image and the comparison result of each image except the reference image in the image set to the scheduler; the mode of obtaining the comparison result of each image except the reference image in the image set by the result comparator is the same as the mode of obtaining the comparison result of the reference image, and the image set comprises at least two images;the scheduler determines the filtering effect of the reference filtering parameter set according to the comparison result of each image in the image set; respectively determining the filtering effects of the filtering parameter sets except the reference filtering parameter set in the N filtering parameter sets, wherein the N filtering parameter sets are N parameter sets obtained by analyzing the N filtering parameter set files by the image processor; configuring the target filter parameter set with the best filter effect in the N filter parameter sets as the filter parameter set of the hardware filter; wherein the scheduler determines a filtering effect of the filtering parameter sets other than the reference filtering parameter set among the N filtering parameter sets in the same manner as the filtering effect of the reference filtering parameter set.
- The method according to any one of claims 11 to 13, wherein the NPU performs image analysis on the reference image filtered image by using a target AI network, and before sending a result of the image analysis to the scheduler, the method further comprises:the NPU obtains an AI network selection instruction input by the input equipment, wherein the AI network selection instruction is used for selecting the target AI network in at least two AI networks to be selected; configuring a network adopted by the NPU as the target AI network according to the AI network selection instruction;alternatively, the first and second electrodes may be,the NPU obtains an AI network programming code input by the input equipment; executing the AI network programming code to implement the target AI network.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2018/123160 WO2020132825A1 (en) | 2018-12-24 | 2018-12-24 | Image processing apparatus and image processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112771566A true CN112771566A (en) | 2021-05-07 |
Family
ID=71126441
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201880098076.9A Pending CN112771566A (en) | 2018-12-24 | 2018-12-24 | Image processing apparatus, image processing method, and program |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112771566A (en) |
| WO (1) | WO2020132825A1 (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH07121702A (en) * | 1993-10-22 | 1995-05-12 | Canon Inc | Image processor |
| DE10202602A1 (en) * | 2001-09-07 | 2003-03-27 | Bosch Gmbh Robert | Method and device for transforming an object image |
| KR20050039422A (en) * | 2003-10-25 | 2005-04-29 | 학교법인 인하학원 | Reconfigurable, real-time hardware filter |
| US20070242898A1 (en) * | 2006-04-18 | 2007-10-18 | Pioneer Corporation | Image processing apparatus, image processing method, and image processing program |
| CN103208094A (en) * | 2012-01-12 | 2013-07-17 | 索尼公司 | Method and system for applying filter to image |
| US20150046675A1 (en) * | 2013-08-08 | 2015-02-12 | Linear Algebra Technologies Limited | Apparatus, systems, and methods for low power computational imaging |
| CN108897777A (en) * | 2018-06-01 | 2018-11-27 | 深圳市商汤科技有限公司 | Target object method for tracing and device, electronic equipment and storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101971632B (en) * | 2008-01-08 | 2013-10-16 | 艾利森电话股份有限公司 | Adaptive filtering |
| US8532429B2 (en) * | 2010-09-28 | 2013-09-10 | Sharp Laboratories Of America, Inc. | Methods and systems for noise reduction and image enhancement involving selection of noise-control parameter |
| US8538193B2 (en) * | 2010-09-28 | 2013-09-17 | Sharp Laboratories Of America, Inc. | Methods and systems for image enhancement and estimation of compression noise |
| CN109003252A (en) * | 2017-05-30 | 2018-12-14 | 正凯人工智能私人有限公司 | Image processing method and system |
-
2018
- 2018-12-24 WO PCT/CN2018/123160 patent/WO2020132825A1/en active Application Filing
- 2018-12-24 CN CN201880098076.9A patent/CN112771566A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH07121702A (en) * | 1993-10-22 | 1995-05-12 | Canon Inc | Image processor |
| DE10202602A1 (en) * | 2001-09-07 | 2003-03-27 | Bosch Gmbh Robert | Method and device for transforming an object image |
| KR20050039422A (en) * | 2003-10-25 | 2005-04-29 | 학교법인 인하학원 | Reconfigurable, real-time hardware filter |
| US20070242898A1 (en) * | 2006-04-18 | 2007-10-18 | Pioneer Corporation | Image processing apparatus, image processing method, and image processing program |
| CN103208094A (en) * | 2012-01-12 | 2013-07-17 | 索尼公司 | Method and system for applying filter to image |
| US20150046675A1 (en) * | 2013-08-08 | 2015-02-12 | Linear Algebra Technologies Limited | Apparatus, systems, and methods for low power computational imaging |
| CN108897777A (en) * | 2018-06-01 | 2018-11-27 | 深圳市商汤科技有限公司 | Target object method for tracing and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2020132825A1 (en) | 2020-07-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10936919B2 (en) | Method and apparatus for detecting human face | |
| CN107545262B (en) | Method and device for detecting text in natural scene image | |
| US10769496B2 (en) | Logo detection | |
| US10860837B2 (en) | Deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition | |
| US9349076B1 (en) | Template-based target object detection in an image | |
| CN107622240B (en) | Face detection method and device | |
| US10572072B2 (en) | Depth-based touch detection | |
| CN110349147B (en) | Model training method, fundus macular region lesion recognition method, device and equipment | |
| JP7125562B2 (en) | Target tracking method, computer program, and electronic device | |
| EP3701488A1 (en) | System and method for image processing using deep neural networks | |
| TW202139183A (en) | Method of detecting object based on artificial intelligence, device, equipment and computer-readable storage medium | |
| US20160098608A1 (en) | System and method for scene text recognition | |
| CN108229658B (en) | Method and device for realizing object detector based on limited samples | |
| US11468296B2 (en) | Relative position encoding based networks for action recognition | |
| US20220198836A1 (en) | Gesture recognition method, electronic device, computer-readable storage medium, and chip | |
| CN113807399A (en) | Neural network training method, neural network detection method and neural network detection device | |
| JP2023501820A (en) | Face parsing methods and related devices | |
| JP2017527013A (en) | Adaptive characterization as a service | |
| US20210097377A1 (en) | Method and apparatus for image recognition | |
| CN112966687B (en) | Image segmentation model training method and device and communication equipment | |
| CN117058421A (en) | Multi-head model-based image detection key point method, system, platform and medium | |
| WO2016149937A1 (en) | Neural network classification through decomposition | |
| CN112771566A (en) | Image processing apparatus, image processing method, and program | |
| CN114360053A (en) | Action recognition method, terminal and storage medium | |
| US20240169702A1 (en) | Image recognition edge device and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |