WO2020132825A1

WO2020132825A1 - Image processing apparatus and image processing method

Info

Publication number: WO2020132825A1
Application number: PCT/CN2018/123160
Authority: WO
Inventors: 文凯; 杨振汉; 王雷; 张浩龙; 李炜
Original assignee: 华为技术有限公司
Priority date: 2018-12-24
Filing date: 2018-12-24
Publication date: 2020-07-02
Also published as: CN112771566A

Abstract

Disclosed are an image processing apparatus and an image processing method applied in the field of artificial intelligence (AI). The image processing apparatus comprises: a scheduler for obtaining a parameter set selection instruction input by an input device, wherein the parameter set selection instruction is an instruction input by a user by means of the input device and is used to instruct the selection of N filtering parameter set files from F filtering parameter set files stored in a memory, F being a positive integer greater than 1 and N being an integer greater than 0; and a hardware filter, coupled to the scheduler, for filtering a reference image by means of a set of filtering parameters included in one filtering parameter set file among the N filtering parameter set files. In the embodiments of the present application, a user can configure, at will, filtering parameter files of a hardware filter, so that the requirements of different scenarios can be met, and the implementation is simple.

Description

Image processing device and image processing method

Technical field

The present application relates to the field of electronic technology, and in particular, to an image processing device and an image processing method.

Background technique

Artificial Intelligence (AI) is a theory, method, technology, and application system that uses digital computers or digital computer-controlled machines to simulate, extend, and expand human intelligence, sense the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machine has the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc. With the development of AI technology, AI networks, such as Neural Networks (NN), Recurrent Neural Networks, Convolutional Neural Networks, and Deep Neural Networks, have become more and more widely used in various fields. In actual life, AI networks have been widely used in the fields of image processing, image recognition, and image classification.

Before the AI network is used for image analysis or image processing, the image to be processed usually needs to be preprocessed. The main purpose of image preprocessing is to eliminate irrelevant information in the image, restore useful real information, enhance the detectability of relevant information, and simplify the data to the greatest extent, thereby improving AI network feature extraction, image segmentation, image matching and Reliability and accuracy of image processing operations such as image recognition. Image filtering is an indispensable operation in image preprocessing. It is to suppress image noise while preserving the details of the image as much as possible. Its processing effect will directly affect the effectiveness and reliability of subsequent image processing and analysis. .

Image filtering in image preprocessing mainly includes adjusting the image size, and performing denoising and smoothing on the noise in the scaled image. For example, image denoising can be performed using median hardware filters, Gaussian hardware filters, and so on. Resizing is mainly based on the size requirements of the AI network for the input image, and the image to be processed of the AI network is cropped and scaled to adapt to the standard size required by the specific AI network. Currently, hardware filters are usually used to implement image filtering during image preprocessing, such as common RC filtering and LC filtering. Since the filtering parameters in the hardware filter are pre-configured, each hardware filter can meet the image filtering requirements of a limited number of image filtering scenarios at most, and no good image filtering effect can be obtained in any image filtering scenario. Therefore, it is necessary to solve the problem that the filtering parameters in the hardware filter cannot be adjusted, so as to obtain a better image filtering effect in any image filtering scene.

Summary of the invention

The embodiments of the present application provide an image processing device and an image processing method. The hardware filter uses a user-configured filter parameter to filter an image, which can solve the problem that the hardware filter can only use a fixed set of filter parameters for filtering.

In a first aspect, an embodiment of the present application provides an image processing apparatus, including:

The scheduler is used to obtain a parameter set selection instruction input by the input device, the parameter set selection instruction is an instruction input by a user through the input device and is used to instruct to select N filters from the F filter parameter set files stored in the memory Parameter set file, where F is a positive integer greater than 1 and N is an integer greater than 0;

A hardware filter coupled to the scheduler is used to filter the reference image using a set of filter parameters included in one filter parameter set file among the N filter parameter set files.

In the embodiment of the present application, the user configures the filter parameter file in the hardware filter by himself, so that the hardware filter uses the user-configured filter parameter file to filter the image, which can meet the needs of different scenarios and is simple to implement.

In an optional implementation, an image processor respectively coupled to the scheduler and the hardware filter is used to parse the one filter parameter set file to obtain a reference filter parameter set; according to the reference image in A first scaling factor in the horizontal direction and a second scaling factor in the vertical direction, determining that the set of filtering parameters in the reference filtering parameter set is the hardware filter using the reference filtering parameter set for the reference image Filter the parameters to be used.

The reference filtering parameter set file includes at least two sets of filtering parameters.

In this implementation, the image processor determines the parameters to be used by the hardware filter to filter the reference image using the reference filtering parameter set according to the first scaling factor in the horizontal direction and the second scaling factor in the vertical direction of the reference image ; So that the size of the filtered image of the reference image meets the requirements.

In an optional implementation manner, the scheduler is further configured to send N file names corresponding to the N filter parameter set files to the image processor according to the parameter set selection instruction, the N file names correspond to the N filter parameter set files in one-to-one correspondence;

The image processor is further configured to obtain the reference filter parameter set file from the memory according to a reference file name, the reference file name is the file name of the reference filter parameter set file and is included in the N files name.

Optionally, the scheduler is further configured to read the reference image from the memory and send it to the image processor.

In this implementation, the scheduler sends the reference file name to the image processor, so that the image processor uses the reference file name to read the reference filtering parameter set from the memory; the implementation is simple and consumes little signaling resources.

In an optional implementation manner, the image processor is specifically configured to select a set of horizontal filtering parameters from the reference filtering parameter set according to the first scaling factor, and select The reference filtering parameter set collectively selects a group of vertical filtering parameters; the group of horizontal filtering parameters and the group of vertical filtering parameters are used as the group of filtering parameters; the group of filtering parameters A register configured to the hardware filter;

The hardware filter is specifically used to read the set of filter parameters in the register and filter the reference image using the set of filter parameters.

The reference filtering parameter set file includes at least two sets of filtering parameters in the horizontal direction and at least two sets of filtering parameters in the vertical direction. The image processor may use the mapping relationship between the horizontal scaling factor and the horizontal filtering parameter to determine a set of horizontal filtering parameters corresponding to any horizontal scaling factor; the vertical scaling factor and The mapping relationship of the filter parameters in the vertical direction determines a set of filter parameters in the vertical direction corresponding to any vertical scaling factor. Optionally, the horizontal scaling factor of the image corresponds to H parameter intervals, and the reference filtering parameter set file includes H sets of horizontal filtering parameters, and the H sets of horizontal filtering parameters are in one-to-one relationship with the H parameter intervals Correspondingly, H is an integer greater than 1. The image processor may select a set of horizontal filtering parameters corresponding to the parameter interval where the first scaling factor is located from the H sets of horizontal filtering parameters. Optionally, the vertical scaling factor of the image corresponds to G parameter intervals, and the reference filtering parameter set file includes G sets of vertical filtering parameters, and the G sets of vertical filtering parameters are in one-to-one relationship with the G parameter intervals Correspondingly, G is an integer greater than 1. The image processor may select a group of vertical filtering parameters corresponding to the parameter interval where the second scaling factor is located from the G group of vertical filtering parameters.

In this implementation, the image processor may configure the filtering parameters for the hardware filter according to the horizontal and vertical scaling factors of the reference image, so that the size of the filtered image of the reference image meets the requirements.

In an optional implementation manner, the hardware filter is further used to send the filtered image of the reference image to the image processor;

The image processor is further configured to send the filtered image of the reference image to the scheduler; the image processing device further includes:

A neural network processor NPU coupled with the scheduler;

The scheduler is further configured to input the filtered image of the reference image to the NPU;

The NPU is used to perform image analysis on the filtered image of the reference image using a target AI network, and send the obtained image analysis result to the scheduler.

In this implementation, the AI network is used to perform image analysis on the filtered image of the reference image, and the image processing efficiency is high.

In an optional implementation manner, the image processing apparatus further includes:

A result comparator coupled with the scheduler;

The scheduler is also used to send the annotation information of the reference image and the image analysis result to the result comparator;

The result comparator is used to compare the image analysis result with the reference information of the reference image to obtain the comparison result of the parameter image.

In this implementation, the result comparison device is used to compare the image analysis result with the image annotation information, so as to determine the filtering effect of the hardware filter.

In an optional implementation manner, the result comparator is further configured to send the comparison result of the reference image and the comparison result of each image in the image set except the reference image to The scheduler; wherein, the result aligner obtains the comparison result of each image in the image set except the reference image in the same way as the reference result. The image set includes at least two images;

The scheduler is further configured to determine the filtering effect of the reference filtering parameter set according to the comparison result of each image in the image set; separately determine the filtering other than the reference filtering parameter set in the N filtering parameter sets The filtering effect of the parameter set, where the N filtering parameter sets are N parameter sets obtained by the image processor parsing the N filtering parameter set files; the N filtering parameter set is the target filter with the best filtering effect The parameter set is configured as a filtering parameter set of the hardware filter; wherein, the manner in which the scheduler determines the filtering effect of the filtering parameter set other than the reference filtering parameter set in the N filtering parameter sets and determines the The filtering effect of the reference filtering parameter set is the same.

In this implementation, the scheduler configures the target filtering parameter set with the best filtering effect in multiple filtering parameter sets as the filtering parameter set of the hardware filter, which can improve the performance of image processing.

In an optional implementation manner, the NPU is further used to obtain an AI network selection instruction input by the input device, and the AI network selection instruction is used to select the target in at least two AI networks to be selected AI network; configure the network adopted by the NPU as the target AI network according to the AI network selection instruction;

or,

The NPU is also used to obtain the AI network programming code input by the input device; execute the AI network programming code to implement the target AI network.

In this implementation manner, the user configures the AI network in the image processing apparatus to facilitate processing different types of images or performing different image processing operations, which can meet different needs.

In a second aspect, an embodiment of the present application provides an image processing method, which is applied to an image processing apparatus. The image processing apparatus includes a scheduler and a hardware filter. The method includes:

The scheduler obtains a parameter set selection instruction input by the input device, where the parameter set selection instruction is an instruction input by the user through the input device and is used to instruct the selection of N filter parameters from the F filter parameter set files stored in the memory Set file, where F is a positive integer greater than 1 and N is an integer greater than 0;

The hardware filter uses a set of filtering parameters included in one filtering parameter set file among the N filtering parameter set files to filter the reference image.

In the embodiment of the present application, the user can configure the filter parameter file of the hardware filter according to oneself, so that the hardware filter uses the user-configured filter parameter file to filter the image, which can meet the needs of different scenarios and is simple to implement.

In an optional implementation manner, the image processor further includes an image processor respectively coupled to the scheduler and the hardware filter; the hardware filter uses one of the N filtering parameter set files Before filtering the reference image with a set of filtering parameters included in the filtering parameter set file, the method further includes:

The image processor parses the one filtering parameter set file to obtain a reference filtering parameter set; and determines the reference filtering parameter set according to the first scaling factor of the reference image in the horizontal direction and the second scaling factor in the vertical direction The set of filtering parameters is a parameter to be used by the hardware filter to filter the reference image by using the reference filtering parameter set.

In an optional implementation manner, before the image processor configures a set of filter parameters in the reference filter parameter set file as the filter parameters to be used by the hardware filter to filter the reference image, the method further includes :

The scheduler sends N file names corresponding to the N filter parameter set files to the image processor according to the parameter set selection instruction, the N file names and the N filter parameter set files One-to-one correspondence

The image processor obtains the reference filter parameter set file from the first memory according to a reference file name, where the reference file name is the file name of the reference filter parameter set file and is included in the N file names.

In an optional implementation manner, the determining of the set of filtering parameters in the reference filtering parameter set is based on the first scaling factor in the horizontal direction and the second scaling factor in the vertical direction of the reference image The hardware filter uses the reference filtering parameter set to filter the reference image, and the parameters to be adopted include:

The image processor selects a set of horizontal filtering parameters from the reference filtering parameter set according to the first scaling factor, and selects a set of vertical filtering from the reference filtering parameter set according to the second scaling factor Parameters; the set of horizontal filtering parameters and the set of vertical filtering parameters as the set of filtering parameters; configuring the set of filtering parameters to the register of the hardware filter;

The hardware filter filtering the reference image using a set of filtering parameters included in one filtering parameter set file among the N filtering parameter set files includes:

The hardware filter reads the set of filter parameters in the register and uses the set of filter parameters to filter the reference image.

In an optional implementation manner, the image processing device further includes a neural network processor NPU coupled to the scheduler; the hardware filter utilizes a filtering parameter set file among the N filtering parameter set files After filtering the reference image with the included set of filtering parameters, the method further includes:

The hardware filter sends the filtered image of the reference image to the image processor;

The image processor sends the filtered image of the reference image to the scheduler;

The scheduler inputs the filtered image of the reference image to the NPU;

The NPU uses the target AI network to perform image analysis on the filtered image of the reference image, and sends the obtained image analysis result to the scheduler.

In an optional implementation manner, the image processing device further includes a result comparator coupled with the scheduler; the NPU performs image analysis on the filtered image of the reference image by using a target AI network, and will obtain After the image analysis result of is sent to the scheduler, the method further includes:

The scheduler sends the annotation information of the reference image and the image analysis result to the result comparator;

The result comparison device compares the image analysis result with the reference information of the reference image to obtain a comparison result of the parameter image.

In an optional implementation manner, the result comparator compares the image analysis result with the reference information of the reference image, and after obtaining the comparison result of the parameter image, the method further includes:

The result comparator sends the comparison result of the reference image and the comparison result of each image in the image set except the reference image to the scheduler; wherein, the result comparator obtains the result The way of comparing the results of each image in the image set except the reference image is the same as the way of obtaining the results of the comparison of the reference image, and the image set includes at least two images;

The scheduler determines the filtering effect of the reference filtering parameter set according to the comparison result of each image in the image set; separately determines the filtering of the filtering parameter sets other than the reference filtering parameter set in the N filtering parameter sets Effect, the N filter parameter sets are N parameter sets obtained by the image processor parsing the N filter parameter set files; the target filter parameter set with the best filtering effect in the N filter parameter sets is configured as A filtering parameter set of the hardware filter; wherein, the manner in which the scheduler determines a filtering effect of a filtering parameter set other than the reference filtering parameter set in the N filtering parameter sets and determines the reference filtering parameter set The filtering effect is the same.

In an optional implementation manner, before the NPU uses the target AI network to perform image analysis on the filtered image of the reference image, and before sending the obtained image analysis result to the scheduler, the method further includes:

The NPU obtains an AI network selection instruction input by the input device, and the AI network selection instruction is used to select the target AI network in at least two AI networks to be selected; according to the AI network selection instruction, the The network configuration adopted by the NPU is the target AI network;

or,

The NPU obtains the AI network programming code input by the input device; executes the AI network programming code to implement the target AI network.

BRIEF DESCRIPTION

1 is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application;

2 is a schematic diagram of a filtering parameter set selection interface provided by an embodiment of the present application;

3 is a schematic structural diagram of another image processing device provided by the present application;

4 is a schematic diagram of calculating the overlapping degree of two bounding boxes provided by an embodiment of the present application;

5 is a schematic structural diagram of another image processing device provided by an embodiment of the present application;

6 is a schematic diagram of an AI network selection interface provided by an embodiment of this application;

7 is a schematic structural diagram of a neural network processor provided by an embodiment of the present application;

8 is a schematic flowchart of an image processing method according to an embodiment of the present application;

9 is a flowchart of a method for comparing filtering effects of multiple filtering parameter set files according to an embodiment of the present application.

detailed description

In order to enable those skilled in the art to better understand the solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only It is a part of the embodiments of this application, but not all the embodiments.

The terms "first", "second", and "third" in the description examples and claims of the present application and the above drawings are used to distinguish similar objects, and are not required to describe a specific order or The order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. A method, system, product, or device need not be limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or equipment.

FIG. 1 is a schematic structural diagram of another image processing apparatus provided by an embodiment of the present application. As shown in FIG. 1, the image processing apparatus 10 includes:

The scheduler 101 is used to obtain a parameter set selection instruction input by the input device 102, where the parameter set selection instruction is an instruction input by a user through the input device 102 and is used to instruct to select N from F filter parameter set files stored in the memory 103 Filter parameter set files, where F is a positive integer greater than 1 and N is an integer greater than 0;

A hardware filter 104 coupled to the scheduler 101 is used to filter the reference image by using a set of filter parameters included in one filter parameter set file among the N filter parameter set files.

The input device 102 may be a keyboard, a mouse, a touch screen, or the like. The F filter parameter set files stored in the memory 103 may be pre-configured when the image processing apparatus is shipped from the factory, or may be configured by the user. The following describes a method of generating a filter parameter set file: first formulate parameter strategies, such as iterative step size (Kaiser step size and Gauss step size), filter parameter interval, etc., so as to traverse the filter parameter interval as quickly and comprehensively as possible; then According to the formulated parameter strategy, the iteration step and filter parameter interval are used as input parameters, and the script is used to generate a script to generate multiple filter parameter set files. The number of filter parameter set files generated by different parameter strategies is different. Users or R&D personnel can formulate different parameter strategies according to their needs to generate the required filter parameter files. For example, more than 200 filtering parameter set files are generated, and each filtering parameter set file has more than 1000 sets of filtering parameters in the horizontal direction and more than 1000 sets of filtering parameters in the vertical direction. Optionally, the image processing apparatus exposes the file name of each filter parameter set file to the user, and the user selects the file name of the filter parameter set through the input device 102 to configure the corresponding filter parameter set. Since the hardware filter is used for image filtering, a set of filter parameters in any filter parameter set file is a set of filter parameters in the spatial domain of the hardware filter.

In the embodiment of the present application, the user can configure the filter parameter file of the hardware filter according to oneself, so that the hardware filter uses the user-configured filter parameter file to filter the image, which can meet the needs of different scenarios and be simple to implement.

In an optional implementation manner, the image processing apparatus further includes an image processor 105 coupled to the scheduler 101 and the hardware filter 104, respectively:

The image processor 105 is configured to parse the one filter parameter set file to obtain a reference filter parameter set; determine the reference filter according to the first scaling factor of the reference image in the horizontal direction and the second scaling factor in the vertical direction The set of filtering parameters in the parameter set is a parameter to be used by the hardware filter to filter the reference image by using the reference filtering parameter set.

Optionally, the image processor 105 is specifically configured to select a set of horizontal filtering parameters from the reference filtering parameter set according to the first scaling factor, and from the reference filtering parameter set according to the second scaling factor Select a set of vertical filtering parameters; use the set of horizontal filtering parameters and the set of vertical filtering parameters as the set of filtering parameters; configure the set of filtering parameters to the hardware filter 101 Register of

The hardware filter 104 is specifically used to read the set of filter parameters in the register and filter the reference image using the set of filter parameters.

The first scaling factor of the reference image in the horizontal direction refers to a scaling factor corresponding to the standard size of the input image required by the image processing device adjusted in the horizontal direction corresponding to the reference image. The second vertical scaling factor of the reference image refers to a scaling factor corresponding to the vertical size of the reference image adjusted to the standard size of the input image required by the image processing device. The above reference filtering parameter set file includes at least two sets of filtering parameters in the horizontal direction and at least two sets of filtering parameters in the vertical direction. The above image processor can use the mapping relationship between the horizontal scaling factor and the horizontal filtering parameter to determine a set of horizontal filtering parameters corresponding to any horizontal scaling factor; the vertical scaling factor and the vertical can be used The mapping relationship of the filter parameters in the direction determines a set of filter parameters in the vertical direction corresponding to any vertical scaling factor. Optionally, the horizontal scaling factor of the image corresponds to H parameter intervals. The reference filter parameter set file includes H sets of horizontal filtering parameters. The H sets of horizontal filtering parameters correspond to the H parameter intervals in one-to-one correspondence. It is an integer greater than 1. The image processor may select a group of horizontal filtering parameters corresponding to the parameter interval where the first scaling factor is located from the H group of horizontal filtering parameters. Optionally, the scaling factor in the vertical direction of the image corresponds to G parameter intervals. The above reference filter parameter set file includes G sets of vertical filtering parameters, and the G sets of vertical filtering parameters correspond one-to-one to the G parameter intervals. G It is an integer greater than 1. The image processor may select a group of vertical filter parameters corresponding to the parameter interval in which the second scaling factor is located from the G group of vertical filter parameters.

The reference image is an image currently to be processed by the data processing device and is an image in the image set. The memory 103 stores the image set. Each image in the image set is annotated, that is, each image has an annotation information (label or true value). It can be understood that each image in the above image set is a test image, which is used to test the image processing effect of the image processing device. The label or true value (annotated information) of the image is called Ground Truth. The Ground Truth of each image is used for subsequent comparison with the inference value (image analysis result) output by the image through the AI network processing.

In practical applications, the image processor can use the reference filter parameter set file to configure the hardware filter with the filter parameters used for processing each image in the image set. In other words, the image processor can use one or more filter parameter set files selected by the user to configure the filter parameters for the hardware filter. The filter parameter set file used by the hardware filter is no longer fixed, but the user can Choose it yourself. In addition, users can also select multiple filter parameter set files. The image processor uses each parameter set corpus in these filter parameter set files to configure filter parameters for the hardware filter, so that the user can determine the most effective filtering effect in these filter parameter set files. A good filtering parameter set file. It can be understood that the user can test the filtering effect of each filtering parameter set file, and then select a filtering parameter set file with the best filtering effect to use in subsequent image processing.

In an alternative implementation, the scheduler 101 is further configured to send the N file names corresponding to the N filter parameter set files to the image processor 105 according to the parameter set selection instruction, and the N file names and The above N filter parameter set files are in one-to-one correspondence;

The image processor 105 is further configured to acquire the reference filter parameter set file from the memory 103 according to the reference file name, where the reference file name is the file name of the reference filter parameter set file and is included in the N file names.

FIG. 2 is a schematic diagram of a filtering parameter set selection interface provided by an embodiment of the present application. As shown in FIG. 2, the filtering parameter set selection interface includes file names of F filtering parameter set files, that is, the first file name to the Fth file name; the square in front of each file name is the filtering parameter set file corresponding to the file name After the user selects the square through the keyboard or the mouse, the square turns black (indicating that the corresponding selection interface of the square is selected). The above parameter set selection instruction is an operation for the user to select the file name through the input device. The filtering parameter set selection interface is an interface of the image processing program. As shown in FIG. 2, the second file name, the third file name, and the fifth file name are the file names selected by the user, and the three filter parameter files corresponding to the three file names are filter parameter files configured by the user, and the image processing device Use these three filter parameter files for image processing.

Optionally, the scheduler 101 is also used to read the reference image from the memory 103 and send it to the image processor. In practical applications, the scheduler 101 reads one image from the image set at a time, and passes the image to the image processor 105.

In this implementation, the scheduler sends the file name of the filter parameter set file to the image processor, so that the image processor reads the corresponding filter parameter set file from the memory according to the file name received; Make less resources.

FIG. 3 is a schematic structural diagram of another image processing device provided by the present application, which is a further refinement of the device shown in FIG. 1. As shown in FIG. 3, the image processing device in FIG. 3 adds an NPU 320 and a result comparator 340 that are both coupled to the scheduler 101 on the basis of the image processing device in FIG. 1. The NPU 320 can implement various AI networks. The NPU 320 can use the AI network to perform image analysis or image processing on the filtered image from the scheduler 101. The result comparer 340 is used to compare the image analysis result (image processing result) with the real result (marking information of the test image), so as to determine the correctness of the image analysis or image processing. The image processing device in FIG. 3 can realize the following operations:

301. The scheduler 101 obtains a parameter set selection instruction input by the input device 102.

The input device 102 may be a keyboard, a mouse, a touch screen, or the like. In practical applications, the operation of the user selecting the file name in the filter parameter set selection interface in FIG. 3 through the input device 102 can be understood as the parameter set selection instruction input by the input device 102.

302. The scheduler 101 reads the reference image from the memory 103.

The above reference image is any image in the image set stored in the memory 103. In practical applications, the scheduler 101 sequentially reads each image in the image set and sends it to the image processor 105 for processing.

303. The scheduler 101 sends the N file names selected by the parameter set selection instruction to the image processor 105.

The above N file names are the file names corresponding to the N filter parameter set files among the F filter parameter set files stored in the memory 103. Optionally, the scheduler 101 sends one of the N file names to the image processor 105 each time. Optionally, the scheduler sends the above reference image together with at least one file name to the image processor 105. The memory storing the F filter parameter set files and the memory storing the image set may be the same or different.

304. The image processor 105 obtains the reference filter parameter set file from the memory 103 using the reference file name.

The reference file name is the file name of the reference filter parameter set file and is included in the N file names.

305. The image processor 105 configures a set of filter parameters in the reference filter parameter set file as a filter parameter to be used by the hardware filter 104 to filter the reference image.

The foregoing embodiment details how the image processor 105 uses the reference filtering parameter set file to configure the filtering parameters for the hardware filter 104, which will not be described in detail here.

306. The hardware filter 104 uses the above set of filtering parameters to filter the reference image, and sends the filtered image of the reference image to the image processor 105.

307. The image processor 105 sends the filtered image of the reference image to the scheduler 101.

308. The scheduler 101 sends the filtered image of the reference image to the NPU 320.

309. The NPU 320 uses the target AI network to perform image analysis on the filtered image of the reference image, and sends the obtained image analysis result to the scheduler 101.

310. The scheduler 101 sends the image analysis result and the annotation information of the reference image to the result comparator 430.

Optionally, the result comparator 340 stores the annotation information of the reference image, and the scheduler 101 only needs to send the image analysis result to the result comparator 340.

311. The result comparator 430 compares the image analysis result with the reference information of the reference image to obtain a comparison result of the parameter image.

The AI network used by the NPU 320 is different or the form of the image analysis result output by the AI network is different, and the comparison algorithm adopted by the result comparator 340 is different. For example, the AI network used by NPU320 is Resnet18 network or Resnet50 network, and the image analysis result output by NPU320 is the label (classification) obtained by identifying the image; the result comparator 340 compares the image analysis result output by NPU320 and the ground to get top1 ~ Top5. top1 refers to the probability that the classification with the highest probability predicted by the NPU320 using the AI network is correct. top5 refers to the probability that the NPU320 uses the AI network to predict the maximum probability of the five categories (labels) that have the correct classification. The meaning of top2～top4 is similar to top5. For another example, the AI network used by the NPU320 is the FasterRCNN network. The image analysis result output by the NPU320 is the bounding box obtained by performing object detection on the image; the result comparator 340 uses the degree of overlap (Intersection over Union, IoU) comparison algorithm, that is, calculating the overlapping degree between the bounding box output by the NPU 320 and the bounding box (labeling information) marked in the image. IoU defines the degree of overlap between two bounding boxes. The formula for calculating the degree of overlap between the first bounding box and the second bounding box is: IoU = (A∩B)/(A∪B). Among them, A represents the area occupied by the first bounding box, B represents the area occupied by the second bounding box, A∩B represents the area where the first bounding box and the second bounding box overlap, and A∪B represents the combination of the first bounding box and the second bounding box The area obtained by the second bounding box. FIG. 4 is a schematic diagram of calculating the overlapping degree of two bounding boxes provided by an embodiment of the present application. As shown in Figure 4, the black area is the overlapping area of the two bounding boxes (A∩B), and the entire area is the area where the two bounding boxes merge (A∪B).

Repeat 302 to 311 until the scheduler 101 obtains the comparison result of processing each image in the image set using the reference filtering parameter set file; repeat 301 to 311 until the scheduler 101 obtains each processing in the image set using the above N filtering parameter set files Comparison results of images.

312. The scheduler 101 determines the filtering effect of the reference filter parameter set according to the comparison result of each image in the image set; separately determines the filtering effect of the filter parameter set other than the reference filter parameter set in the N filter parameter sets; The target filtering parameter set with the best filtering effect in the N filtering parameter sets is configured as the filtering parameter set of the hardware filter.

The N filter parameter sets are N parameter sets obtained by the image processor parsing the N filter parameter set files. The scheduler 101 determines the filtering effect of the filtering parameter sets other than the reference filtering parameter set in the N filtering parameter sets in the same manner as determining the filtering effect of the reference filtering parameter set. It can be understood that the better the image filtering effect of the image processing device, the better the effect of image processing or image analysis. Therefore, the advantages and disadvantages of the image analysis result obtained by the image processing device using each filter parameter set to process the image set can reflect the effect of using the filter parameter set to perform the filtering process. The filtering effect of the aforementioned reference filtering parameter set may be obtained according to multiple comparison results (overlaps) output by the result comparator. Optionally, the filtering effect of each filtering parameter set in the N filtering parameter sets is the top1 obtained by the image processing device using the filtering parameter set to perform image recognition on each image in the image set. In other words, the accuracy of image recognition is positively correlated with the filtering effect. Optionally, the filtering effect of each filtering parameter set in the N filtering parameter sets is the number of overlapping degrees that exceed the target threshold value in each overlapping degree obtained by the image processing device using the filtering parameter set to perform object detection on each image in the image set. The target threshold may be 0.5, 0.6, 0.75, 0.8, etc. For example, the image processing apparatus uses the first filter parameter set to process the images in the image set to obtain 600 degrees of overlap that exceed the target threshold, and uses the second filter parameter set to process the images in the image set to obtain 800 degrees of overlap that exceeds the target threshold; Then, the filtering effect of the second filtering parameter set is better than the filtering effect of the first filtering parameter set. In practical applications, various methods can be used to compare the filtering effects of different filtering parameter sets, which is not limited in this application.

In the embodiment of the present application, the image processing device compares the filtering effects of multiple filtering parameter set files selected by the user, and selects one filtering parameter set file with the best filtering effect to quickly determine the filtering parameter set file with the best filtering effect, Improve the efficiency of image analysis.

FIG. 5 is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application. The image processing device may be a device without a hardware filter and an NPU, such as a desktop computer, a notebook computer, etc., which can be applied to a developer scenario. As shown in FIG. 1, the image processing apparatus includes a main device 501 and an external device 500. The external device 500 is coupled to the main device 501 through the IO interface 511. Optionally, the external device 500 is coupled to the main device 501 in other ways or the external device 100 is integrated with the main device (that is, the external device 500 is integrated into the main device 501). The processor 503 and the memory 103 are coupled to the system bus 505. The processor 503 may be one or more processors, where each processor may include one or more processor cores. A video adapter 507. The video adapter 507 can drive the display 509, which is coupled to the system bus 505. The display 509 can be used to display information input by the user or provided to the user and various program interfaces. The system bus 505 is coupled to the I/O interface 511. The I/O interface 511 communicates with various I/O devices, such as input device 102 (eg, keyboard, mouse, touch screen, etc.), multimedia disk (media) 515, (eg, CD-ROM, multimedia interface, etc.), The transceiver 517 (may transmit and/or receive radio communication signals.

The processor 503 may use a general-purpose central processing unit (Central Processing Unit, CPU), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits for executing related programs, to The technical solutions provided by the embodiments of the present application are implemented.

The memory 103 stores various software programs and/or multiple sets of instructions. In a specific implementation, the memory 103 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 103 may store an operating system (hereinafter referred to as a system), such as an embedded operating system such as ANDROID, IOS, WINDOWS, or LINUX. The memory 103 may also store a network communication program, which may be used to communicate with one or more additional devices, one or more terminal devices, and one or more network devices. The memory 103 can also store a user interface program, which can display the content of the application program vividly through a graphical operation interface, and receive user control operations on the application program through input controls such as menus, dialog boxes, and keys. .

The image processing apparatus can communicate with other devices through the network interface 521. Optionally, the network interface 521 communicates with other devices through the Internet, Ethernet, virtual private network, WiFi network, cellular network, and other networks.

The hard disk drive interface 523 is coupled to the system bus 505. The hardware drive interface 523 and the hard disk drive 525 are coupled. The system memory 527 is coupled to the system bus 505. The data running in the system memory 527 may include the operating system 529 and application programs 531 of the image processing apparatus. The application 531 may include a social application, an image management application (for example, an album), a map application (for example, Google Maps), a browser, a game application, programming software, and so on.

The operating system includes a shell 533 and a kernel 535. The shell 533 is an interface between the user and the kernel of the operating system. The shell is the outermost layer of the operating system. The shell manages the interaction between the user and the operating system: waits for the user's input, interprets the user's input to the operating system, and processes the output of various operating systems.

The kernel 535 is composed of those parts of the operating system that are used to manage memory, files, peripherals, and system resources. Directly interacting with the hardware, the operating system kernel usually runs processes and provides communication between processes, providing CPU time slice management, interrupts, memory management, IO management, and so on.

The image processor 105 is coupled to the processor 503. The image processor 105 is used to process the image. It is a hardware dedicated to image processing, and is used to configure the filtering parameters of the hardware filter 104. The result comparator 340 is coupled to the processor 503, and the result comparator 340 is used to compare the image analysis result (image processing result) and the real result (marking information of the test image) in order to determine the image analysis or image processing Correctness. The image analysis result may come from a neural network processor (Neural-network Processing Unit, NPU) 320. The NPU320 can be replaced with a tensor processor (Tensor Processing Unit, TPU) and other processors that can perform machine learning. The external device 500 may include a hardware filter 104, an NPU 320, and a scheduler 101. The hardware filter 104 is coupled to the image processor 105, and the image processor 105 may configure the filtering parameters of the hardware filter 104. Optionally, the processor 503 allocates the image processing task to the scheduler 101, and the scheduler 101 completes the task assigned by the processor 503 by calling the hardware filter 104 and the NPU 320. Optionally, the processor 503 directly calls the hardware filter 104 and the NPU 320 to complete the image processing task, that is, the scheduler 101 is optional. The NPU 320 can implement a variety of AI networks for image analysis and image processing. The user can configure the desired network through the commands input by the input device 102. The memory 103 stores at least two filter parameter set files that can be used by the hardware filter 104, and the user can configure the filter parameter set file used by the hardware filter 104 through an instruction input by the input device 102.

An objective of the present application is to configure the filter parameters for the hardware filter in the external device 500, that is, to complete the development of the hardware filter 104, and then the hardware filter 104 with the configured filter parameters can be directly provided to other devices. For example, the configured hardware filter is connected to a computer to provide hardware filtering for the computer. The external device 500 and the main device 501 may be independent devices. For example, the main device 501 is a desktop computer, and the external device 500 is a mobile device. The external device 500 may be connected to the main device 501 through a USB interface or other interfaces. Another objective of the present application is to enhance the image processing capability of a host device that does not have a hardware filter and NPU. At present, devices such as desktop computers and notebook computers do not have NPU and hardware filters. After the external device 500 is connected to these devices (main device 501), these devices can use the external device 500 to enhance their image processing performance. In this way, the image processing performance of these devices can be enhanced without changing the hardware structure of these devices, and the cost is low.

In the embodiment of the present application, the user can not only configure the filtering parameter set used by the image processing device, but also configure the AI network used by the image processing device. The foregoing embodiments describe the implementation of the user-selected filter parameter set. The following describes the implementation of the AI network used by the user to configure the image processing device.

In an optional implementation manner, the NPU 320 obtains the AI network selection instruction input by the input device 102; according to the above-mentioned AI network selection instruction, the network it adopts is configured as the target AI network.

The AI network selection instruction is used to select the target AI network among at least two AI networks to be selected. The at least two AI networks to be selected may include resnet50 network, SSD (Single Shot MultiBox Detector) network, Faster RCNN network, resnet18 network, etc. 6 is a schematic diagram of an AI network selection interface provided by an embodiment of the present application. As shown in Figure 6, the AI network selection interface includes R AI network selection interfaces; the circle in front of each AI network name is the selection interface of the AI network. After the user selects the circle with the keyboard or mouse, the circle becomes black (Indicates that the corresponding AI network is selected). The above AI network selection instruction is an operation for the user to select an AI network through an input device. The AI network selection interface is an interface of an image processing program. The user can start the image processing program to select the AI network and configure the filtering parameter set file. As shown in FIG. 6, the fifth AI network is a target AI network selected by the user, and the image processing device uses the fifth AI network for image processing.

In this implementation, the user selects the AI network used by the image processing device according to his own needs, which is simple to implement and can satisfy different image processing tasks.

In an optional implementation manner, the NPU 320 obtains the AI network programming code input by the input device 102; executes the AI network programming code to implement the target AI network.

In practical applications, users can write their own AI network in the AI network programming interface. When the AI network preset by the image processing device does not include the AI network required by the user, the user can write the AI network required for self-fetching.

In this implementation, users can write their own code to implement the required AI network, rather than being limited to the AI network preset by the image processing device, in order to develop an AI network with better performance.

7 is a schematic structural diagram of a neural network processor provided by an embodiment of the present application. As shown in FIG. 7, the NPU 320 is mounted on the scheduler 101, and the core part of the NPU 320 is an arithmetic circuit 703, and the arithmetic circuit is controlled by a controller 704 703 Extract the matrix data in the memory and perform multiplication. Optionally, the processor 503 allocates tasks to the scheduler 101, and the scheduler 101 schedules the NPU 320, the hardware filter 104, and other components to perform corresponding tasks.

In some implementations, the arithmetic circuit 703 internally includes multiple processing units (Process Engine, PE). In some implementations, the arithmetic circuit 703 is a two-dimensional pulsating array. The arithmetic circuit 703 may also be a one-dimensional pulsating array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 703 is a general-purpose matrix processor.

For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit takes the data corresponding to the matrix B from the weight memory 702 and caches it on each PE in the arithmetic circuit. The arithmetic circuit takes matrix A data and matrix B from the input memory 701 to perform matrix operation, and the partial result or final result of the obtained matrix is stored in the accumulator 708.

The unified memory 706 is used to store input data and output data. The weight data is directly transferred to the weight memory 702 through a storage unit access controller (Direct Memory Access Controller, DMAC) 705. The input data is also transferred to the unified memory 706 through the DMAC.

A bus interface unit (Bus Interface Unit, BIU) 710 is used for the interaction between the AXI bus and the DMAC and instruction retrieval memory (Instruction Fetch Buffer) 709.

The bus interface unit 710 is used to obtain the instruction from the instruction fetch memory 709, and is also used to access the controller 705 to obtain the original data of the input matrix A or the weight matrix B.

The DMAC is mainly used to carry input data to the unified memory 706 or weight data to the weight memory 702 or input data to the input memory 701.

The vector calculation unit 707 has a plurality of operation processing units. If necessary, it further processes the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithm operation, size comparison, and so on. It is mainly used for non-convolution/FC layer network calculation in neural networks, such as Pooling, Batch Normalization, Local Response Normalization, etc.

In some implementations, the vector calculation unit 707 can store the processed output vector to the unified buffer 706. For example, the vector calculation unit 707 may apply a non-linear function to the output of the arithmetic circuit 703, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 707 generates normalized values, merged values, or both. In some implementations, the vector of the processed output can be used as an activation input to the arithmetic circuit 703, for example for use in subsequent layers in a neural network.

An instruction fetch memory (709) connected to the controller 704 is used to store instructions used by the controller 704. The unified memory 706, the input memory 701, the weight memory 702, and the fetch memory 709 are all On-Chip memories.

It should be understood that FIG. 7 is only a schematic diagram of a hardware structure of an NPU provided by an embodiment of the present application. The present application does not limit the structure of the NPU.

FIG. 8 is a schematic flowchart of an image processing method provided by an embodiment of the present application. The image processing method is applied to the image processing apparatuses in FIGS. 1, 3, and 5. As shown in FIG. 8, the method may include:

801. The scheduler 101 obtains a parameter set selection instruction input by the input device 102.

The above parameter set selection instruction is used to select N filter parameter set files among the F filter parameter set files stored in the memory 103, where F is an integer greater than 1, and N is an integer greater than 0.

802. The image processor 105 configures a set of filter parameters in the reference filter parameter set file as a filter parameter to be used by the hardware filter to filter the reference image.

The reference filtering parameter set file is any one of the N filtering parameter set files, and the reference image is any image in the image set stored in the memory 103.

803. The hardware filter 105 uses the above set of filtering parameters to filter the above reference image.

The image processing method in FIG. 8 describes a process in which the image processing device uses the filter parameter set file selected by the user to filter the images in the image set. The following describes how to determine the implementation of the parameter set file with the best filtering effect among the multiple filter parameter set files selected by the user.

FIG. 9 is a flowchart of a method for comparing filtering effects of multiple filtering parameter set files provided by an embodiment of the present application. The method is applied to the image processing apparatuses in FIGS. 1, 3, and 5. As shown in FIG. 9, the method may include:

901. The image processing device performs image preprocessing on the images in the image set.

The general process of image preprocessing is as follows: unified format, grayscale, filtering, and sample enhancement. Unified format: that is, the format of each image in the image set is unified, for example, all are unified into the jpg format. Grayscale: Different image processing devices have different requirements for image sets. Some require grayscale images and some require RGB color images. Before processing an image, the image processing device needs to convert the image in the image set to an image that meets requirements, for example, to an RGB color image. Filtering: Image filtering in image preprocessing mainly includes adjusting the image size, and denoising and smoothing the noise in the scaled image. Sample enhancement: Some people think that neural networks are fed by data. If you can increase the sample size of training data and provide massive data for training, you can effectively improve the quality of image analysis algorithms. Common sample enhancement methods are: horizontal flip image, random cropping, translation transformation, color, lighting transformation, etc. Image preprocessing is a common technical method in the art, and will not be described in detail here.

In the process of image preprocessing, the image processing device respectively uses multiple filtering parameter set files selected by the user to filter the images in the image set. As shown in FIG. 9, first, the image processing device uses the filter parameter set file 1 to filter the image in the image set and performs subsequent image processing; then, uses the filter parameter set file 2 to filter the image in the image set and performs subsequent Image processing; Finally, the filtering parameter set file F is used to filter the images in the image set and perform subsequent image processing. The implementation of the filtering operation can be referred to 301 to 306, and will not be described in detail here.

902. The image processing device performs image analysis on each image in the image set to obtain an image analysis result of each image.

The image processing device can use the AI network to perform image analysis on the preprocessed image, such as object detection and image classification. The image analysis result may be a label (classification) obtained by the image processing device recognizing the image in the image set, or a bounding box obtained by the image processing device performing object detection on the image in the image set. The implementation is the same as 309 in FIG. 3.

903. The image processing device compares the image analysis result of each image with the annotation information to obtain a comparison result of each image.

For the method of the image processing device comparing the image analysis result of each image and the annotation information, refer to 311 in FIG. 3.

904. The image processing device determines a target filtering parameter set file with the best filtering effect among the N filtering parameter set files; and configures the target filtering parameter set file as the filtering parameter set file that it uses.

In practical applications, the image processing device can separately test the effect of using each filter parameter set file selected by the user to perform image analysis on the image in the image set, and then select the filter parameter set file with the best filtering effect as the filter parameter set to be adopted file.

In the embodiment of the present application, the image processing device can determine the filter parameter set file with the best filtering effect among the multiple filter parameter set files selected by the user, which can further improve the image analysis performance of the image processing device.

The technical solutions of the present application are introduced in the foregoing embodiments, and the beneficial effects achieved by the technical solutions of the present application are described below.

An image processing scheme (first scheme) provided by an embodiment of the present application is as follows: The image processing apparatus configures an optimal filter parameter set for 1000 test images (samples) of the resnet50 network, and then inputs the filter parameter to resnet50 after being filtered by a hardware filter On the network, the obtained output (image analysis result) is compared with ground truth to get top1～top5.

The image processing scheme used for comparison (the second scheme) is as follows: 1000 test images of resnet50 network are input to pillow, after filtering, they are input to resnet50 network, and the obtained output (image analysis result) is compared with ground truth to get top1 ~ Top5. PIL (Python Imaging Library) is a commonly used image processing library in Python, and pillow is a friendly fork of PIL that provides extensive file format support and powerful image processing capabilities, including image storage, image display, format conversion, and basic Image processing operations, etc.

Table 1

Table 1 is a comparison table of image recognition rates. It can be seen from Table 1 that the performance of the first scheme is better than that of the second scheme. For top1, the first plan exceeds the second plan (pillow) by 0.3%. Among the five statistical results from top1 to top5, top1 is the main evaluation index.

Another image processing scheme (third scheme) provided by the embodiment of the present application is as follows: The image processing device configures an optimal filter parameter set for 2000 test images of the lane line recognition network, and then inputs the filter to the lane after being filtered by the hardware filter Line identification network, the obtained output is compared with ground truth to get the accuracy rate and recall rate.

The image processing scheme used for comparison (the fourth scheme) is as follows: input 2000 test images of the lane line recognition network to OpenCV, and then input the filtered line to the lane line recognition network. The obtained output is compared with the ground to obtain the accuracy rate And recall rate.

Precision refers to the number of correctly identified samples/the number of identified samples. Recall (recall) refers to identifying the correct number of samples/the correct number of all samples.

Table 2

图像处理方案Image processing scheme	整体精确率Overall accuracy	整体召回率Overall recall rate	当前精确率Current accuracy	当前召回率Current recall rate
第三方案Third option	0.94460.9446	0.66820.6682	0.93180.9318	0.90580.9058
第四方案Fourth option	0.95360.9536	0.64670.6467	0.94280.9428	0.90610.9061

Table 2 is a table of accuracy and recall rates for image analysis. The overall accuracy rate in Table 2 refers to the accuracy rate of each lane line in the recognition image. The current accuracy rate refers to the accuracy rate of the lane in which the vehicle is currently traveling. The overall recall rate refers to the recall rate of each lane line in the recognition image. The current recall The rate refers to the recall rate that identifies the lane in which the vehicle is currently traveling. It can be seen from Table 2 that the performance of the third scheme is better than that of the fourth scheme. The third scheme exceeds the fourth scheme by 1.1% in the accuracy rate of identifying the current lane, and exceeds the fourth scheme by 0.3% in identifying the current lane recall rate.

As can be seen from the above, the technical solution in this application can effectively improve the accuracy of image recognition.

The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented using software, the above-described embodiments may be fully or partially implemented in the form of computer program products.

The above is only the specific implementation of this application, but the scope of protection of this application is not limited to this, any person skilled in the art can easily think of various equivalents within the technical scope disclosed in this application Modifications or replacements, these modifications or replacements should be covered within the scope of protection of this application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

An image processing device, characterized in that it includes:

The scheduler is used to obtain a parameter set selection instruction input by the input device, the parameter set selection instruction is an instruction input by a user through the input device and is used to instruct to select N filters from the F filter parameter set files stored in the memory Parameter set file, where F is a positive integer greater than 1 and N is an integer greater than 0;

A hardware filter coupled to the scheduler is used to filter the reference image using a set of filter parameters included in one filter parameter set file among the N filter parameter set files.
The image processing apparatus according to claim 1, wherein:

Image processors coupled to the scheduler and the hardware filter, respectively, for parsing the one filter parameter set file to obtain a reference filter parameter set; based on the first scaling factor of the reference image in the horizontal direction and the The second scaling factor in the vertical direction determines that the set of filter parameters in the reference filter parameter set is a parameter to be used by the hardware filter to filter the reference image by using the reference filter parameter set.
The image processing apparatus according to claim 2, wherein:

The image processor is specifically configured to select a group of horizontal filtering parameters from the reference filtering parameter set according to the first scaling factor, and select a group of reference filtering parameter sets from the reference filtering parameter set according to the second scaling factor Vertical filtering parameters; the set of horizontal filtering parameters and the set of vertical filtering parameters as the set of filtering parameters; configuring the set of filtering parameters to the register of the hardware filter ;

The hardware filter is specifically used to read the set of filter parameters in the register and filter the reference image using the set of filter parameters.
The image processing apparatus according to claim 3, wherein

The hardware filter is also used to send the filtered image of the reference image to the image processor;

The image processor is further configured to send the filtered image of the reference image to the scheduler; the image processing device further includes:

A neural network processor NPU coupled with the scheduler;

The scheduler is further configured to input the filtered image of the reference image to the NPU;

The NPU is used to perform image analysis on the filtered image of the reference image using a target AI network, and send the obtained image analysis result to the scheduler.
The image processing device according to claim 4, wherein the image processing device further comprises:

A result comparator coupled with the scheduler;

The scheduler is also used to send the annotation information of the reference image and the image analysis result to the result comparator;

The result comparator is used to compare the image analysis result with the reference information of the reference image to obtain the comparison result of the parameter image.
The image processing apparatus according to claim 5, wherein:

The result comparison device is further used to send the comparison result of the reference image and the comparison result of each image in the image set except the reference image to the scheduler; wherein, the result comparison The matcher obtains the comparison result of each image in the image set except the reference image in the same way as the comparison result of the reference image, and the image set includes at least two images;

The scheduler is further configured to determine the filtering effect of the reference filtering parameter set according to the comparison result of each image in the image set; separately determine the filtering other than the reference filtering parameter set in the N filtering parameter sets The filtering effect of the parameter set, where the N filtering parameter sets are N parameter sets obtained by the image processor parsing the N filtering parameter set files; the N filtering parameter set is the target filter with the best filtering effect The parameter set is configured as a filtering parameter set of the hardware filter; wherein, the manner in which the scheduler determines the filtering effect of the filtering parameter set other than the reference filtering parameter set in the N filtering parameter sets and determines the The filtering effect of the reference filtering parameter set is the same.
The image processing device according to any one of claims 4 to 6, wherein

The NPU is also used to obtain an AI network selection instruction input by the input device, and the AI network selection instruction is used to select the target AI network among at least two AI networks to be selected; the selection is based on the AI network Instruction to configure the network adopted by the NPU as the target AI network;

or,

The NPU is also used to obtain the AI network programming code input by the input device; execute the AI network programming code to implement the target AI network.
An image processing method is applied to an image processing device. The image processing device includes a scheduler and a hardware filter. The method includes:

The scheduler obtains a parameter set selection instruction input by the input device, where the parameter set selection instruction is an instruction input by the user through the input device and is used to instruct the selection of N filter parameters from the F filter parameter set files stored in the memory Set file, where F is a positive integer greater than 1 and N is an integer greater than 0;

The hardware filter uses a set of filtering parameters included in one filtering parameter set file among the N filtering parameter set files to filter the reference image.
The method according to claim 8, wherein the image processor further comprises an image processor respectively coupled to the scheduler and the hardware filter; the hardware filter uses the N filtering parameters Before filtering a reference image by a set of filtering parameters contained in a filtering parameter set file in the set file, the method further includes:

The image processor parses the one filtering parameter set file to obtain a reference filtering parameter set; and determines the reference filtering parameter set according to the first scaling factor of the reference image in the horizontal direction and the second scaling factor in the vertical direction The set of filtering parameters is a parameter to be used by the hardware filter to filter the reference image by using the reference filtering parameter set.
The method according to claim 9, wherein the group of the reference filter parameter set is determined according to a first scaling factor in the horizontal direction and a second scaling factor in the vertical direction of the reference image The filtering parameters are the parameters to be adopted by the hardware filter for filtering the reference image by using the reference filtering parameter set, including:

The image processor selects a set of horizontal filtering parameters from the reference filtering parameter set according to the first scaling factor, and selects a set of vertical filtering from the reference filtering parameter set according to the second scaling factor Parameters; the set of horizontal filtering parameters and the set of vertical filtering parameters as the set of filtering parameters; configuring the set of filtering parameters to the register of the hardware filter;

The hardware filter filtering the reference image using a set of filtering parameters included in one filtering parameter set file among the N filtering parameter set files includes:

The hardware filter reads the set of filter parameters in the register and uses the set of filter parameters to filter the reference image.
The method according to claim 10, wherein the image processing device further comprises a neural network processor NPU coupled to the scheduler; the hardware filter uses one of the N filtering parameter set files for filtering After filtering the reference image by a set of filtering parameters contained in the parameter set file, the method further includes:

The hardware filter sends the filtered image of the reference image to the image processor;

The image processor sends the filtered image of the reference image to the scheduler;

The scheduler inputs the filtered image of the reference image to the NPU;

The NPU uses the target AI network to perform image analysis on the filtered image of the reference image, and sends the obtained image analysis result to the scheduler.
The method according to claim 11, wherein the image processing device further comprises a result comparator coupled with the scheduler; the NPU uses the target AI network to image the filtered image of the reference image After analyzing and sending the obtained image analysis result to the scheduler, the method further includes:

The scheduler sends the annotation information of the reference image and the image analysis result to the result comparator;

The result comparison device compares the image analysis result with the reference information of the reference image to obtain a comparison result of the parameter image.
The method according to claim 12, wherein the result comparator compares the image analysis result with the reference information of the reference image, and after obtaining the comparison result of the parameter image, the The method also includes:

The result comparator sends the comparison result of the reference image and the comparison result of each image in the image set except the reference image to the scheduler; wherein, the result comparator obtains the result The way of comparing the results of each image in the image set except the reference image is the same as the way of obtaining the results of the comparison of the reference image, and the image set includes at least two images;

The scheduler determines the filtering effect of the reference filtering parameter set according to the comparison result of each image in the image set; separately determines the filtering of the filtering parameter sets other than the reference filtering parameter set in the N filtering parameter sets Effect, the N filter parameter sets are N parameter sets obtained by the image processor parsing the N filter parameter set files; the target filter parameter set with the best filtering effect in the N filter parameter sets is configured as A filtering parameter set of the hardware filter; wherein, the manner in which the scheduler determines a filtering effect of a filtering parameter set other than the reference filtering parameter set in the N filtering parameter sets and determines the reference filtering parameter set The filtering effect is the same.
The method according to any one of claims 11 to 13, wherein the NPU performs image analysis on the filtered image of the reference image using a target AI network, and sends the obtained image analysis result to the scheduler Previously, the method also included:

The NPU obtains an AI network selection instruction input by the input device, and the AI network selection instruction is used to select the target AI network in at least two AI networks to be selected; according to the AI network selection instruction, the The network configuration adopted by the NPU is the target AI network;

or,

The NPU obtains the AI network programming code input by the input device; executes the AI network programming code to implement the target AI network.