WO2023045197A1 - 一种图像处理方法、装置及设备 - Google Patents

一种图像处理方法、装置及设备 Download PDF

Info

Publication number
WO2023045197A1
WO2023045197A1 PCT/CN2022/074064 CN2022074064W WO2023045197A1 WO 2023045197 A1 WO2023045197 A1 WO 2023045197A1 CN 2022074064 W CN2022074064 W CN 2022074064W WO 2023045197 A1 WO2023045197 A1 WO 2023045197A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
level
current
stage
image processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/074064
Other languages
English (en)
French (fr)
Inventor
蒋东东
董刚
赵雅倩
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Wave Intelligent Technology Co Ltd
Original Assignee
Suzhou Wave Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Wave Intelligent Technology Co Ltd filed Critical Suzhou Wave Intelligent Technology Co Ltd
Priority to US18/564,341 priority Critical patent/US20240265496A1/en
Publication of WO2023045197A1 publication Critical patent/WO2023045197A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/60—Memory management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • This application relates to the field of deep learning, in particular to an image processing method, device and equipment.
  • the inventor realized that for the former, the external memory will reduce the reading and writing speed in the calculation process, causing the calculation speed to drop sharply, and even the calculation unit is forced to use serial calculation due to the large amount of data, which further deteriorates the calculation performance;
  • the hardware performance improvement of the arithmetic unit will lead to a sharp increase in cost: its internal memory capacity doubles, and the price may increase tenfold, and is limited by the adaptability of the computing unit, and may face frequent replacement of different models The problem of the calculator further increased the cost pressure.
  • an embodiment of the present application provides an image processing method, which is applied to a computing unit in an image processing device, wherein the image processing device includes at least a first computing unit and a second computing unit, and the above-mentioned first computing unit and the second computing unit
  • the processors are cascaded through serial transceivers, and the above-mentioned first computing unit and the second computing unit execute the above-mentioned image processing method in parallel;
  • the above image processing methods include:
  • the execution process of the image processing method is cyclically executed.
  • obtaining the sub-image obtained by dividing the original image by the external processor includes:
  • the convolution kernel has width N and height M;
  • the width N is a positive odd number
  • the height M is a positive odd number
  • the number of channels of the convolution kernel is the same as the number of channels of the image to be processed at this stage.
  • the convolution kernel parameters are obtained through pre-training.
  • extracting the current-level image elements in the current-level result image according to the specified convolution kernel and the segmentation method of the obtained sub-image includes:
  • synthesizing the current-level result image and the subsequent-level image elements into the current-level to-be-processed image includes:
  • the spliced image is used as the image to be processed at this stage.
  • performing the convolution operation on the image to be processed at this stage includes:
  • the embodiment of the present application further provides an image processing device, including at least:
  • both the first computing unit and the second computing unit execute the image processing method provided by the first aspect above in parallel;
  • the first arithmetic unit and the second arithmetic unit are cascaded through a serial transceiver;
  • the first operator acts as a front-stage operator to receive image elements
  • the second operator is used as a subsequent operator to send image elements
  • the serial transceiver is used to send the image elements of the post-stage operator as the post-stage image elements to the front-stage operator.
  • the calculator includes:
  • An internal storage module for storing images to be processed at the current level and result images at the current level
  • the convolution operation module is used to perform convolution operation on the image to be processed at the current level stored in the internal storage module
  • performing the convolution operation includes:
  • the above specified convolution kernel has a width N and a height M;
  • the width N is a positive odd number
  • the height M is a positive odd number
  • the number of channels of the above convolution kernel is the same as the number of channels of the image to be processed at this stage
  • the step value is a uniform step value set for a complete convolution operation.
  • the image preprocessing module is used to extract the image elements of the current stage from the result image of the current stage, and send it to the image preprocessing module of the previous stage through the serial transceiver; and the subsequent image elements received by the serial transceiver, and Synthesizing the result images of the current level in the internal storage module of the current level to generate the image to be processed at the current level;
  • extracting the current-level image elements in the current-level result image according to the specified convolution kernel and the segmentation method of the obtained sub-image includes:
  • synthesizing the current-level result image and the subsequent-level image elements into the current-level to-be-processed image includes:
  • the spliced image is used as the image to be processed at this stage.
  • the serial transceiver is used to send the image operation of the current stage to the pre-stage operator.
  • the arithmetic unit in the image processing device sends the result image of the current stage to the external processor
  • the execution process of the image processing method is cyclically executed.
  • the image processing device when the image processing task starts, acquires the sub-image obtained by dividing the original image by the external processor; and stores the sub-image as the result image of the current stage in the internal storage module of the computing unit.
  • the embodiment of the present application provides an image processing electronic device, including a memory and one or more processors, and computer-readable instructions are stored in the memory, and the computer-readable instructions are executed by the one or more processors When executed, the one or more processors are made to execute the steps of any one of the above image processing methods.
  • the embodiment of the present application has one or more non-volatile computer-readable storage media storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more A plurality of processors execute the steps of any one of the above image processing methods.
  • FIG. 1 is a schematic diagram of a hardware architecture for performing image processing tasks provided by the present application according to one or more embodiments;
  • Fig. 2 is a schematic diagram of an image processing task flow provided by the present application according to one or more embodiments
  • Fig. 3(a) is a schematic diagram of an original image evenly divided in the horizontal direction according to one or more embodiments of the present application;
  • Fig. 3(b) is a schematic diagram of an original image evenly divided in the vertical direction provided by the present application according to one or more embodiments;
  • Fig. 4 is a schematic diagram of internal functional modules of a single arithmetic unit provided by the present application according to one or more embodiments;
  • Figure 5(a) is a schematic diagram of extracting image elements at this level according to sub-images divided by convolution kernel and horizontal mode provided by the present application according to one or more embodiments;
  • Fig. 5(b) is a schematic diagram of extracting current-level image elements according to the convolution kernel and sub-images segmented vertically according to one or more embodiments of the present application;
  • Fig. 6(a) is a schematic diagram of a horizontally segmented current-level result image synthesis process of post-level image elements provided by the present application according to one or more embodiments;
  • Fig. 6(b) is a schematic diagram of a process of synthesizing post-level image elements of a vertically segmented current-level result image according to one or more embodiments of the present application;
  • Fig. 7 is a schematic diagram of a convolution operation provided by the present application according to one or more embodiments.
  • Fig. 8 is a schematic diagram of an extended computing module provided by the present application according to one or more embodiments.
  • Fig. 9 is a schematic structural diagram of an image processing electronic device according to one or more embodiments of the present application.
  • Words such as “connected” or “connected” are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Up”, “Down”, “Left”, “Right” and so on are only used to indicate the relative positional relationship. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
  • the hardware architecture for performing image processing tasks as shown in Figure 1 includes:
  • the computing module 100 is configured to execute the image processing method disclosed in this application. Include at least:
  • the first computing unit 111 , the second computing unit 112 , and the first transceiver 141 are respectively connected to the bus 400 in communication.
  • the first computing unit 111 and the second computing unit 112 are cascaded in series through the first transceiver 141 , and the first computing unit 111 receives the image data sent by the second computing unit 112 through the first transceiver 141 .
  • the relationship between two serially cascaded arithmetic units is defined as: the arithmetic unit that receives data through the transceiver is the front-stage arithmetic unit, and the arithmetic unit that sends data through the transceiver is the subsequent-stage arithmetic unit.
  • Each arithmetic unit in the above-mentioned arithmetic module 100 has an independent image processing capability, and can be selected from Field Programmable Gate Array (Field Programmable Gate Array, referred to as FPGA), central processing unit (Central Processing Unit, referred to as CPU), graphics processing unit (Graphics Processing Unit, referred to as GPU), neural network processing (Neural-network Processing Unit, referred to as NPU), digital signal processing (Digital Signal Processing, referred to as DSP), etc.
  • FPGA Field Programmable Gate Array
  • CPU Central Processing Unit
  • GPU Graphics Processing Unit
  • NPU neural network processing
  • DSP Digital Signal Processing
  • transceivers use high-speed serial transceivers, including: GT (Gigabyte Transceiver), GTX, GTH, etc.
  • GT Gigabyte Transceiver
  • GTX Gigabyte Transceiver
  • GTH GTH
  • the external processor 200 is used for segmenting the original image and synthesizing the images processed by the operation module.
  • Select a central processing unit Central Processing Unit, referred to as CPU
  • the CPU can adopt the X86 architecture or the ARM architecture.
  • the external memory 300 is used for storing image files.
  • DRAM Dynamic Random Access Memory
  • SDRAM Synchronous Dynamic Random Access Memory
  • the working frequency is synchronized with the working frequency of the central processing unit, so that the data transmission is in place in real time.
  • single data rate synchronous dynamic random access memory Single Data Rate SDRAM, referred to as SDR SDRAM
  • SDR SDRAM single Data rate synchronous dynamic random access memory
  • DDR SDRAM double data rate Synchronous Dynamic Random Access Memory
  • SDR SDRAM Single Data Rate SDRAM
  • DDR SDRAM Double Data Rate Synchronous Dynamic Random Access Memory
  • SDR SDRAM Single Data Rate Synchronous Dynamic Random Access Memory
  • DDR SDRAM Double Data Rate Synchronous Dynamic Random Access Memory
  • DDR SDRAM Double Data Rate Synchronous Dynamic Random Access Memory
  • DDR SDRAM Double Data Rate Synchronous Dynamic Random Access Memory
  • the bus 400 is used for exchanging data between the computing module 100 and the external processor 200 .
  • Use high-speed serial computer expansion bus Peripheral Component Interconnect Express, referred to as PCIE).
  • Fig. 2 shows a schematic diagram of an image processing task flow according to at least one embodiment disclosed in the present application, including:
  • S1 Obtain the sub-image obtained by dividing the original image by the external processor, and use the sub-image as the result image of the current stage;
  • S3 Send the current-level image element to the front-level operator, and receive the subsequent-level image element sent by the subsequent-level operator;
  • the processing step S1 is first performed: including:
  • S12 The external processor performs lossless segmentation processing on the original image to obtain sub-images arranged in sequence;
  • the external processor sends the sequentially arranged sub-images to corresponding arithmetic units in the arithmetic module.
  • the image to be processed is stored in the external memory 300, and the image has a regular geometric boundary, which may be a square image or a rectangular image.
  • the external processor 200 divides the original image into n sub-images on average along a direction parallel to the wide side of the original image.
  • the external processor 200 divides the original image into n sub-images on average along a direction parallel to the high side of the original image.
  • segmentation is a lossless segmentation, that is, the sum of the data volume of the sub-images has no loss relative to the data volume of the original image.
  • the size is evenly divided.
  • N min the minimum number of operators required.
  • C represents the data size of the original image file
  • m represents the capacity of the internal storage module of each computing unit.
  • the minimum unit for segmentation is a pixel, that is, the width or height of a sub-image is an integer multiple of the width or height of a single pixel, and the width or height of a sub-image that is a fractional multiple of pixels does not appear.
  • the external processor 200 sequentially sends the sub-images to the corresponding computing unit in the computing module 100 through the bus 400, and the computing unit stores the received sub-images as the result image of the current stage in the internal storage module.
  • the internal functional modules of each computing unit in the computing module 100 are as shown in Figure 4, and the internal functional modules of a single computing unit 110 include:
  • the convolution operation module 11 is used to perform convolution operation on the image to be processed at this level stored in the internal storage module 12;
  • the internal storage module 12 is used to store the image to be processed and the result image of the current stage
  • the image and processing module 13 is used to extract the image elements of the current level from the result image of the current level, and send to the image preprocessing module of the previous stage through the serial transceiver;
  • the level image element is synthesized with the result image of the level in the internal storage module of the level to generate the image to be processed at the level; and the image to be processed at the level is stored in the internal storage module of the level;
  • the operator 110 is cascaded with the subsequent operator through the serial transceiver 14i to receive the image elements sent by the latter operator; it is cascaded with the previous operator through the serial transceiver 14o to be used for the previous operator Send this level image element.
  • step S2 extracting the image elements of the current level in the result image of the current level according to the specified convolution kernel and the above-mentioned sub-image segmentation method is described.
  • the dimension of the convolution kernel can be 3 ⁇ 3, 5 ⁇ 5, . . . , as a general case, the dimension of the convolution kernel can be expressed as N ⁇ M.
  • the dimension of the convolution kernel is N ⁇ M
  • the sub-image is horizontally segmented in a direction parallel to the width of the original image.
  • the second computing unit 112 when extracting the current-level image elements in the current-level result image, takes M-1 pixels adjacent to the dividing line and copies them to obtain the current-level image elements.
  • the second operator 112 extracts the original image in the result image of the current stage.
  • the image element of the first level is selected, the N-1 pixels adjacent to the dividing line are taken and copied to obtain the image element of the current level.
  • S3 Send the current-level image element to the previous-stage operator, and receive the subsequent-stage image element sent by the subsequent-stage operator.
  • the image elements are transferred by the serial transceiver 141 between the serially cascaded first operator 111 and the second operator 112 .
  • the external processor 200 divides the original picture in a horizontal division mode, then, when the operator extracts the image elements of the current level, the boundary of the result image of the current level adjacent to the previous level M- An image with a height of one pixel is copied and sent to the pre-stage operator through the serial transceiver 141 .
  • the subsequent image elements sent by the subsequent operator are received, and the subsequent image elements are synthesized into the current-stage result image at the boundary between the current-stage result image and the subsequent-stage sub-image.
  • the external processor 200 divides the original picture in the manner of vertical segmentation, then, when the operator extracts the image elements of the current level, the N- An image with a width of 1 pixel is copied and sent to the front-stage operator through the serial transceiver 141 .
  • the subsequent image elements sent by the subsequent operator are received, and the subsequent image elements are synthesized into the current-stage result image at the boundary between the current-stage result image and the subsequent-stage sub-image.
  • Complementing the preceding image can eliminate the noise at the boundary of the image, and achieve the effect that the image processing quality is not affected by the image segmentation method.
  • the synthesized image is stored in the internal storage module 12 as the image to be processed at the current stage.
  • the convolution kernel To perform convolution operation on the image to be processed at this level, the convolution kernel must first be selected according to the image processing task. Typically, the convolution kernel has a dimension of 3 ⁇ 3 on one channel. However, an original image is composed of red (R), green (G), and blue (B) channels of pixels superimposed, and the convolution kernel must be trained on the three channels of red, green, and blue in order to Get a suitable convolution kernel.
  • R red
  • G green
  • B blue
  • the arithmetic unit does not need to use an external memory, which effectively avoids the reading and writing of data between the arithmetic unit and the external memory, avoids the impact of the memory wall effect on the image processing performance, and achieves the effect of greatly improving the image processing performance .
  • the external processor 200 sequentially stitches the result images sent back by each computing unit to synthesize the final result image.
  • This splicing process is the reverse process of the aforementioned S1 segmentation process.
  • processing is performed on images with a large amount of data.
  • the basic structure of the computing module 100 remains unchanged, and the computing units in it need to be expanded to adapt to the corresponding image processing tasks.
  • the number n of computing devices required in the computing module 100 has been introduced above and will not be repeated here. .
  • it needs to be serially cascaded with the front-stage arithmetic unit through the serial transceiver 142 .
  • the working processes of each computing unit, computing module 100, processing 200, external memory 300, and bus 400 are the same as those described above, and will not be repeated here.
  • each arithmetic unit still adopts the image processing method disclosed in the present application, so that the arithmetic unit can perform parallel processing on images with a large amount of data, and multiple arithmetic units are used for parallel computing and serially cascaded hardware
  • the architecture can achieve the effect of compressing the image processing time and improving the overall image processing performance.
  • the original image is losslessly segmented into sub-images.
  • the amount of data processed by a single arithmetic unit is reduced, and no external memory is needed, which effectively avoids data processing.
  • the read and write actions between the memory device and the external memory avoid the impact of the memory wall effect on the image processing performance, and achieve the effect of greatly improving the image processing performance.
  • each arithmetic unit in the computing module 100 uses FPGA
  • external processor 200 uses CPU
  • external memory 300 uses DDR
  • bus 400 uses PCIE bus
  • serial transceiver 141 selects GT high-speed serial transceiver, single
  • the channel convolution kernel is 3 ⁇ 3, and the number of channels corresponds to the number of channels of the original image; the original image is processed by horizontal bisection.
  • the CPU reads the rectangular original image from the DDR, and divides the original image along the horizontal cutting line parallel to the wide side. Get subimages with the same area.
  • the sub-images are respectively sent to the corresponding FPGA1 and FPGA2 computing units, and stored in the internal storage modules of each computing unit.
  • FPGA1 is equivalent to the pre-stage arithmetic unit
  • FPGA2 is equivalent to the post-stage arithmetic unit.
  • the sub-images stored in the internal storage module will be used as the result images of each FPGA, and will be further processed by two computing units.
  • the dimension of the convolution kernel is 3 ⁇ 3, and the number of channels of the convolution kernel is consistent with the number of channels of each sub-image.
  • the second-stage FPGA2 operator copies two rows of pixel images adjacent to the dividing line from the image dividing line as image elements, and sends the above-mentioned image elements to the front-stage FPGA1 through the GT high-speed serial transceiver, and the front-stage FPGA1 receives the above-mentioned image elements , add the above image elements to the current-level result image along the dividing line to generate the current-level image to be processed, and the current-level result image of the subsequent-stage FPGA is the current-level image to be processed by the subsequent-stage FPGA.
  • the front-stage FPGA1 and the rear-stage FPGA2 will perform parallel convolution operations on their respective images to be processed to obtain their own respective result images.
  • FPGA1 and FPGA2 respectively transmit their own result images to the CPU through the PCIE bus, and the CPU synthesizes the final result images and stores them in the external DDR to complete the image processing task.
  • the CPU reads the rectangular original image from the DDR, and divides the original image along the longitudinal cutting line parallel to the wide side. Get subimages with the same area.
  • the sub-images are respectively sent to the corresponding GPU and stored in the video memory of the GPU.
  • the relationship between the front-end and the back-end of the GPU can be defined by the direction of data transmission: two GPUs are serially cascaded, the one that receives the image elements is the front-end GPU, and the one that sends the image elements is the back-end GPU.
  • the sub-images stored in the video memory will be further processed by each GPU as the current-level result image of each GPU.
  • the current-level GPU processes the resulting image stored in the video memory, and the subsequent-level GPU copies the N-1 column pixel images adjacent to the dividing line from the image dividing line as image elements, and passes the above-mentioned image elements through the serial transceiver Send to the front-level GPU.
  • the front-level GPU receives the above image elements, it will add the above-mentioned image elements to the result image of the current level along the dividing line to generate the image to be processed at the current level. The image to be processed at this level.
  • each GPU performs a convolution operation on the image to be processed at the current stage to obtain a result image and save it in the video memory. Repeat the process of copying image elements, synthesizing images to be processed, convolution operation, and obtaining the result image until the above result image meets the preset requirements.
  • each GPU transmits its own result image to the CPU through the PCIE bus, and the CPU synthesizes the final result image and stores it in DDR to complete the image processing task.
  • the processes described above with reference to the flowcharts can be implemented as computer software programs.
  • the embodiments of the present application include a computer program product, which includes a computer program loaded on a computer-readable medium, where the computer program includes program codes for executing the methods shown in the flowcharts.
  • the computer program may be downloaded and installed from a network through the communication means, or installed from a memory, or installed from a ROM.
  • the computer program is executed by the external processor, the above-mentioned functions defined in the methods of the embodiments of the present application are executed.
  • the computer-readable medium in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two.
  • a computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • a computer-readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, in which computer-readable program codes are carried. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device .
  • the program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency, radio frequency), etc., or any suitable combination of the above.
  • the above-mentioned computer-readable medium may be included in the above-mentioned server; or it may exist independently without being incorporated into the server.
  • the above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the server, the server: in response to detecting that the peripheral mode of the terminal is not activated, obtains the frame rate of the application on the terminal ; When the frame rate meets the off-screen condition, determine whether the user is obtaining the screen information of the terminal; in response to the determination result that the user has not obtained the screen information of the terminal, control the screen to enter the immediate dimming mode.
  • Computer program codes for performing the operations of the embodiments of the present application may be written in one or more programming languages or combinations thereof, including object-oriented programming languages—such as Java, Smalltalk, C++, including A conventional procedural programming language - such as the "C" language or a similar programming language.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
  • the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through an Internet service provider). Internet connection).
  • an image processing electronic device may include a computer device, and the computer device may be a terminal or a server.
  • the internal structure diagram of the image processing electronic device may be shown in FIG. 9 .
  • the image processing electronics includes a processor, a memory, a network interface and an input device connected through a system bus. Among them, the processor is used to provide calculation and control capabilities.
  • the memory includes a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium stores an operating system and computer readable instructions.
  • the internal memory provides an environment for the execution of the operating system and computer readable instructions in the non-volatile storage medium.
  • the network interface of the computer device is used to communicate with an external terminal or server through a network connection.
  • FIG. 9 is only a block diagram of a part of the structure related to the solution of this application, and does not constitute a limitation on the equipment to which the solution of this application is applied.
  • the specific equipment may include More or fewer components are shown in the figures, or certain components are combined, or have different component arrangements.
  • the embodiment of the present application also provides a non-volatile readable storage medium, the non-volatile readable storage medium stores computer-readable instructions, and the computer-readable instruction computer program is executed by one or more processors
  • the steps of the image processing method described in any of the foregoing embodiments may be implemented at this time.
  • each embodiment in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments.
  • the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment.
  • the systems and system embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is It can be located in one place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without creative effort.
  • Nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.
  • Volatile memory can include random access memory (RAM) or external cache memory.
  • RAM random access memory
  • RAM is available in many forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Chain Synchlink DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Geometry (AREA)
  • Image Processing (AREA)

Abstract

本申请公开一种应用于图像处理系统的图像处理方法、装置及设备，属于深度学习领域。该图像处理装置至少包括第一运算器和第二运算器，第一运算器和第二运算器通过串行收发器级联，并且并行执行所述图像处理方法，所述图像处理方法包括：获取外部处理器分割原图像得到的子图像，并作为本级结果图像；根据指定的卷积核和得到子图像的分割方式提取本级结果图像中的本级图像元素；向前级运算器发送本级图像元素，并接收后级运算器发送的后级图像元素；将本级结果图像与后级图像元素合成为本级待处理图像；对本级待处理图像进行卷积运算，得到本级结果图像。

Description

一种图像处理方法、装置及设备
相关申请的交叉引用
本申请要求于2021年09月23日提交中国专利局，申请号为202111114230.7，申请名称为“一种图像处理方法、装置及设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及深度学习领域，特别涉及一种图像处理方法、装置及设备。
背景技术
深度学习算法的出现极大地促进了机器学习的发展，并在图像、语音和自然语言处理领域得到广泛应用。有赖于卷积神经网络(Convolutional Neural Networks,CNN)模型的改进和硬件计算能力的提升，深度学习算法可以借助更多的训练数据进行拟合，提高任务处理的精确性。得益于深度学习技术的应用，图像识别产品已在人们日常的工作、生活中得以广泛应用，为社会的有序运行提供了有力的技术保障。
在图像处理应用的早期，受限于图像采集技术，需要被处理的图像数据量往往有限，采用运算器自带的内部存储器即可完成对图像的处理任务。但随着图像信息的逐渐增大，数据量已经大大超过了计算单元自带存储器的容量。面对这种情况，技术人员可以选择外接存储器方案：即在计算单元之外扩展存储器，将每次卷积计算的结果缓存到外接的扩展存储器中，再读出进行下一轮计算；也可以选择升级运算器方案：即更换带有高容量内部存储器的运算器，依旧延续单个运算器。发明人意识到，对于前者，外接存储器会降低计算过程中的读写速度，使计算速度急剧下降，甚至由于数据量过大导致计算单元被迫采用串行计算的方式，进一步恶化了计算性能；对于后者，运算器的硬件性能提升将导致成本的急剧增大：其内部存储器容量增大一倍，价格可能增加十倍，并且受限于计算单元的适配性，可能面临频繁更换不同型号运算器的问题，进一步增加了成本压力。
发明内容
第一方面，本申请实施例提供一种图像处理方法，应用于图像处理装置中的运算器，其中图像处理装置至少包括第一运算器和第二运算器，上述第一运算器和第二运算 器通过串行收发器级联，上述第一运算器和第二运算器并行执行上述图像处理方法；
上述图像处理方法包括：
获取外部处理器分割原图像得到的子图像，将该子图像作为本级结果图像；
根据指定的卷积核和得到子图像的分割方式提取本级结果图像中的本级图像元素；
向前级运算器发送本级图像元素，并接收后级运算器发送的后级图像元素；
将本级结果图像与后级图像元素合成为本级待处理图像；和
对本级待处理图像进行卷积运算，得到本级结果图像。
在其中一个实施例中，在本级结果图像达到预设标准时，结束图像处理方法的执行过程；和
在本级结果图像未达到预设标准时，循环执行图像处理方法的执行过程。
在其中一个实施例中，获取外部处理器分割原图像得到的子图像包括：
以平行于原图像宽边的方向，横向分割原图像，得到子图像；
或，
以平行于原图像高边的方向，纵向分割原图像，得到子图像。
在其中一个实施例中，卷积核具有宽度N和高度M；
宽度N为正奇数；
高度M为正奇数；
卷积核的通道数与本级待处理图像的通道数相同；和
卷积核参数通过预先训练得到。
在其中一个实施例中，根据指定的卷积核和得到子图像的分割方式提取本级结果图像中的本级图像元素包括：
在以平行于原图像宽边的方向横向分割原图像时，复制本级结果图像中与前级结果图像边界毗邻的M-1条像素作为本级图像元素；和
在以平行于原图像高边的方向纵向分割原图像时，复制本级结果图像中与前级结果图像边界毗邻的N-1条像素作为本级图像元素。
在其中一个实施例中，将本级结果图像与后级图像元素合成为本级待处理图像包括：
将后级图像元素顺序拼接至与本级结果图像衔接的边界处；和
将拼接后的图像作为本级待处理图像。
在其中一个实施例中，对本级待处理图像进行卷积运算包括：
为一次完整的卷积运算设置统一的步进值；和
开始卷积运算过程；
第二方面，本申请实施例还提供一种图像处理装置，至少包括：
第一运算器和第二运算器；
其中，第一运算器与第二运算器均并行执行上述第一方面提供的图像处理方法；
第一运算器与第二运算器通过串行收发器级联；
第一运算器作为前级运算器，接收图像元素；
第二运算器作为后级运算器，发送图像元素；和
串行收发器用于将后级运算器的图像元素作为后级图像元素，发送至前级运算器。
在其中一个实施例中，运算器包括：
内部存储模块，用于存储本级待处理图像和本级结果图像；和
卷积运算模块，用于对存储于内部存储模块的本级待处理图像进行卷积运算；
在其中一个实施例中，进行卷积运算包括：
使用指定的卷积核与设置的步进值进行卷积运算；
在其中一个实施例中，上述指定的卷积核具有宽度N和高度M；
其中，宽度N为正奇数，高度M为正奇数；上述卷积核的通道数与本级待处理图像的通道数相同；
上述卷积核参数通过预先训练得到。
在其中一个实施例中，步进值为了一次完整的卷积运算设置的统一的步进值。
图像预处理模块，用于从本级结果图像中提取本级图像元素，并通过串行收发器发送至前级的图像预处理模块；以及将串行收发器接收到的后级图像元素，与本级内部存储模块中的本级结果图像进行合成，生成本级待处理图像；
并将本级待处理图像存储于本级内部存储模块；
在其中一个实施例中，根据指定的卷积核和得到子图像的分割方式提取本级结果图像中的本级图像元素包括：
在以平行于原图像宽边的方向横向分割原图像时，复制本级结果图像中与前级结果图像边界毗邻的M-1条像素作为本级图像元素；
在以平行于原图像高边的方向纵向分割原图像时，复制本级结果图像中与前级结果图像边界毗邻的N-1条像素作为本级图像元素。
在其中一个实施例中，将本级结果图像与后级图像元素合成为本级待处理图像包括：
将后级图像元素顺序拼接至与本级结果图像衔接的边界处；
将拼接后的图像作为本级待处理图像。
串行收发器，用于将本级图像运算发送至前级运算器。
在其中一个实施例中，在本级结果图像达到预设标准时，图像处理装置中的运算器将本级结果图像发送至外部处理器；
在本级结果图像未达到预设标准时，循环执行图像处理方法的执行过程。
在其中一个实施例中，图像处理装置在图像处理任务开始时，获取外部处理器分割原图像得到的子图像；并将子图像作为本级结果图像存储于运算器的内部存储模块中。
第三方面，本申请实施例提供一种图像处理电子设备，包括存储器及一个或多个处理器，存储器中储存有计算机可读指令，所述计算机可读指令被所述一个或多个处理器执行时，使得所述一个或多个处理器执行上述任意一项图像处理方法的步骤。第四方面，本申请实施例一个或多个存储有计算机可读指令的非易失性计算机可读存储介质，所述计算机可读指令被一个或多个处理器执行时，使得所述一个或多个处理器执行上述任意一项图像处理方法的步骤。
本申请的一个或多个实施例的细节在下面的附图和描述中提出。本申请的其它特征和优点将从说明书、附图以及权利要求书变得明显。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请根据一个或多个实施例提供的一种执行图像处理任务的硬件架构示意图；
图2是本申请根据一个或多个实施例提供的一种图像处理任务流程示意图；
图3(a)是本申请根据一个或多个实施例提供的一种在水平方向平均分割原图像的示意图；
图3(b)是本申请根据一个或多个实施例提供的一种在垂直方向平均分割原图像的示意图；
图4是本申请根据一个或多个实施例提供的一种单个运算器内部功能模块示意图；
图5(a)是本申请根据一个或多个实施例提供的一种根据卷积核和横向方式分割的子图 像提取本级图像元素的示意图；
图5(b)是本申请根据一个或多个实施例提供的一种根据卷积核和纵向方式分割的子图像提取本级图像元素的示意图；
图6(a)是本申请根据一个或多个实施例提供的一种横向分割的本级结果图像合成后级图像元素过程示意图；
图6(b)是本申请根据一个或多个实施例提供的一种纵向分割的本级结果图像合成后级图像元素过程示意图；
图7是本申请根据一个或多个实施例提供的一种卷积运算示意图；
图8是本申请根据一个或多个实施例提供的一种扩展运算模块的示意图；
图9是本申请根据一个或多个实施例提供的一种图像处理电子设备的结构示意图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合本申请实施方式中的附图，对本申请实施方式中的技术方案进行清楚、完整地描述，显然，所描述的实施方式仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
除非另外定义，本公开使用的技术术语或者科学术语应当为本公开所属领域内具有一般技能的人士所理解的通常意义。本公开中使用的“第一”、“第二”以及类似的词语并不表示任何顺序、数量或者重要性，而只是用来区分不同的组成部分。同样，“一个”、“一”或者“该”等类似词语也不表示数量限制，而是表示存在至少一个。说明书附图中的编号，仅表示对各个功能部件或模块的区分，不表示部件或模块之间的逻辑关系。“包括”或者“包含”等类似的词语意指出现该词前面的元件或者物件涵盖出现在该词后面列举的元件或者物件及其等同，而不排除其他元件或者物件。“连接”或者“相连”等类似的词语并非限定于物理的或者机械的连接，而是可以包括电性的连接，不管是直接的还是间接的。“上”、“下”、“左”、“右”等仅用于表示相对位置关系，当被描述对象的绝对位置改变后，则该相对位置关系也可能相应地改变。
下面，将参照附图详细描述根据本公开的各个实施例。需要注意的是，在附图中，将相同的附图标记赋予基本上具有相同或类似结构和功能的组成部分，并且将省略关于它们的重复描述。
在一个实施例中，如图1所示的执行图像处理任务的硬件架构，包括：
运算模块100，用于执行本申请公布的图像处理方法。至少包括：
第一运算器111，第二运算器112，以及第一收发器141。其中第一运算器111，第二运算器112分别与总线400通信连接。第一运算器111与第二运算器112通过第一收发器141串行级联，第一运算器111通过第一收发器141接收第二运算器112发送的图像数据。为方便叙述，定义两个串行级联的运算器的关系为：通过收发器接收数据的运算器为前级运算器，通过收发器发送数据的运算器为后级运算器。
上述运算模块100中的每个运算器均具有独立的图像处理能力，可以选用现场可编程门阵列(Field Programmable Gate Array,简称FPGA)，中央处理器(Central Processing Unit,简称CPU)，图形处理器(Graphics Processing Unit,简称GPU)，神经网络处理(Neural-network Processing Unit,简称NPU)，数字信号处理(Digital Signal Processing,简称DSP)等。
上述收发器选用高速串行收发器，包括：GT(Gigabyte Transceiver),GTX,GTH等。
外部处理器200，用于对原图像进行分割，和对经运算模块处理后的图像进行合成。选用中央处理器(Central Processing Unit,简称CPU)，CPU可以采用X86架构也可以采用ARM架构。
外部存储器300，用于存储图像文件。作为动态随机存储器(Dynamic Random Access Memory,简称DRAM)中的一种，通常选用同步动态随机存储器(Synchronous Dynamic Random Access Memory,简称SDRAM)作为外部存储器，同步动态随机存储器(SDRAM)能够将外部存储器的工作频率与中央处理器的工作频率同步，使数据传输实时到位。作为同步动态随机存储器(SDRAM)的一种，单倍数据速率同步动态随机存储器(Single Data Rate SDRAM,简称SDR SDRAM)可以作为外部存储器的一种选择；作为一种优选的情况，双倍数据速率同步动态随机存储器(Double Data Rate SDRAM,简称DDR SDRAM)可以在时钟信号的上升、下降沿传输数据，相比单倍数据速率同步动态随机存储器(SDR SDRAM)只能在上升沿传输数据，选用双倍数据速率同步动态随机存储器(DDR SDRAM)能够有效提升系统性能。
总线400，用于在运算模块100和外部处理器200之间交换数据。选用高速串行计算机扩展总线(Peripheral Component Interconnect Express,简称PCIE)。
外部处理器200与外部存储器300之间通信连接；外部处理器200与总线400之间通信连接；总线400与运算模块100中的每个运算器通信连接。
图2示出了根据本申请公开的至少一实施例的一种图像处理任务流程的示意图，包括：
S1：获取外部处理器分割原图像得到的子图像，将该子图像作为本级结果图像；
S2：根据指定的卷积核和上述子图像的分割方式提取本级结果图像中的本级图像元素；
S3：向前级运算器发送该本级图像元素，并接收后级运算器发送的后级图像元素；
S4：将上述本级结果图像与后级图像元素合成为本级待处理图像；
S5：对上述本级待处理图像进行卷积运算，得到本级结果图像；
S6：当上述本级结果图像达到预设标准时，结束图像处理方法的执行过程；否则，循环执行图像处理方法的执行过程。
当图像处理任务开始时，首先执行处理步骤S1：包括：
S11：外部处理器从外部存储器中提取原图像；
S12：外部处理器对原图像进行无损分割处理，得到按顺序排列的子图像；
S13：外部处理器将按顺序排列的子图像发送至运算模块中对应的运算器。
其中，外部存储器300中存储有待处理的图像，该图像具有规则的几何边界，可以是正方形图像，也可以是长方形图像。
作为一种情况的，如图3(a)所示，外部处理器200沿着与原图像宽边平行的方向，将原图像平均分割为n个子图像。
作为另一种情况的，如图3(b)所示，外部处理器200沿着与原图像高边平行的方向，将原图像平均分割为n个子图像。
需要说明的是，上述分割是无损分割，即子图像数据量的总和相对于原图像数据量没有损失；平均分割的意思可以是按原图像面积大小进行平均分割，也可以按原图像数据量的大小进行平均分割。
将原图像平均分割为n个子图像，其中n≥N min，N min表示需要最少运算器的数量。N min由公式计算得到：
Figure PCTCN2022074064-appb-000001
其中，C表示原图像文件的数据大小，m表示每个运算器内部存储模块的容量大小。
进行分割的最小单位是一个像素点，即子图像的宽或高为像单个素点宽或高的整数倍，不出现分数倍数像素点的子图像宽度或高度。
对于原图片宽边或高边像素点数量不能被n整除的情况，对排序首位的子图像或排序末位的子图像的宽边或高边进行调整，以保证其他顺位的子图像被平均分割，并且每个子图像的数据量均小于对应运算器的内部存储器容量m。
外部处理器200通过总线400将子图像依次发送至运算模块100中对应的运算器，运 算器将接收到的子图像作为本级结果图像保存在内部存储模块中。
运算模块100中每个运算器内部的功能模块如图4所示，单个运算器110内部功能模块包括：
卷积运算模块11，用于对存储于内部存储模块12的本级待处理图像进行卷积运算；
内部存储模块12，用于存储本级待处理图像和本级结果图像；
图像与处理模块13，用于从本级结果图像中提取本级图像元素，并通过所述串行收发器发送至前级的所述图像预处理模块；以及将串行收发器接收到的后级图像元素，与本级所述内部存储模块中的所述本级结果图像进行合成，生成本级待处理图像；并将本级待处理图像存储于所述本级内部存储模块；
运算器110通过串行收发器14i与后级运算器级联，用于接收后级运算器发送的图像元素；通过串行收发器14o与前级运算器级联，用于向前级运算器发送本级图像元素。
以图1所示的运算模块100结构为例，叙述S2步骤根据指定的卷积核和上述子图像的分割方式提取本级结果图像中的本级图像元素的过程。其中，卷积核维度可以是3×3，5×5，……，作为一种普遍的情况，卷积核维度可以表示为N×M。
如图5(a)所示，卷积核维度为N×M，子图像以平行于原图像宽边的方向，横向分割得到。第二运算器112，提取本级结果图像中的本级图像元素时，取与分割线毗邻的M-1条像素进行复制，得到本级图像元素。如图5(b)所示，对于卷积核维度为N×M，子图像以平行于原图像高边的方向，纵向分割的情况，第二运算器112，提取本级结果图像中的本级图像元素时，取与分割线毗邻的N-1条像素进行复制，得到本级图像元素。
S3：向前级运算器发送该本级图像元素，并接收后级运算器发送的后级图像元素。图像元素由串行级联的第一运算器111和第二运算器112之间的串行收发器141传递。
S4：将上述本级结果图像与后级图像元素合成为本级待处理图像。
针对原图像不同的分割方式，S2-S4的过程可以由图6(a)和图6(b)表示。
如图6(a)所示，若外部处理器200是以横向分割的方式划分原图片，则，运算器在提取本级图像元素时，将与前级毗邻的本级结果图像分界处M-1条像素点高度的图像复制，并通过串行收发器141发送至前级运算器。接收后级运算器发送的后级图像元素，在本级结果图像与后级子图像的分界处将后级图像元素合成到本级结果图像。
如图6(b)所示，若外部处理器200是以纵向分割的方式划分原图片，则，运算器在提取本级图像元素时，将与前级毗邻的本级结果图像分界处N-1条像素点宽度的图像复制，并通过串行收发器141发送至前级运算器。接收后级运算器发送的后级图像元素，在本级结果图像与后级子图像的分界处将后级图像元素合成到本级结果图像。
通过对前级图像进行补图处理，起到了消除图像分界处噪声的作用，达到了图像处理质量不受图像分割方法影响的效果。
将合成后的图像作为本级待处理图像保存在内部存储模块12中。
S5：对上述本级待处理图像进行卷积运算，得到本级结果图像；
对本级待处理图像进行卷积运算，首先要根据图像处理任务选取卷积核。通常，卷积核在一个通道上的维度为3×3。然而，一张原始图像是由红(R)、绿(G)、蓝(B)三个通道的像素叠加而成，要将卷积核在红、绿、蓝三个通道上进行训练，才能得到合适的卷积核。
如图7所示，使用训练得到的卷积核，设置步进值为1(表示卷积核每次在图像上移动一个像素宽或高的单位)，对待处理图像进行卷积运算。可以得到本级结果图像，并将其存储于内部存储模块。
运算器在图像处理过程中，无需使用外部存储器，有效规避了数据在运算器和外部存储器之间的读写动作，避免了内存墙效应对图像处理性能的影响，达到大幅提升图像处理性能的效果。
S6：当上述本级结果图像达到预设标准时，结束图像处理方法的执行过程；否则，循环执行图像处理方法的执行过程。在本级结果图像达到预设标准后，运算模块100，将其中每个运算器处理的子图像通过总线400发送给外部处理器200。
最终，由外部处理器200将各个运算器发回的结果图像依次拼接，合成最终结果图像。这一拼接过程即为前述S1分割过程的逆过程。
在另一个实施例中，对于数据量较大的图像进行处理。如图8，运算模块100的基本架构不变，需要对其中的运算器进行扩展，即可适应相应的图像处理任务，其中运算模块100中需要的运算器数量n，前文已介绍，不再赘述。对于扩展的运算器(第n运算器11n)需要与总线400建立通信连接。同时，需要与前级运算器通过串行收发器142进行串行级联。各运算器，以及运算模块100，处理200，外部存储器300，总线400的工作流程与前文叙述相同，在此不再赘述。
通过扩展运算模块中运算器的数量，即可执行对大数据量图像的处理任务。仍然采用原型号的运算器，无需更换高性能，高价格的运算器。
对于扩展后的运算模块，其中每一个运算器仍然采用本申请公开的图像处理方法，使运算器对数据量大的图像可以进行并行处理，采用多个运算器并行计算，串行级联的硬件架构可以达到压缩图像处理时间，提高图像整体处理性能的效果。
在使用高性能运算器受到客观制约的情况下，使用多个低性能的运算器可以起到替 代一个高性能的运算器的作用，并且达到与单个高性能运算器相同、甚至更高的计算能力，同时大大降低成本的效果。
本申请实施例提供的技术方案带来的有益效果是：
1.使用本申请公开的一种图像处理方法，将原图像无损分割为子图像，在图像处理过程中，降低了单个运算器处理图像的数据量，无需使用外部存储器，有效规避了数据在运算器和外部存储器之间的读写动作，避免了内存墙效应对图像处理性能的影响，达到大幅提升图像处理性能的效果。
2.在图像数据量增大的时候，只需扩展运算模块中的运算器数量，采用原型号的运算器即可并行进行图像处理任务。无需更换高性能，高价格的运算模块，即保证了图像处理性能，又降低了硬件成本。
上述所有可选技术方案，可以采用任意结合形成本申请的可选实施例，在此不再一一赘述。
实施例一
在本实施例中，运算模块100中的各运算器使用FPGA，外部处理器200使用CPU，外部存储器300使用DDR，总线400使用PCIE总线，串行收发器141选择GT高速串行收发器，单通道卷积核为3×3，通道数量与原图像通道数量对应；采用横向二分原图像进行图像处理。
当图像处理任务开始时，CPU从DDR中读取矩形原图像，沿与宽边平行的横向切割线对原图像进行分割操作。得到面积相同的子图像。经由PCIE总线，将子图像分别发送至对应的FPGA1运算器和FPGA2运算器，存储于各个运算器的内部存储模块中。其中FPGA1相当于前级运算器，FPGA2相当于后级运算器。存储于内部存储模块中的子图像，将作为各FPGA的本级结果图像，由两个运算器分别对其进行进一步处理。
在本实施例中，卷积核的维度为3×3，卷积核的通道数与各个子图像通道数一致。
后级FPGA2运算器自图像分割线处复制毗邻分割线的两行像素图像作为图像元素，并将上述图像元素通过GT高速串行收发器发送至前级FPGA1，前级FPGA1接收到上述图像元素后，沿分割线将上述图像元素补充至本级结果图像，生成本级待处理图像，后级FPGA的本级结果图像即为后级FPGA的本级待处理图像。
此时，前级FPGA1和后级FPGA2将对各自的待处理图像进行并行卷积运算，分别得到各自的本级果图像。
重复执行上述过程，直至结果图像达到预设要求。
之后，FPGA1，FPGA2分别将各自的本级结果图像通过PCIE总线传输至CPU， CPU合成最终结果图像，存储于外接的DDR中，完成图像处理任务。
实施例二
为了处理数据量较大的高清图像，在本实施例中提供了按照如图1所示的运算模块架构，对运算器硬件设备进行扩展的情况，运算器的数量为n，n的计算方法由前述公式计算得到。在本实施例中，运算模块100中的各运算器使用GPU，外部处理器200使用CPU，外部存储器300使用DDR，总线400使用PCIE总线，串行收发器141，串行收发器142等，选择GT高速串行收发器，单通道卷积核为3×3，通道数量与原图像通道数量对应；采用纵向分割的处理方法将原图像分割为n份。
当图像处理任务开始时，CPU从DDR中读取矩形原图像，沿与宽边平行的纵向切割线对原图像进行分割操作。得到面积相同的子图像。经由PCIE总线，将子图像分别发送至对应的GPU，存储于GPU的显存中。其中GPU的前级、后级关系可以由数据传递方向定义：串行级联的两个GPU，接收图像元素的一个为前级GPU，发送图像元素的一个为后级GPU。存储于显存中的子图像，将作为各GPU的本级结果图像，由各GPU进一步处理。
本实施例中，卷积核维度为N×M，卷积核的通道数与各个子图像通道数一致。
随后，本级GPU对存储于显存的本级结果图像进行处理，后级GPU自图像分割线处复制毗邻分割线的N-1列像素图像作为图像元素，并将上述图像元素通过串行收发器发送至前级GPU，前级GPU接收到上述图像元素后，沿分割线将上述图像元素补充至本级结果图像，生成本级待处理图像，后级GPU的本级结果图像即为后级GPU的本级待处理图像。
此时，各个GPU对本级待处理图像进行卷积运算，得到结果图像，并保存在显存中。重复进行图像元素复制，待处理图像合成，卷积运算，得到结果图像的过程，直到上述结果图像符合预设要求。
之后，各个GPU分别将各自的本级结果图像通过PCIE总线传输至CPU，CPU合成最终结果图像，存储于DDR中，完成图像处理任务。
特别地，根据本申请的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本申请的实施例包括一种计算机程序产品，其包括装载在计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置从网络上被下载和安装，或者从存储器被安装，或者从ROM被安装。在该计算机程序被外部处理器执行时，执行本申请的实施例的方法中限定的上述功能。
需要说明的是，本申请的实施例的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本申请的实施例中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本申请的实施例中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、RF(Radio Frequency,射频)等等，或者上述的任意合适的组合。
上述计算机可读介质可以是上述服务器中所包含的；也可以是单独存在，而未装配入该服务器中。上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该服务器执行时，使得该服务器：响应于检测到终端的外设模式未激活时，获取终端上应用的帧率；在帧率满足息屏条件时，判断用户是否正在获取终端的屏幕信息；响应于判断结果为用户未获取终端的屏幕信息，控制屏幕进入立即暗淡模式。
可以以一种或多种程序设计语言或其组合来编写用于执行本申请的实施例的操作的计算机程序代码，程序设计语言包括面向对象的程序设计语言—诸如Java,Smalltalk,C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
在一个实施例中，提供了一种图像处理电子设备，该图像处理电子设备可以包括计算机设备，该计算机设备可以是终端或服务器，该图像处理电子设备的内部结构图可以 如图9所示。该图像处理电子设备包括通过系统总线连接的处理器、存储器、网络接口和输入装置。其中，该处理器用于提供计算和控制能力。该存储器包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作系统和计算机可读指令。该内存储器为非易失性存储介质中的操作系统和计算机可读指令的运行提供环境。该计算机设备的网络接口用于与外部的终端或者服务器通过网络连接通信。该计算机可读指令被处理器执行时以实现一种图像处理方法。该输入装置可以是显示屏上覆盖的触摸层，也可以是计算机设备外壳上设置的按键、轨迹球或触控板，还可以是外接的键盘、触控板或鼠标等。
本领域技术人员可以理解，图9中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的设备的限定，具体的设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
本申请实施例还提供了一种非易失性可读存储介质，该非易失性可读存储介质中存储有计算机可读指令，该计算机可读指令计算机程序被一个或多个处理器执行时可实现上述任意实施例所述的图像处理方法的步骤。
本说明书中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于系统或系统实施例而言，由于其基本相似于方法实施例，所以描述得比较简单，相关之处参见方法实施例的部分说明即可。以上所描述的系统及系统实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。本领域普通技术人员在不付出创造性劳动的情况下，即可以理解并实施。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机可读指令来指令相关的硬件来完成，所述的计算机可读指令可存储于一非易失性计算机可读取存储介质中，该计算机可读指令在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和/或易失性存储器。非易失性存储器可包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦除可编程ROM(EEPROM)或闪存。易失性存储器可包括随机存取存储器(RAM)或者外部高速缓冲存储器。作为说明而非局限，RAM以多种形式可得，诸如静态RAM(SRAM)、动态RAM(DRAM)、同步DRAM(SDRAM)、双数据率SDRAM(DDRSDRAM)、增强 型SDRAM(ESDRAM)、同步链路(Synchlink)DRAM(SLDRAM)、存储器总线(Rambus)直接RAM(RDRAM)、直接存储器总线动态RAM(DRDRAM)、以及存储器总线动态RAM(RDRAM)等。
以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。
以上所述实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。

Claims (11)

  1. 一种图像处理方法，应用于图像处理装置中的运算器，其特征在于，其中所述图像处理装置至少包括第一运算器和第二运算器，所述第一运算器和第二运算器通过串行收发器级联，所述第一运算器和第二运算器并行执行所述图像处理方法，所述图像处理方法包括：
    获取外部处理器分割原图像得到的子图像，将所述子图像作为本级结果图像；
    根据指定的卷积核和得到所述子图像的分割方式提取本级结果图像中的本级图像元素；
    向前级运算器发送所述本级图像元素，并接收后级运算器发送的后级图像元素；
    将所述本级结果图像与所述后级图像元素合成为本级待处理图像；和
    对所述本级待处理图像进行卷积运算，得到所述本级结果图像。
  2. 根据权利要求1所述的一种图像处理方法，其特征在于，在所述本级结果图像达到预设标准时，结束所述图像处理方法的执行过程；和
    在所述本级结果图像未达到所述预设标准时，循环执行所述图像处理方法的执行过程。
  3. 根据权利要求1所述的一种图像处理方法，其特征在于，所述获取外部处理器分割原图像得到的子图像包括：
    以平行于所述原图像宽边的方向，横向分割原图像，得到所述子图像；
    或，
    以平行于所述原图像高边的方向，纵向分割原图像，得到所述子图像。
  4. 根据权利要求1所述的一种图像处理方法，其特征在于，所述卷积核具有宽度N和高度M；
    所述宽度N为正奇数；
    所述高度M为正奇数；
    所述卷积核的通道数与所述本级待处理图像的通道数相同；和
    所述卷积核参数通过预先训练得到。
  5. 根据权利要求4所述的一种图像处理方法，其特征在于，所述根据指定的卷积核和得到所述子图像的分割方式提取本级结果图像中的本级图像元素包括：
    在以平行于所述原图像宽边的方向横向分割原图像时，复制所述本级结果图像中与前级结果图像边界毗邻的M-1条像素作为所述本级图像元素；和
    在以平行于所述原图像高边的方向纵向分割原图像时，复制所述本级结果图像中与前级结果图像边界毗邻的N-1条像素作为所述本级图像元素。
  6. 根据权利要求1所述的一种图像处理方法，其特征在于，所述将所述本级结果图像与所述后级图像元素合成为本级待处理图像包括：
    将所述后级图像元素顺序拼接至与所述本级结果图像衔接的边界处；和
    将拼接后的图像作为本级待处理图像。
  7. 根据权利要求1所述的一种图像处理方法，其特征在于，所述对所述本级待处理图像进行卷积运算包括：
    为一次完整的卷积运算设置统一的步进值；和
    开始卷积运算过程。
  8. 一种图像处理装置，其特征在于，至少包括：
    第一运算器和第二运算器；
    其中，所述第一运算器与所述第二运算器均并行执行如权利要求1-7任意一项所述的图像处理方法；
    所述第一运算器与所述第二运算器通过串行收发器级联；
    所述第一运算器作为前级运算器，接收图像元素；
    所述第二运算器作为后级运算器，发送图像元素；和
    所述串行收发器用于将所述后级运算器的图像元素作为后级图像元素，发送至所述前级运算器。
  9. 根据权利要求8所述的一种图像处理装置，其特征在于，所述运算器包括：
    内部存储模块，用于存储本级待处理图像和本级结果图像；
    卷积运算模块，用于对存储于内部存储模块的本级待处理图像进行卷积运算；和
    图像预处理模块，用于从本级结果图像中提取本级图像元素，并通过所述串行收发器发送至前级的所述图像预处理模块；以及将串行收发器接收到的后级图像元素，与本级所述内部存储模块中的所述本级结果图像进行合成，生成本级待处理图像；并将所述本级待处理图像存储于所述本级内部存储模块。
  10. 一种图像处理电子设备，其特征在于，包括存储器及一个或多个处理器，所述存储器中储存有计算机可读指令，所述计算机可读指令被所述一个或多个处理器执行时，使得所述一个或多个处理器执行如权利要求1-7任意一项所述的方法的步骤。
  11. 一个或多个存储有计算机可读指令的非易失性计算机可读存储介质，其特征在于，所述计算机可读指令被一个或多个处理器执行时，使得所述一个或多个处理器执行 如权利要求1-7任意一项所述的方法的步骤。
PCT/CN2022/074064 2021-09-23 2022-01-26 一种图像处理方法、装置及设备 Ceased WO2023045197A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/564,341 US20240265496A1 (en) 2021-09-23 2022-01-26 Image processing method, apparatus and device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111114230.7 2021-09-23
CN202111114230.7A CN113570612B (zh) 2021-09-23 2021-09-23 一种图像处理方法、装置及设备

Publications (1)

Publication Number Publication Date
WO2023045197A1 true WO2023045197A1 (zh) 2023-03-30

Family

ID=78174053

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/074064 Ceased WO2023045197A1 (zh) 2021-09-23 2022-01-26 一种图像处理方法、装置及设备

Country Status (3)

Country Link
US (1) US20240265496A1 (zh)
CN (1) CN113570612B (zh)
WO (1) WO2023045197A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113570612B (zh) * 2021-09-23 2021-12-17 苏州浪潮智能科技有限公司 一种图像处理方法、装置及设备
JPWO2023112581A1 (zh) * 2021-12-14 2023-06-22

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107704923A (zh) * 2017-10-19 2018-02-16 珠海格力电器股份有限公司 卷积神经网络运算电路
CN110555847A (zh) * 2019-07-31 2019-12-10 瀚博半导体(上海)有限公司 一种基于卷积神经网络的图像处理方法及装置
US20200160111A1 (en) * 2018-11-15 2020-05-21 Adobe Inc. Learning copy space using regression and segmentation neural networks
CN111738276A (zh) * 2020-06-19 2020-10-02 浪潮(北京)电子信息产业有限公司 基于多核卷积神经网络的图像处理方法、装置及设备
CN113570612A (zh) * 2021-09-23 2021-10-29 苏州浪潮智能科技有限公司 一种图像处理方法、装置及设备

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103493039B (zh) * 2012-04-28 2016-06-29 华为技术有限公司 数据处理方法、数据处理装置、接入设备和用户设备
CN104760550A (zh) * 2015-04-29 2015-07-08 张金甫 一种汽车门信号转换器及汽车伸缩踏板
CN105243399B (zh) * 2015-09-08 2018-09-25 浪潮(北京)电子信息产业有限公司 一种实现图像卷积的方法和装置、实现缓存的方法和装置
JP7058010B2 (ja) * 2016-09-08 2022-04-21 ソニーグループ株式会社 演算装置およびセンサー
JP6767660B2 (ja) * 2017-01-27 2020-10-14 富士通株式会社 プロセッサ、情報処理装置及びプロセッサの動作方法
CN110196053B (zh) * 2019-06-13 2023-06-20 内蒙古大学 一种基于fpga的实时田间机器人视觉导航方法与系统
CN110516799A (zh) * 2019-08-16 2019-11-29 浪潮电子信息产业股份有限公司 一种数据处理系统、方法及介质
CN112995491B (zh) * 2019-12-13 2022-09-16 阿里巴巴集团控股有限公司 视频生成方法、装置、电子设备及计算机存储介质
CN111507950B (zh) * 2020-04-08 2021-08-31 推想医疗科技股份有限公司 图像分割的方法和装置、电子设备及计算机可读存储介质
CN111898081B (zh) * 2020-07-09 2024-02-27 上海兆芯集成电路股份有限公司 卷积运算方法及卷积运算装置
CN111931877B (zh) * 2020-10-12 2021-01-05 腾讯科技（深圳）有限公司 目标检测方法、装置、设备及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107704923A (zh) * 2017-10-19 2018-02-16 珠海格力电器股份有限公司 卷积神经网络运算电路
US20200160111A1 (en) * 2018-11-15 2020-05-21 Adobe Inc. Learning copy space using regression and segmentation neural networks
CN110555847A (zh) * 2019-07-31 2019-12-10 瀚博半导体(上海)有限公司 一种基于卷积神经网络的图像处理方法及装置
CN111738276A (zh) * 2020-06-19 2020-10-02 浪潮(北京)电子信息产业有限公司 基于多核卷积神经网络的图像处理方法、装置及设备
CN113570612A (zh) * 2021-09-23 2021-10-29 苏州浪潮智能科技有限公司 一种图像处理方法、装置及设备

Also Published As

Publication number Publication date
CN113570612B (zh) 2021-12-17
CN113570612A (zh) 2021-10-29
US20240265496A1 (en) 2024-08-08

Similar Documents

Publication Publication Date Title
US12236556B2 (en) Video resolution enhancement method, storage medium, and electronic device
KR102637732B1 (ko) 이미지 신호 프로세서, 상기 이미지 신호 프로세서의 동작 방법 및 상기 이미지 신호 프로세서를 포함하는 애플리케이션 프로세서
CN108632624B (zh) 图像数据处理方法、装置、终端设备及可读存储介质
US11669943B2 (en) Dual-stage system for computational photography, and technique for training same
CN116862762A (zh) 一种视频超分方法、装置、设备及存储介质
WO2023045197A1 (zh) 一种图像处理方法、装置及设备
US20220309320A1 (en) Almost-indirect convolution in quantized neural networks
CN114003859A (zh) 卷积运算方法、可读介质和电子设备
CN114418844A (zh) 图像缩放方法、装置、计算机可读存储介质及终端设备
CN110222829A (zh) 基于卷积神经网络的特征提取方法、装置、设备及介质
US20240029263A1 (en) System and method for identifying auxiliary areas of interest in an image
CN107657587A (zh) 图像处理方法、装置及系统
US10127001B2 (en) Virtualizing applications for per-monitor displaying
WO2017112087A1 (en) Reconstruction of signals using a gramian matrix
CN115456858B (zh) 图像处理方法、装置、计算机设备及计算机可读存储介质
WO2026056842A1 (zh) 图像分类方法及装置、计算机程序产品、电子设备
CN110705653A (zh) 图像分类方法、图像分类装置及终端设备
US20230040673A1 (en) Optimised machine learning processing
CN111462160A (zh) 图像处理方法、装置及存储介质
EP4517647A1 (en) Method and apparatus for scaling and super-resolution based on machine learning
WO2025026443A1 (zh) 图像处理方法、装置、设备及存储介质
WO2025138715A1 (zh) 一种图像处理方法及其相关设备
CN115147297B (zh) 一种图像处理方法及装置
CN112099737B (zh) 存储数据的方法、装置、设备和存储介质
US12417631B1 (en) Techniques for adaptive frame-based clip matching

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22871264

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22871264

Country of ref document: EP

Kind code of ref document: A1