CN117082337A - Image processing system, method and related device - Google Patents

Abstract

The application provides an image processing system, an image processing method and a related device. The image processing system includes: the image processing module comprises at least one noise reduction module and is used for carrying out noise reduction processing on the image through the at least one noise reduction module; the controller is internally provided with a plurality of noise reduction modes, the image processing modules corresponding to the plurality of noise reduction modes are different, the plurality of noise reduction modes are in one-to-one correspondence with a plurality of application scenes, and the controller is used for: determining a target noise reduction mode according to the requirements of an application scene; and controlling a noise reduction module in the corresponding image processing module according to the target noise reduction mode, and scheduling and noise reduction processing is carried out on the video image shot by the camera according to a preset rule. The image processing system can quickly schedule and reduce noise of the video images according to the preset rules by the controller through the noise reduction module corresponding to the application scene, the scheduling process is relatively simple, and the real-time performance and the flexibility are high for switching and reducing noise of the video images in different scenes.

Description

Image processing system, method and related device

Technical Field

The present application relates to the field of image processing technologies, and in particular, to an image processing system, method, and related apparatus.

Background

Along with the increasing abundance of shooting functions of electronic devices and the increasing requirements of people on image quality, the electronic devices are required to have high-quality noise reduction effects in various shooting application scenes. For this purpose, a noise reduction module may be provided in the image processing system to perform noise reduction.

However, in the related art, when the noise reduction module is used to reduce noise of video images in different application scenes, the scheduling process is complex, and the real-time performance and flexibility of switching noise reduction for different application scenes are not high.

Disclosure of Invention

The application provides an image processing system, an image processing method and a related device. Various aspects of embodiments of the application are described below.

In a first aspect, there is provided an image processing system comprising: the image processing module comprises at least one noise reduction module and is used for carrying out noise reduction processing on the image through the at least one noise reduction module; the controller is internally provided with a plurality of noise reduction modes, the image processing modules corresponding to the plurality of noise reduction modes are different, the plurality of noise reduction modes are in one-to-one correspondence with a plurality of application scenes, and the controller is used for: determining a target noise reduction mode according to the requirements of an application scene; and controlling the at least one noise reduction module in the corresponding image processing modules according to the target noise reduction mode, and scheduling and carrying out noise reduction processing on the video image shot by the camera according to a preset rule.

In a second aspect, an image processing method is provided, where the method is applied to a controller, where the controller is located in an image processing system, where the image processing system further includes an image processing module, where the image processing module includes at least one noise reduction module, the image processing module is configured to perform noise reduction processing on an image through the at least one noise reduction module, multiple noise reduction modes are configured in the controller, the image processing modules corresponding to the multiple noise reduction modes are different, and the multiple noise reduction modes are in one-to-one correspondence with multiple application scenarios, and the method includes: determining a target noise reduction mode according to the requirements of an application scene; and controlling the at least one noise reduction module in the corresponding image processing modules according to the target noise reduction mode, and scheduling and carrying out noise reduction processing on the video image shot by the camera according to a preset rule.

In a third aspect, there is provided an electronic device comprising: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to implement the method of the second aspect via the executable instructions.

In a fourth aspect, a computer-readable storage medium is provided, comprising a computer program, which, when executed by a processor, implements a method as described in the second aspect

The image processing system provided by the embodiment of the application is provided with the controller, the controller is internally provided with the noise reduction modes corresponding to various application scenes one by one, and different noise reduction modes correspond to different image processing modules comprising at least one noise reduction module. When the image processing system is used for denoising video images in different application scenes, the controller can be used for quickly enabling the related denoising module to schedule and denoise the video images according to preset rules, the scheduling process is relatively simple, and the real-time performance and the flexibility are high for switching and denoising in different scenes of the video images.

Drawings

Fig. 1 is a schematic structural diagram of an image processing system according to an embodiment of the present application.

Fig. 2 is a schematic diagram of an operation process of the image processing system according to the embodiment of the present application.

FIG. 3 is a timing diagram illustrating the scheduling of the image processing system of FIG. 2 during operation.

Fig. 4 is a schematic diagram of another working procedure of the image processing system according to the embodiment of the present application.

Fig. 5 is a schematic diagram of still another working procedure of the image processing system according to the embodiment of the present application.

Fig. 6 is a flowchart of an image processing method according to an embodiment of the present application.

Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. It should be understood that the same or similar reference numerals are used throughout the drawings to designate the same or similar components.

With the popularization and application of electronic devices, the shooting functions of the electronic devices are becoming more and more abundant, so that the electronic devices are required to have high-quality shooting effects in various application scenes. For example, the electronic device may perform shooting with better image quality at night, or may use multiple cameras to perform shooting at the same time, so as to meet the requirement of a photographer on the frame rate or resolution of the image quality, or shooting in the electronic device may be docked with a third party application, so that the third party application performs subsequent operations on the shot image.

Generally, an electronic device performs noise reduction processing on a captured image by using an image processing system therein to obtain a high-quality capturing effect. The noise reduction processing in the embodiment of the present application may refer to a processing technique of reducing noise in a digital image. Specifically, one or more noise reduction modules may be disposed in the image processing system, and when noise reduction processing is performed on the captured image under different application scenarios, different noise reduction modules or a combination of different noise reduction modules in the image processing system may perform scheduling and noise reduction processing on the captured image.

It can be understood that the electronic device in the embodiment of the present application may refer to any one of various types of electronic devices capable of performing shooting, that is, the electronic device has a camera. For example, the electronic device may be a mobile phone or a smart phone (e.g., may be an iPhone (TM) -based phone or an Android (TM) -based phone), a personal digital assistant (personal digital assistant, PDA), a portable gaming device, a laptop, a drone, a robot, an in-vehicle device, or an electric car, etc.

The noise reduction module is not particularly limited in the embodiment of the application. For example, the noise reduction module may be any noise reduction module in an image signal processor (Image Signal Processor, ISP) (e.g., a 3d noise reduction module (3 DNR), a time domain noise reduction (2 TNR), a mean filtering noise reduction module, a bilateral filtering noise reduction module, etc.), and/or any noise reduction module in an embedded Neural Network processor (Neural-Network Processing Units, NPU) (e.g., an AI noise reduction module (AI Noise Reduction, AINR) or a recursive Self-guided Network-based noise reduction module (RSGN)).

As described above, the photographed application scenes of the electronic device are very abundant, and when the noise reduction processing is performed on the photographed image corresponding to different application scenes, different noise reduction modules or combinations of different noise reduction modules may be required to perform the noise reduction processing on the image. Particularly, when shooting is video shooting, as the video images comprise multiple frames of images, and the multiple frames of images are arranged according to a certain time sequence, in order to effectively noise reduction of the video images, a noise reduction module or a combination of the noise reduction modules is needed to schedule and noise reduction treatment on the video in order. And when the switching from one application scene to another application scene is needed, the corresponding noise reduction module needs to be quickly switched and scheduled.

In order to enable the image processing system to meet the requirement of effectively reducing noise for different application scenes, one feasible way is to schedule and reduce noise of video images by using Firmware (Firmware) running on each noise reduction module to control the noise reduction modules required by different application scenes. It should be noted that, in the embodiment of the present application, the firmware may be system firmware, which refers to a program written in a memory.

However, since the firmware itself needs to control other driving devices in the system of the electronic device, when the firmware controls the noise reduction module to schedule the video image, on one hand, many modifications in the program are required, and on the other hand, it is sometimes required to wait until the firmware controls the other driving devices in time sequence and then controls the processing of the noise reduction module (i.e. needs to wait). Particularly, when switching and denoising are performed on different application scenes of the video image, the scheduling process is particularly complex, and real-time performance and flexibility cannot be ensured.

Alternatively, the noise reduction module in the image processing system may include a noise reduction module in the NPU, such as the AINR or RSGN noise reduction module described above, when a dedicated NPU is required for image processing. At this time, each noise reduction module may be scheduled by a scheduling mechanism of the NPU. However, since the self-scheduling mechanism of the NPU is based on a general application scenario and not specific to various different application scenarios, the scheduling mechanism of the NPU needs to be modified for each application scenario. Because some other noise reduction modules are possibly located in the ISP, when the switching noise reduction is carried out on different application scenes of the video image by modifying the scheduling mechanism of the NPU, the scheduling process is more complex than the scheduling by the firmware, and the unified management is not good.

In order to solve the above problems, an embodiment of the present application provides an image processing system, in which a controller is provided, and noise reduction modes corresponding to multiple application scenes are configured in the controller, and different noise reduction modes correspond to different image processing modules including at least one noise reduction module. When the image processing system is used for denoising video images under different application scenes, the controller can be used for quickly enabling the relevant denoising module to schedule and denoise the video images according to preset rules. The scheduling process is relatively simple, and the method has high instantaneity and flexibility when switching and denoising are performed under different scenes of the video image.

An image processing system 10 in accordance with an embodiment of the present application is described in detail below with reference to fig. 1.

Referring to fig. 1, an image processing system 10 may include an image processing module 12 and a controller 14.

The image processing module 12 may include at least one noise reduction module 121 for performing noise reduction processing on the image through the at least one noise reduction module 121. The noise reduction module 121 in the image processing module 12 may be any one or more of the noise reduction modules described above.

Optionally, as shown in fig. 1, the image processing module 12 may further include some preprocessing modules 122, for example, a frame alignment module (simply referred to as Homo) or other types of correction modules (such as color correction, etc.), through which some preprocessing operations may be performed on the image to facilitate the subsequent noise reduction process.

In an embodiment of the present application, the image processing module 12 may be variable, i.e. the image processing module 12 is different in different application scenarios. The differences in the image processing modules 12 may be embodied in that the composition of the noise reduction modules in the processing modules 12 that participate in image noise reduction are different. In other words, the plurality of noise reduction modules 121 may be included in the overall image processing module frame, and the plurality of noise reduction modules 121 may be different types of noise reduction modules or portions thereof may be different types of noise reduction modules. The noise reduction modules 121 may not participate in the noise reduction process at the same time, and any one or more noise reduction modules are selected from the plurality of noise reduction modules 121 to perform the noise reduction process on the video image in the corresponding application scene in different application scenes, so that the noise reduction modules 121 participating in the noise reduction process form the image processing module 12 corresponding to the application scene.

For example, when the image processing module performs simple noise reduction processing on the video images captured by the two cameras, only one noise reduction module 121 (for example, the above 3DNR or RSGN) may participate in noise reduction, and at this time, the image processing module 12 is formed by one noise reduction module. For another example, when the image processing module performs complex noise reduction processing on a video image captured by a camera, two or three equal noise reduction modules (for example, the above-mentioned ainr+3dnr or 2tnr+ainr+3dnr) participate in the noise reduction processing, where the image processing module 12 is formed by two or three equal noise reduction modules.

A plurality of noise reduction modes are configured within the controller 14. The plurality of noise reduction modes are in one-to-one correspondence with the plurality of application scenarios, and the plurality of noise reduction modes are associated with the image processing module 12. As previously described, the image processing module 12 may be variable, i.e., the image processing module 12 may take a variety of forms, and the association of the plurality of noise reduction modes with the image processing module 12 is embodied as: the image processing modules 12 to which the plurality of noise reduction modules respectively correspond are different.

It should be noted that, the controller 14 in the embodiment of the present application may refer to a hardware controller. The hardware controller is internally configured with a plurality of noise reduction modes, namely a plurality of noise reduction modes corresponding to a plurality of application scenes are preconfigured in the hardware controller. The configured noise reduction mode not only indicates the corresponding relation between the application scene and the image processing module 12, that is, indicates which noise reduction modules 121 are required to participate in the noise reduction processing work under a certain application scene, but also indicates the processing rules of the noise reduction modules 121 corresponding to the application scene when participating in the noise reduction processing, for example, the working time and the sequence of the noise reduction modules 121, the language representation when the noise reduction modules 121 interact with the controller 14, and/or which storage units the noise reduction modules 121 need to read from during the processing, write to those storage units, and so on. That is, the hardware controller 14 may perform unified scheduling management on the noise reduction modules 121 participating in the noise reduction operation, so as to complete the starting, information transfer, state feedback and related operations in the exception handling process of each noise reduction module 121.

The setting of the application scene and the specific noise reduction module 121 in the image processing module 12 corresponding to the application scene are not specifically limited, and the corresponding noise reduction module 121 or the combination of the noise reduction modules 121 can be set according to the requirements of the actual application scene.

As an example, the following 8 application scenarios and their corresponding noise reduction modules 121 may be configured in the controller 14: the method comprises the steps of applying scene 1, and receiving video images +2TNR+AINR+2TNR+3DNR of two cameras; the method comprises the steps of applying a scene 2, and receiving a video image +2TNR+AINR+3DNR of a camera; an application scene 3, which receives video images of two cameras +AINR +3DNR; application scenario 4: receiving a video image +AINR+3DNR of a camera; application scenario 5: receiving video images +RSGN of two cameras; application scenario 6: receiving a video image +RSGN of a camera; application scenario 7: receiving video images +3DNR of two cameras; application scenario 8: and receiving the video image +3DNR of one camera.

In the embodiment of the present application, the controller 14 may be specifically used in practical applications: determining a target noise reduction mode according to the requirements of an application scene; and at least one noise reduction module 121 in the image processing module 12 corresponding to the target noise reduction mode control performs scheduling and noise reduction processing on the video image shot by the camera according to a preset rule.

The requirements of the application scenario may be from other systems or modules in the electronic device that have an interactive relationship with the image processing system 10, or from modules within the electronic device that require image processing, such as third party applications or camera sensor modules. The preset rule may be understood as the processing rule of the noise reduction module 121 to be involved in the noise reduction processing work described above.

By means of the image processing system, when the video images in different application scenes are subjected to noise reduction, the relevant noise reduction module 121 can rapidly schedule and reduce the video images according to preset rules through the controller 14, so that the scheduling process is relatively simple, and the real-time performance and the flexibility are high when the switching noise reduction is performed in different scenes of the video images.

In particular, for the cases described below, the image processing system provided by the embodiments of the present application has very outstanding real-time performance and flexibility.

As described above, in order to enrich the shooting function of the electronic device to meet the frame rate or resolution requirement of the photographer for image quality, in some embodiments, shooting may be performed by providing a plurality of cameras. Considering that the noise reduction module occupies an area in the related processor, if the corresponding noise reduction module 121 is imaged by each camera participating in the shooting at the same time, the area of the noise reduction module 121 will consume a great deal of power. In view of this, the video image may be scheduled for transmission on at least one of the noise reduction modules in the image processing module, typically in a time division multiplexed manner.

In this case, each noise reduction module 121 corresponding to a different application scenario needs to schedule the video image according to very strict timing requirements. If the implementation scheduling of the video image is implemented by the system firmware or by modifying the scheduling mechanism of the NPU itself, the operation process will be very complex and the latency will be very high.

The controller in the embodiment of the application can avoid the problem, and the controller in the embodiment of the application can uniformly control the functions of starting, information transmission, state feedback, exception handling and the like of each noise reduction module, so that the controller can be separated from the scheduling coordination of the system firmware when the noise reduction processing is carried out on the video image. In addition, the controller takes the noise reduction module in the NPU as an off-line module which is the same as the noise reduction module in the ISP, so that the controller has the characteristics of special use, convenience, high efficiency and low risk, and realizes the unified management of each noise reduction module. That is, the controller is used as the most core real-time scheduling control center, and can ensure real-time switching multiplexing of different frame rates and different resolutions among different cameras, thereby ensuring high frame rate and low delay requirements of video requirements.

As previously described, the preset rule may include a variety of contents. Alternatively, the preset rule may include a processing order and a start condition of at least one noise reduction module of the image processing modules 12, and the at least one noise reduction module 121 may be understood as a noise reduction module participating in a noise reduction processing job of the image processing modules. The specific process of the controller 14 controlling the noise reduction module 121 may be: determining a corresponding image processing module according to the target noise reduction mode; receiving state information sent by at least one noise reduction module in the corresponding image processing modules; and controlling at least one noise reduction module to schedule and reduce noise of the video images shot by the cameras according to the processing sequence according to whether the state information meets the starting condition.

The status information may be information indicating the operation status of the noise reduction module 121 at the current time, for example, whether the last frame of video image that is being processed by the current noise reduction module 121 has been processed, or where the noise reduction module writes the data transmitted according to time division multiplexing, how much data is written in the current noise reduction module 121, and which camera video image the noise reduction module 121 is processing, and so on.

The start condition is a condition for characterizing that the noise reduction module 121 can respond to a start instruction of the controller 14. Specifically, the controller 14 may determine whether it satisfies the start condition according to the state information of the noise reduction module 121, and if so, send a start instruction to the noise reduction module 121. For example, the start condition may be how much data is required for each start of the noise reduction module 121, or/and whether the start of the noise reduction module 121 requires processing of a next frame after the processing of the next frame.

The process of controlling the noise reduction module 121 by the controller 14 is exemplarily described below with reference to fig. 2 and 3.

Referring to fig. 2, the base frame of the image processing module 12 may include the following: two preprocessing modules 122 and three noise reduction modules 121. Illustratively, the preprocessing module 122 is the same as described above, and the three noise reduction modules 121 are 2TNR, AINR, and 3DNR, respectively, from left to right. The Homo module can comprise 2 modules, and can receive information of the image pickup sensor and perform online processing. Whereas 2TNR, AINR and 3DNR can perform off-line processing on video images transmitted from Homo.

Assuming that the above modules schedule and process video images according to the above arrangement sequence, the controller will pre-configure the processing sequence of video images to be first Homo, then 2TNR, AINR and 3DNR in sequence.

And a scheduling result timing diagram of the two paths of cameras is shown in combination with fig. 3. It should be noted that, the preprocessing modules 122 in fig. 2 and 3 are respectively from top to bottom, and the noise reduction modules 121 are respectively from top to bottom, and are respectively 2TNR, AINR, and 3DNR. Taking two cameras as an example, the main camera is 30fps, the auxiliary camera is 10fps, and when video images of the two cameras are respectively transmitted to the image processing system in the embodiment of the present application through a first sensor (denoted as sensor_0 and corresponding to the main camera) and a second sensor (denoted as sensor_1 and corresponding to the auxiliary camera), the following steps are used by the controller 14:

step 1: two homos (wherein homo0 corresponds to sensor_0 and homo1 corresponds to sensor_1) are controlled to perform frame alignment work. In addition, homo sends status information (e.g., which frame image (denoted as frame cnt) is processed, where data is written (denoted as Ring cnt→e.g., in the A, B, C/a, b, c modules written in memory in fig. 2 and 3) and which time division multiplexed pipeline (denoted as Line cnt) is processed in real time to the controller 14 during operation.

Step 2: the controller 14 judges the preset starting conditions of the Homo and the 2TNR according to the state information sent by the Homo and the 2TNR, and if the starting conditions are met, sends a starting instruction to the 2TNR to control the 2TNR to schedule and reduce noise on the video image at the Homo. Illustratively, the start condition may be that 2TNR requires 750 lines of current frame data to be processed at a time into memory while 2TNR is already processed for the previous frame image. In some embodiments, to ensure that the last frame of the primary + secondary (30 + 10) also satisfies 750 lines, the start condition may also include a line number that would require the first frame to start. The start command may also include data of which sensor is processed by the 2TNR, where to read from and write to after processing, the size of the current frame image, etc. When the 2TNR receives the start command, it starts to operate, and at the same time, status information (e.g., which frame of image is processed (denoted as frame cnt), where data is written (denoted as Ring cnt→e.g., D/D, E/E modules written in the memory in fig. 2 and 3) and which time division multiplexing pipeline is processed (denoted as Line cnt) may be sent to the controller 14 in real time.

Step 3: the controller 14 judges the preset starting condition of the AINR according to the state information sent by the 2TNR and the AINR, and if the starting condition is met, sends a starting instruction to the AINR to control the AINR to schedule and reduce noise of the video image of the 2 TNR. For example, the start condition may be that the AINR (NPU) requests that 300 lines in the current frame data be processed at a time into the NPU's buffer, while the AINR (NPU) has already processed the previous frame image. The start-up instruction may also include data of which sensor the AINR processes, where to read from and write to after processing, the size of the current frame image, etc. When the AINR receives the start command, it starts to operate, and at the same time, status information (e.g., which frame of image is processed (denoted as frame cnt), where data is written (denoted as Ring cnt. Fwdarw. For example, in the F/F module written in the buffer of the NPU in FIG. 3) and which time division multiplexing pipeline is processed (denoted as Line cnt) may be sent to the controller 14 in real time.

Step 4: the controller 14 determines the preset start condition of the 3DNR based on the status information transmitted from the AINR (NPU) and the 3 DNR. And if the starting condition is met, sending a starting instruction to the 3DNR to control the 3DNR to schedule and reduce noise of the AINR (NPU) video image. For example, the startup condition may be that the AINR (NPU) requires writing 200 lines into the NPU's buffer while the 3DNR (NPU) has been processed for the previous frame of image. The start-up instruction may also include data of which sensor is processed by the 3DNR, where to read from and write to after processing, the size of the current frame image, and the like. When the 3DNR receives the start command, it starts to operate, and at the same time, it can send status information (e.g. which frame of image is processed (denoted as frame cnt), where data is written (denoted as Ring cnt→e.g. G/G module written in the buffer of NPU in fig. 3) and which time division multiplexing pipeline is processed (denoted as Line cnt) to the controller 14 in real time.

In the above steps, the state information may be uniformly expressed as:

Status:

Frame cnt

Ring cnt

Line cnt

…

Eof

the startup instructions may be collectively expressed as:

Rd Buf Addr

Wr Buf Addr

Frame Size

Sensor Sel

...

Sof

as can be seen from fig. 3, the controller 14 controls the various modules in the image processing system 10 to operate in real time, orderly and with low latency with very little maximum delay. In addition, since the control language used in the scheduling process is uniform, the control process is relatively convenient and efficient.

In some embodiments, as shown in FIGS. 1-2 and 4-5, the controller 14 may also be used to communicate with the firmware 20 via a bus (HUB) to enable the hardware controller to control the noise reduction module 121 in the image processing module via information sent by the firmware 20. For example, before the controller 14 sends the start command to the 2TNR, some index data (e.g., LSQ) about the start condition may be calculated in the firmware 20, and after the calculation by the firmware 20, may be sent to the controller 14, so that the controller 14 may make further determination.

From the foregoing, it can be seen that, when at least one noise reduction module 121 in the image processing module 12 corresponding to the application scene may include both a first noise reduction module that is a noise reduction module in the embedded neural network processor and a second noise reduction module that is a noise reduction module in the image signal processor, the data of the video image may be read from or written to the memory in the system or may be read from or written to the buffer of the NPU. That is, in this case, the image processing system 10 may include a memory and a buffer 123 of the NPU as shown in FIGS. 2 and 4-5. The memory may be a general purpose memory in an electronic device, for example, a Double Data Rate (DDR) memory. And the buffer in the NPU may be a storage unit built into the neural network processor (e.g., a neural network processor System-buffer).

Correspondingly, the controller 14 can control the first noise reduction module and the second noise reduction module in the corresponding image processing modules according to the target noise reduction mode to schedule the video image shot by the camera through the buffer and the memory in the embedded neural network processor according to the preset rule.

This is explained below in connection with the application scenario of fig. 4 and 5.

The application scenario in fig. 4 is: the video images +2tnr+ainr+2tnr+3dnr of the two cameras are received, in fig. 4, three noise reduction modules 121 from left to right are 2TNR, AINR, and 3DNR, respectively, and two preprocessing modules 122 from top to bottom are homo0 and homo1. In this application scenario, the working process of each module of the controller 12 is substantially identical to the foregoing example, and will not be described herein. In contrast, the data processed by the 2TNR may be written directly into the system buffer of the AINR (NPU), and the output of the AINR may be transmitted to the 3DNR through the system buffer of the AINR (NPU). In this application scenario, the method can also be used to schedule data of video images. However, if an obstruction is encountered in scheduling using this method, such as an AINR (NPU) system buffer that cannot support the corresponding data size, then the method may revert back to using the D space of memory to receive data written to the AINR and using the F space of memory to receive data read from the AINR. By the mode, on one hand, each storage unit in the system can be effectively utilized, and on the other hand, the storage power consumption of the memory can be effectively reduced.

The application scenario in fig. 5 is: video images +RSGN of two cameras are received. The noise reduction module 121 in FIG. 5 is RSGN, and the two preprocessing modules 122 from top to bottom are AVP/sf_0 and AVP/sf_1.AVP/sf_0 and AVP/sf_1 can be understood as a preprocessing module that processes a large image into a plurality of small images. As can be seen from fig. 5, in this application scenario, the memory spaces A, B, C/a, b, c of the memory and the memory spaces D, E/d, e in the system buffer 123 of the RSGN (NPU) can be divided into a plurality of smaller spaces.

When video images (which may be output by an ISP, for example) are input to the AVP/sf_0 and AVP/sf_1 modules simultaneously, both the AVP module and the Shuffle module in the AVP/sf_0 and AVP/sf_1 modules may output the preprocessed thumbnail. At this point, the binding store may be set so that the controller 14 manages a large space (i.e., A, B, C/a, b, c, discussed above), while the internal small map is read offline by the NPU's base address setting for a relatively large space. The base address may be the free memory space after A, B, C/a, b, c in FIG. 5.

In some embodiments, there will be some Hidden stages (Hidden State) in the RSGN module that will output a partial number of small figures, e.g., 4 small figures. The controller 14 can also manage large space by setting the binding store, while the NPU base address for relatively large space is set for offline reading.

In summary, all data entering the NPU (e.g., video image output by ISP, output data of AVP/SF module, output data of Hidden State) and data sent from the NPU (e.g., video image, output data of Hidden State) pass through the system buffer 123 of the NPU. That is, the NPU treats the system buffer 123 as an input first-in first-out (input fifo) and output first-in first-out (output fifo) memory. The reading sequence of the data streams of different channels is controlled well only by the NPU, and the data streams are subdivided in the NPU. In the embodiment of the application, the reading sequence of the NPU for controlling different channel data streams can be consistent with the reading requirement of the NPU frame, for example, reading according to a plurality of rows and a plurality of lines.

As described above, when the image processing system in the embodiment of the present application performs noise reduction processing on a video image, each stage in which data is read may be scheduling of a plurality of rows of pixels in a frame of image. That is, the controller 14 in the embodiment of the present application may be configured to: and controlling at least one noise reduction module in the corresponding image processing modules according to the target noise reduction mode, and scheduling and carrying out noise reduction processing on a plurality of rows of pixels of each frame of image in the multi-frame image according to a preset rule. In this way, row-level scheduling control can be realized, control effort is finer, and a memory (DDR) of a system is not required to be called, so that delay and power consumption are reduced.

An embodiment of the device of the present application is described in detail above in connection with fig. 1 to 5. An embodiment of the method of the present application is described in detail below in conjunction with fig. 6. It is to be understood that the description of the method embodiments corresponds to the description of the device embodiments, and that parts not described in detail can therefore be seen in the preceding device embodiments.

Fig. 6 is a schematic flowchart of an image processing method provided by an embodiment of the present application. The method may be implemented, for example, using a controller as described above. The controller is arranged in the image processing system, the image processing system further comprises an image processing module, the image processing module comprises at least one noise reduction module, the image processing module is used for carrying out noise reduction processing on an image through the at least one noise reduction module, a plurality of noise reduction modes are configured in the controller, the image processing modules corresponding to the plurality of noise reduction modes are different, and the plurality of noise reduction modes are in one-to-one correspondence with a plurality of application scenes.

Referring to fig. 6, in step S610, a target noise reduction mode is determined according to the requirements of an application scene.

In step S620, at least one noise reduction module in the image processing modules corresponding to the target noise reduction mode control performs scheduling and noise reduction processing on the video image shot by the camera according to a preset rule.

Optionally, the preset rule includes a processing sequence and a starting condition of at least one noise reduction module in the image processing modules, and the controlling, according to the target noise reduction mode, the at least one noise reduction module in the corresponding image processing modules to schedule and reduce the noise of the video image shot by the camera according to the preset rule includes:

determining a corresponding image processing module according to the target noise reduction mode;

receiving state information sent by at least one noise reduction module in the image processing module;

and controlling at least one noise reduction module to schedule and reduce noise of the video images shot by the cameras according to the processing sequence according to whether the state information meets the starting condition.

Optionally, the video image includes a plurality of frames of images, and the scheduling and denoising processing for the video image shot by the camera according to a preset rule by at least one denoising module in the image processing modules corresponding to the target denoising mode control includes: and controlling at least one noise reduction module in the corresponding image processing modules according to the target noise reduction mode, and scheduling and carrying out noise reduction processing on a plurality of rows of pixels of each frame of image in the multi-frame image according to a preset rule.

Optionally, the number of cameras includes a plurality of cameras, and video images scheduled by at least one noise reduction module in the image processing modules are transmitted in a time division multiplexing manner.

Optionally, the at least one noise reduction module includes a first noise reduction module, and the first noise reduction module is a noise reduction module in the embedded neural network processor.

Optionally, the image processing system further comprises a memory, wherein the memory is used for reading and writing video images, the at least one noise reduction module further comprises a second noise reduction module, the second noise reduction module is a noise reduction module in the image signal processor, the at least one noise reduction module in the image processing module corresponding to the target noise reduction mode is controlled to schedule the video images shot by the camera according to a preset rule, and the first noise reduction module and the second noise reduction module in the image processing module corresponding to the target noise reduction mode are controlled to schedule the video images shot by the camera according to the preset rule through a buffer and the memory in the embedded neural network processor respectively.

Optionally, the controller is a hardware controller, the hardware controller further configured to communicate with firmware, the method further comprising: and controlling the image processing module in response to the information sent by the firmware.

Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The apparatus 700 shown in fig. 7 may be an electronic device capable of performing the above-described image processing method. The apparatus 700 may be a receiving end in a communication system, and may be, for example, a User Equipment (UE), an access Terminal, a subscriber unit, a subscriber station, a Mobile Station (MS), a Mobile Terminal (MT), a remote station, a remote Terminal, a mobile device, a user Terminal, a wireless communication device, a user agent, a user equipment, or the like. The apparatus 700 may comprise a processor 710 and a memory 720, wherein the memory 720 has stored therein instructions executable by the processor 710, which when executed by the processor 720, implement the steps of the respective methods described hereinbefore. In some implementations, the electronic device 700 may also include a network interface 730, and data exchange by the processor 720 with external devices may be accomplished through the network interface 730.

The embodiment of the present application also provides a computer-readable storage medium having stored thereon a program which, when executed by a processor, implements the steps of the respective methods described above.

It should be understood that in embodiments of the present application, "B corresponding to a" means that B is associated with a, from which B may be determined. It should also be understood that determining B from a does not mean determining B from a alone, but may also determine B from a and/or other information.

It should be understood that the term "and/or" is merely an association relationship describing the associated object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.

It should be understood that, in various embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application.

In the several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and apparatuses may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be read by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disk (digital video disc, DVD)), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.

The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (15)

1. An image processing system, comprising:

the image processing module comprises at least one noise reduction module and is used for carrying out noise reduction processing on the image through the at least one noise reduction module;

the controller is internally provided with a plurality of noise reduction modes, the image processing modules corresponding to the plurality of noise reduction modes are different, the plurality of noise reduction modes are in one-to-one correspondence with a plurality of application scenes, and the controller is used for:

determining a target noise reduction mode according to the requirements of an application scene;

and controlling the at least one noise reduction module in the corresponding image processing modules according to the target noise reduction mode, and scheduling and carrying out noise reduction processing on the video image shot by the camera according to a preset rule.

2. The system of claim 1, wherein the preset rules include a processing order and a start condition of the at least one of the image processing modules, the controller to:

Determining the corresponding image processing module according to the target noise reduction mode;

receiving state information sent by at least one noise reduction module in the image processing module;

and controlling the at least one noise reduction module to schedule and reduce noise of video images shot by the camera according to the processing sequence according to whether the state information meets the starting condition.

3. The system of claim 1, wherein the video image comprises a multi-frame image, the controller to:

and controlling the at least one noise reduction module in the corresponding image processing module according to the target noise reduction mode, and scheduling and carrying out noise reduction processing on a plurality of rows of pixels of each frame of image in the multi-frame image according to a preset rule.

4. The system of claim 1, wherein the number of cameras comprises a plurality, and wherein the video images scheduled by the at least one noise reduction module in the image processing module are transmitted in a time division multiplexed manner.

5. The system of claim 1, wherein the at least one noise reduction module comprises a first noise reduction module, the first noise reduction module being a noise reduction module in an embedded neural network processor.

6. The system of claim 5, wherein the system further comprises: the memory is used for reading and writing the video image;

the at least one noise reduction module further comprises a second noise reduction module, the second noise reduction module is a noise reduction module in the image signal processor, and the controller is further configured to:

and controlling the first noise reduction module and the second noise reduction module in the corresponding image processing modules according to the target noise reduction mode, and scheduling video images shot by the cameras through a buffer and the memory in the embedded neural network processor according to preset rules.

7. The system of claim 1, wherein the controller is a hardware controller further configured to communicate with firmware to cause the hardware controller to control the image processing module via information sent by the firmware.

8. An image processing method, wherein the method is applied to a controller, the controller is located in an image processing system, the image processing system further comprises an image processing module, the image processing module comprises at least one noise reduction module, the image processing module is used for performing noise reduction processing on an image through the at least one noise reduction module, a plurality of noise reduction modes are configured in the controller, the image processing modules corresponding to the plurality of noise reduction modes are different, and the plurality of noise reduction modes are in one-to-one correspondence with a plurality of application scenes, and the method comprises:

Determining a target noise reduction mode according to the requirements of an application scene;

and controlling the at least one noise reduction module in the corresponding image processing modules according to the target noise reduction mode, and scheduling and carrying out noise reduction processing on the video image shot by the camera according to a preset rule.

9. The method according to claim 8, wherein the preset rule includes a processing sequence and a start condition of the at least one noise reduction module in the image processing modules, and the controlling the at least one noise reduction module in the corresponding image processing module according to the target noise reduction mode includes scheduling and performing noise reduction processing on a video image captured by the camera according to the preset rule:

determining the corresponding image processing module according to the target noise reduction mode;

receiving state information sent by at least one noise reduction module in the image processing module;

and controlling the at least one noise reduction module to schedule and reduce noise of video images shot by the camera according to the processing sequence according to whether the state information meets the starting condition.

10. The method according to claim 8, wherein the video image includes a plurality of frames of images, and the at least one noise reduction module of the image processing modules corresponding to the target noise reduction mode control performs scheduling and noise reduction processing on the video image captured by the camera according to a preset rule, including:

And controlling the at least one noise reduction module in the corresponding image processing module according to the target noise reduction mode, and scheduling and carrying out noise reduction processing on a plurality of rows of pixels of each frame of image in the multi-frame image according to a preset rule.

11. The method of claim 8, wherein the at least one noise reduction module comprises a first noise reduction module, the first noise reduction module being a noise reduction module in an embedded neural network processor.

12. The method of claim 11, wherein the image processing system further comprises a memory for reading and writing the video image, the at least one noise reduction module further comprises a second noise reduction module, the second noise reduction module is a noise reduction module in an image signal processor, and the controlling the at least one noise reduction module in the corresponding image processing module according to the target noise reduction mode includes scheduling the video image captured by the camera according to a preset rule:

and controlling the first noise reduction module and the second noise reduction module in the corresponding image processing modules according to the target noise reduction mode, and scheduling video images shot by the cameras through a buffer and the memory in the embedded neural network processor according to preset rules.

13. The method of claim 8, wherein the controller is a hardware controller, the hardware controller further configured to communicate with firmware, the method further comprising:

and controlling the image processing module in response to the information sent by the firmware.

14. An electronic device, comprising:

a processor;

a memory for storing executable instructions of the processor;

wherein the processor is configured to implement the method of any of claims 8-13 via the executable instructions.

15. A computer readable storage medium comprising a computer program which, when executed by a processor, implements the method of any of claims 8-13.