WO2021102947A1

WO2021102947A1 - Image signal processing apparatus and method, camera, and mobile platform

Info

Publication number: WO2021102947A1
Application number: PCT/CN2019/122084
Authority: WO
Inventors: 曾志豪; 曹子晟
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2019-11-29
Filing date: 2019-11-29
Publication date: 2021-06-03
Also published as: CN112514364A

Abstract

Provided in the present application are an image signal processing apparatus and method, a camera, and a mobile platform. The apparatus comprises a pixel classification module and at least two first image signal processing modules; the pixel classification module is used for classifying each pixel in an input image, generating a plurality of pieces of classification information, and transmitting the input image and the plurality of pieces of classification information into a first image signal processing module; the classification information is used for characterizing an image feature of the pixel; the first image signal processing module is used for executing, based on the plurality of pieces of classification information, corresponding image processing operations on the pixels in the input image, and transmitting a generated image and the plurality of pieces of classification information to the next first image signal processing module. The present embodiment can reduce the same or similar steps repeated by the first image signal processing modules, and reduce the hardware cost.

Description

Image signal processing device, method, camera and movable platform

Technical field

This application relates to the field of image processing, and in particular to an image signal processing device, method, camera, and movable platform.

Background technique

In various shooting devices (such as digital cameras, camcorders or camera phones, etc.), the raw image data collected by the image sensor is usually processed by the image signal processor (Image Signal Processor, ISP). The ISP includes multiple functional modules, such as sensors. The correction module, color correction module, etc., finally present an image that is visible to the naked eye. Among them, the processing level of the ISP largely determines the image quality. In the traditional ISP, each functional module is relatively independent and realizes their respective functions. However, some modules have similar or the same basic operations. The relative independence between the modules makes the similar or identical basic operations between the modules need to be repeated, and there are many Multiple hardware resources are repeatedly implemented, resulting in high hardware costs and long image signal processing time.

Summary of the invention

In view of this, the present application provides an image signal processing device, method, camera, and movable platform.

First, the first aspect of the present application provides an image signal processing device, including a pixel classification module and at least two first image signal processing modules;

After the pixel classification module, at least two first image signal processing modules are connected in sequence;

The pixel classification module is used to classify each pixel in the input image, generate a number of classification information, and transmit the input image and a number of classification information to the first image signal processing module connected to it; The classification information is used to characterize the image characteristics of the pixel;

The first image signal processing module is configured to receive the image input by the pixel classification module or the previous first image signal processing module and the plurality of classification information; based on the plurality of classification information, the pixels in the input image Perform a corresponding image processing operation, and transmit the generated image and the plurality of classification information to the next first image signal processing module.

According to a second aspect of the embodiments of the present application, an image signal processing method is provided, which is applied to an image signal processing device, and the method includes:

In the pixel classification module, each pixel in the input image is classified to generate a number of classification information; the classification information is used to characterize the image characteristics of the pixel;

In the first image signal processing module, the image input by the pixel classification module or the previous first image signal processing module and the plurality of classification information are received, and based on the plurality of classification information, the pixels in the input image Perform a corresponding image processing operation, and transmit the generated image and the plurality of classification information to the next first image signal processing module.

According to a third aspect of the embodiments of the present application, a camera is provided, including:

shell;

The lens assembly is arranged inside the housing;

An image sensor, arranged inside the housing, for sensing light passing through the lens assembly and generating an electrical signal; and,

The image signal processing device according to any one of the first aspect.

According to the fourth aspect of the embodiments of the present application, a movable platform is also provided, including:

Body

A power system installed in the body for powering the movable platform; and, the camera according to the third aspect.

The technical solutions provided by the embodiments of the present application may include the following beneficial effects:

The image signal processing device in the embodiment of the present application includes a pixel classification module and at least two first image signal processing modules. The pixel classification module classifies each pixel in the input image to generate a number of classification information. The classification information represents the image feature of the pixel, and the generated classification information is transmitted to the first image signal processing module, so that the first image signal processing module does not need to repeat the image feature determination step of the same or similar pixels, which can Determining the image processing operations that each pixel should perform directly based on the several classification information reduces the hardware resources for performing the same or similar steps, thereby effectively reducing hardware costs, and the deletion of repeated steps also effectively reduces the length of image signal processing. Improve the efficiency of image signal processing.

It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the application.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained from these drawings without creative labor.

Fig. 1 is a structural diagram of a first image signal processing device 10 according to an exemplary embodiment of the present application.

Fig. 2 is a structural diagram showing a second image signal processing device 10 according to an exemplary embodiment of the present application.

FIG. 3 is a structural diagram showing a third image signal processing device 10 according to an exemplary embodiment of the present application.

Fig. 4 is a structural diagram showing a fourth image signal processing device 10 according to an exemplary embodiment of the present application.

FIG. 5 is a structural diagram showing a fifth image signal processing device 10 according to an exemplary embodiment of this application.

Fig. 6 is a structural diagram showing a sixth image signal processing device 10 according to an exemplary embodiment of the present application.

Fig. 7 is a flowchart of an image signal processing method according to an exemplary embodiment of this application.

Fig. 8 is a structural diagram of a camera according to an exemplary embodiment of the present application.

Fig. 9 is a structural diagram of a movable platform according to an exemplary embodiment of this application.

Detailed ways

The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and/or" as used herein refers to and includes any or all possible combinations of one or more associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, the information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determination". The terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements that are not explicitly listed. Elements, or also include elements inherent to such processes, methods, articles, or equipment. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, article, or equipment that includes the element.

To solve the problems of high hardware cost and long image signal processing time caused by the relative independence of each functional module in the traditional ISP, an image signal processing device 10 is provided in the embodiment of the present application, and the image signal processing device 10 combines each function The same or similar operations among the modules are integrated into a pixel classification module 11, which avoids the repeated execution process and does not require excessive hardware resources, reduces hardware costs, and also shortens the image signal processing time; wherein, the image The signal processing device 10 can be applied to fields that require processing of captured image signals, such as cameras, mobile terminals (such as mobile phones, tablets, or computers), vehicle-specific cameras, monitors, educational equipment, or medical equipment.

Please refer to FIG. 1, which shows a structural diagram of a first image signal processing apparatus 10 according to an exemplary embodiment of this application. The device includes: a pixel classification module 11 and at least two first image signal processing modules 12 (in FIG. 1, two first image signal processing modules 12 are taken as an example for illustration); wherein, after the pixel classification module 11 , At least two first image signal processing modules 12 are connected in sequence.

The pixel classification module 11 is used to classify each pixel in the input image, generate some classification information, and transmit the input image and some classification information to the first image signal processing module 12 connected to it; The classification information is used to characterize the image characteristics of the pixel.

It is understandable that this embodiment does not impose any restrictions on the representation of the classification information, and can be specifically set according to actual conditions. In an example, the classification information can be expressed in the form of a hash value or a hash vector. The hash value includes, but is not limited to, an integer value and a floating point value, and the hash vector includes, but is not limited to, an integer vector, a floating point vector, etc.; in one example, for example, the classification information may be an integer value Sequence: 01011010101010 or 1253716581651, etc., each position can represent a certain type of image feature, and the specific value at this position can represent the specific meaning of this type of image feature. For example, the image feature includes edge features and flat area features. 1 position represents the edge feature, where the value part "0" means there is an edge, "1" means no edge; the second position means the flat area feature, where the value part "0" means the flat area, and "1" means the non-flat area , If the classification information is "01", it means that the image feature of the pixel is: there is an edge and a non-flat area.

In this embodiment, considering that most of the first image signal processing module 12 in the image signal processing link needs to know the image characteristics of each pixel in the input image to decide to perform the corresponding image processing operation on the pixel, and the first The process of repeatedly determining image features in the image signal processing module 12 not only lengthens the image signal processing time, but the repeated hardware resources also increase the hardware cost. Based on this, the embodiment of the present application sets all the image signals in the image signal processing device 10 The pixel classification module 11, the pixel classification module 11 classifies each pixel in the input image, and generates classification information for each pixel. The classification information represents the image characteristics of the pixel. This embodiment is for the image The features are not limited and can be specifically set according to the application scenario. For example, the image features can include at least one or more of the following: edge strength features, edge direction features, flat area features, texture features, color features, isolated point features And domain features; as an example, for example, the pixels in the image are divided into 3 categories, the first category represents the pixel is non-edge, in a flat area, no texture and non-isolated points, the second category represents the pixel is a weak edge and 0° direction, non-flat area, textured and non-isolated dots; 3 types characterize that the pixel is a strong edge and 90° direction, non-flat area, textured and non-isolated dots, etc.; in this embodiment, the pixel classification module 11 The pixels are classified and the generated classification information is transmitted to the first image signal processing module 12, so that the first image signal processing module 12 does not need to repeat the same or similar image feature determination process, which can be directly based on the several image features. The classification information determines the image processing operations that each pixel should perform.

In a possible implementation manner, the pixel classification module 11 classifies each pixel in the input image based on the pixel and the neighboring pixels of the pixel, and generates classification information of the pixel. The information is used to characterize the image characteristics of the pixel. This embodiment does not impose any restrictions on the image characteristics, and can be specifically set according to the actual situation. For example, the image characteristics may be edge strength characteristics, edge direction characteristics, flatness, and texture. Richness, whether there are isolated points, and neighborhood features, etc.; wherein, the neighborhood feature can be an image feature of a designated neighborhood, for example, it can be an image feature of a small area (such as 3×3) and a large neighborhood (such as 7 ×7) image features; as an example, the pixel classification module 11 may classify the pixels and the gray changes of the pixels in the neighborhood of the pixel to generate classification information of the pixel.

In an example, the pixel classification module 11 may calculate the horizontal and vertical gradient matrices of each pixel based on the pixels and the neighboring pixels of the pixel, and then determine the gradient direction of the pixel based on the gradient matrix. , Amplitude and correlation, and then classify according to the gradient direction, amplitude and correlation to obtain the classification information of the pixel. For example, the input image is divided into N (N≥1) categories in total, and each category uses a different Ha Greek value or hash vector representation, assuming that the gradient direction of one of the pixels is t, the amplitude is s, and the correlation is c, the necessary and sufficient conditions for the pixel block to belong to the Kth (1≤K≤N) category are: tl _k ≤f(t)<tr _k and sl _k ≤f(s)<sr _k and cl _k ≤f(c)<cr _k , where tl _k , tr _k , sl _k , sr _k , cl _k , cr _k is a preset parameter, which can be specifically set according to the actual situation. This embodiment does not impose any restrictions on this. f(t), f(s), f(c) are respectively the gradient direction, amplitude and correlation degree. function.

In another possible implementation manner, the pixel classification module 11 classifies each pixel in the input image based on the pixel, the neighboring pixels of the pixel, and specified image information, and generates the pixel's Classification information, the classification information is used to characterize the image characteristics of the pixel; wherein, the embodiment of the present application does not impose any restriction on the image information, and can be specifically set according to actual conditions. For example, the image information may include but is not limited to Exposure parameters such as sensitivity (ISO value).

The first image signal processing module 12 is configured to receive the image input by the pixel classification module 11 or the previous first image signal processing module 12 and the plurality of classification information; based on the plurality of classification information, the input image The pixels in the image processing unit perform corresponding image processing operations, and transmit the generated image and the plurality of classification information to the next first image signal processing module 12.

In an example, the first image signal processing module 12 includes but is not limited to: a dead pixel correction module, a black level correction module, a shadow correction module, a white balance correction module, a demosaicing module, a color correction module, and a brightness adjustment module , A noise reduction module and a sharpening module; the dead pixel correction module is used to eliminate pixels in the pixel array that are significantly different from the surrounding pixel points; the black level correction module is used to subtract from the input image signal The dark current signal is removed; the shadow correction module is used to compensate for the brightness loss of surrounding pixels; the white balance correction module is used to remove the influence of ambient light; the demosaicing module is used to reconstruct a complete color; the color correction module It is used to correct color deviation; the brightness adjustment module is used to adjust overall or partial brightness; the noise reduction module and the sharpening module are used to restore relevant details of the image.

Wherein, the first image signal processing module 12 stores the correspondence between classification information and image processing operations. In an example, in a first image processing module, for example, the pixels in the image are divided into two categories. Type 1 corresponds to a 3×3 convolution kernel, and type 2 corresponds to a 6×6 convolution kernel; it should be noted that the correspondence between the classification information stored in the different first image signal processing module 12 and the image processing operation is also different. Different, specific settings can be made according to the actual image processing function of the first image signal processing module 12; the first image signal processing module 12 receives the pixel classification module 11 or the previous first image signal processing module 12 After the input image and the plurality of classification information, the plurality of classification information is used as an index, the image processing operation corresponding to each pixel in the input image is obtained from the corresponding relationship, and the image processing operation is executed to generate The image sent to the next first image signal processing module 12, and the first image signal processing module 12 sends the plurality of classification information along with the generated image to the next first image signal processing module 12, so that A first image signal processing module 12 can directly perform corresponding image processing operations on the input image based on the plurality of classification information.

In this embodiment, the several classification information is transmitted to the first image signal processing module 12 in parallel along with the image, so that the first image signal processing module 12 does not need to repeat the same or similar steps, which can be directly based on the several The classification information performs corresponding image processing operations on each pixel in the input image, which reduces hardware resources for the same or similar steps, thereby effectively reducing hardware costs, and the reduction of repeated steps also effectively reduces the length of image signal processing. Improve the efficiency of image signal processing.

It is understandable that this application does not impose any restrictions on the storage method of the corresponding relationship, and can be specifically set according to actual conditions. For example, the corresponding relationship may be stored in a hash table format, and the hash table includes one or A plurality of key-value pair relationships, wherein the classification information is used as a hash key, and the image processing operation is stored as a key-value pair relationship of the hash value; wherein, the key-value pair relationship can be designed based on human experience, It can also be obtained through an intelligent learning algorithm. The intelligent learning algorithm can be a machine learning algorithm such as a random forest model, a decision tree model, etc., a deep learning algorithm such as a neural network model, etc., or other algorithms such as least squares. Etc., in an example, a key-value pair relationship sample (including a hash key sample and a hash value sample) can be obtained, and the hash key sample in the key-value pair relationship sample can be input into a specified model (such as a random forest model) In the process, the prediction result of the designated model is obtained, and the parameters of the designated model are adjusted according to the difference between the prediction result of the designated model and the hash value samples in the key-value pair relationship sample to obtain the training completed Model, so that any hash key can be input into the trained model to obtain the corresponding hash value, thereby obtaining the key-value pair relationship, thereby effectively reducing the manual debugging process and improving the development efficiency.

In an embodiment, different classification information corresponds to different image processing operations. The image processing operations include, but are not limited to, filters of different scales and different types of image processing functions. As an example, the first image signal processing module 12 is an image denoising module. The image denoising module achieves the purpose of image smoothing through low-pass filtering. For each pixel in the input image, based on its corresponding classification information, the corresponding smoothing operator is obtained. The classification information corresponds to different smoothing operators. For example, if the classification information indicates that the pixel is in a flat area, the corresponding image processing operation may be a non-directional smoothing operator; if the classification information indicates that the pixel is in the texture area, then The corresponding image processing operation can be a directional smoothing operator.

It should be noted that the embodiment of the present application does not impose any restriction on the specific position of the pixel classification module 11 in the image signal processing device 10, and specific settings can be made according to actual conditions.

In an embodiment, please refer to FIG. 2, which shows a structure diagram of a second image signal processing device 10 according to an exemplary embodiment of this application. The pixel classification module 11 is located in the image signal processing chain of the image signal processing device 10 At the top of the road, the pixel classification module 11 is directly connected to the image sensor 20, and the image sensor 20 is used to collect images and transmit them to the pixel classification module 11, so that the pixel classification module 11 can analyze the input image Classify each pixel of the pixel to generate classification information corresponding to each pixel, and the classification information is used to characterize the image characteristics of the pixel, so that the subsequent first image signal processing module 12 does not need to repeat the same or similar steps; wherein The image sensor 20 may be a CMOS image sensor 20 or a CCD image sensor 20, and the image sensor 20 converts the captured light source signal into a digital signal to complete image collection.

In another embodiment, please refer to FIG. 3, which shows a structure diagram of a third image signal processing device 10 according to an exemplary embodiment of this application (in FIG. 3, two second image signal processing modules 13 are taken as an example. For example), the pixel classification module 11 may be placed after the second image signal processing module 13 that does not need to determine the image characteristics of the pixels, and the image signal processing device 10 further includes one second image signal processing module 13 or multiple sequential image signal processing modules 13 Connected to the second image signal processing module 13, the pixel classification module 11 is connected to the image sensor 20 through the one or more second image signal processing modules 13 to receive the image sensor 20 through the one or more The image transmitted by the second image signal processing module 13; wherein the second image signal processing module 13 is used to perform corresponding image signal processing operations on the input image.

In one example, the second image signal processing module 13 includes a dead pixel correction module (function: eliminates pixels that are significantly different from the surrounding pixels in the pixel array), a black level correction module (function: input from The dark current signal is subtracted from the image signal), a shadow correction module (function: to compensate for the brightness loss of surrounding pixels), and a white balance correction module (function: to remove the influence of ambient light). The first image signal processing module 12 includes a mosaic Module (function: rebuild the complete color), color correction module (function: correct color deviation), noise removal module (function: restore the relevant details of the image), sharpening module (function: restore the relevant details of the image); then The sequence of connection between the image sensor 20 and each module in the image signal processing device 10 is: image sensor 20→dead pixel correction module→black level correction module→shadow correction module→white balance correction module→the pixel classification module 11→Mosaic module→Color correction module→Noise removal module→Sharp module.

In one embodiment, considering that if the input image is a high-resolution image, some of the first image signal processing module 12 has a poor processing effect on the high-resolution image, for example, the image needs to be filtered in the noise removal module. , And the size of the filter in the noise removal module is fixed. If the input image is a high-resolution image, it contains more pixels, the noise removal module may not be able to fully identify the noise in the high-resolution image. Therefore, the image details cannot be restored, and the input image needs to be down-sampled to reduce the number of pixels. Based on this, the image signal processing device 10 further includes a down-sampling module 14, which is used to perform a down-sampling on the image sensor. 20. Perform one or more down-sampling processing on the collected image, and transmit the generated large-scale image to the pixel classification module 11 connected to it, so that the pixel classification module 11 can analyze each of the large-scale images. Pixels are classified and the classification information of each pixel is obtained, thereby ensuring that the first image signal processing module 12 that needs to process large-scale images does not need to repeat the same or similar down-sampling processing steps and image feature determination steps, which not only reduces The image signal processing time is reduced, and the hardware expenditure cost is also reduced.

In an implementation manner, please refer to FIG. 4, which shows a structure diagram of a fourth image signal processing device 10 according to an exemplary embodiment of this application. The image sensor 20 transmits the collected image to the down-sampling module. 14 and the pixel classification module 11, the down-sampling module 14 performs one or more down-sampling processing on the input image, and transmits the generated large-scale image to the pixel classification module 11 connected to it, that is to say The pixel classification module 11 may perform classification processing on the image collected by the image sensor 20 and the large-scale image generated by the down-sampling module 14.

In addition, the image sensor 20 transmits the collected current image to the down-sampling module 14 and the pixel classification module 11 in real time. Considering that the down-sampling process requires a certain processing time, the pixel classification module 11 is After the pixels of the image are classified to obtain certain classification information, if you wait for the large-scale image after the down-sampling process of the current image, and then classify and transmit it, this process will consume too much and invalid waiting time, resulting in the delay of image signal processing. The time is longer, which reduces the efficiency of image signal processing. In this embodiment, the inventor found that when the image sensor 20 transmits the currently collected image to the pixel classification module 11, within a specified time range (less than the waiting time), The down-sampling module 14 also finishes down-sampling the image collected last time by the image sensor 20 and sends the generated large-scale image to the pixel classification module 11. The pixel classification module 11 can classify the image currently collected by the image sensor 20 and the down-sampling large-scale image collected last time by the image sensor 20, respectively. Several classification information of the two images are then transmitted to the first image signal processing module 12 connected thereto, thereby effectively saving waiting time and improving image signal processing efficiency.

In another implementation manner, please refer to FIG. 5, which shows a structure diagram of a fifth image signal processing apparatus 10 according to an exemplary embodiment of this application. The image sensor 20 is processed by one or more second image signals. The module 13 transmits the collected image to the down-sampling module 14 and the pixel classification module 11, that is, the image collected by the image sensor is processed by each second image signal processing module 13, and then is processed by the last second image signal The processing module 13 outputs to the down-sampling module 14 and the pixel classification module 11. The down-sampling module 14 performs one or more down-sampling processing on the input image, and transmits the generated large-scale image to the connected The pixel classification module 11; wherein, the image input to the pixel classification module 11 includes: the image currently collected by the image sensor 20 and the last large-scale image collected after the down-sampling process, the image classification module The generated classification information corresponding to the above two images is transmitted together to the first image signal processing module 12 connected thereto, which is beneficial to shorten the waiting time and improve the efficiency of image signal processing.

In an embodiment, considering that there are fewer first image signal processing modules 12 that require a plurality of classification information corresponding to the large-scale image, if the plurality of classification information corresponding to the large-scale image is transmitted together in the image processing link , It may consume too much transmission resources. Therefore, the image signal processing device 10 may further include a storage module 15. Please refer to FIG. 6, which shows a sixth image signal processing device 10 according to an exemplary embodiment of this application. In a structural diagram, the storage module 15 is connected to the pixel classification module 11, and the pixel classification module 11 is also used to store some classification information corresponding to the large-scale image in the storage module 15; the first The image signal processing module 12 is also configured to obtain the plurality of classification information from the storage module 15, and perform corresponding image processing operations on the input image based on the plurality of classification information; in this embodiment, it needs to be based on the large-scale image corresponding The first image signal processing module 12 for processing a number of classification information can directly obtain the plurality of classification information from the storage module 15, thereby effectively saving the transmission of a number of classification information corresponding to large-scale images in the image processing link. Transmission resources.

Correspondingly, please refer to FIG. 7, which shows a flowchart of an image signal processing method according to an exemplary embodiment of this application. The image signal is applied to an image signal processing device, and the image signal processing device includes a pixel classification module and At least two first image signal processing modules; the method includes:

In step S101, in the pixel classification module, each pixel in the input image is classified to generate several classification information; the classification information is used to characterize the image characteristics of the pixel.

In step S102, in the first image signal processing module, the image input by the pixel classification module or the previous first image signal processing module and the plurality of classification information are received, and based on the plurality of classification information, the input The pixels in the image perform corresponding image processing operations, and transmit the generated image and the plurality of classification information to the next first image signal processing module.

In an embodiment, the classification information is expressed in any of the following ways: a hash value or a hash vector.

In an embodiment, the image features include at least one or more of the following: edge strength features, edge direction features, flat area features, texture features, color features, isolated point features, and domain features.

In an embodiment, the step S102 includes: using the plurality of classification information as an index, obtaining an image processing operation corresponding to each pixel in the input image from a pre-stored correspondence relationship, and executing the image processing operation; The corresponding relationship represents the corresponding relationship between the classification information and the image processing operation.

In an embodiment, the correspondence relationship is stored in a hash table format.

The hash table includes one or more key-value pair relationships; wherein the classification information is used as a hash key, and the image processing operation is stored as a key-value pair relationship of the hash value.

In an embodiment, the image processing operation includes filters of different scales and different types of image processing functions.

In an embodiment, the step S101 includes: for each pixel in the input image, classify based on the pixel and the neighboring pixels of the pixel, and generate classification information of the pixel.

In an embodiment, the input image is obtained from an image sensor.

In an embodiment, the method further includes: performing one or more down-sampling processing on the image collected by the image sensor to generate a large-scale image.

In an embodiment, the images to be classified include: the image currently collected by the image sensor and the large-scale image collected last time after the down-sampling process.

In an embodiment, the method further includes: storing a plurality of classification information corresponding to the large-scale image, so that the first image signal processing module obtains the plurality of stored classification information, and compares the input information based on the plurality of classification information. The image performs the corresponding image processing operation.

In an embodiment, the first image signal processing module includes at least one or more of the following: a dead pixel correction module, a black level correction module, a shadow correction module, a white balance correction module, a demosaicing module, and a color correction module , Brightness adjustment module, noise reduction module and sharpening module.

As for the method embodiment, since it basically corresponds to the device embodiment, for the relevant parts, please refer to the part of the description of the device embodiment, which will not be repeated here.

Correspondingly, referring to FIG. 8, an embodiment of the present application also provides a camera 100, including:

壳30。 30.

The lens assembly 40 is arranged inside the housing 30.

The image sensor 20 is arranged inside the housing 30 and is used to sense light passing through the lens assembly and generate electrical signals.

And the above-mentioned image signal processing device 10 is used to process the electrical signal.

Those skilled in the art can understand that FIG. 8 is only an example of the camera 100 and does not constitute a limitation on the camera 100. It may include more or less components than those shown in the figure, or a combination of certain components, or different components, such as The camera 100 may also include a network access device and the like.

Correspondingly, please refer to FIG. 9. An embodiment of the present application also provides a movable platform 001, including:

Body 02.

The power system 03 is installed in the body 02 and is used to provide power to the movable platform 001.

And the aforementioned camera 100.

In an embodiment, the movable platform may be an unmanned aerial vehicle, an unmanned vehicle or an unmanned ship.

Those skilled in the art can understand that FIG. 9 is only an example of the movable platform 001, and does not constitute a limitation on the movable platform 001. It may include more or less components than shown in the figure, or combine certain components, or different For example, the movable platform 001 may also include input and output devices, network access equipment, etc.; it is understandable that the camera 100 can be fixedly installed on the movable platform 001, or can be installed in a detachable manner On the movable platform 001, the embodiment of the present application does not impose any restriction on this.

The methods and devices provided by the embodiments of the present invention are described in detail above. Specific examples are used in this article to illustrate the principles and implementations of the present invention. The descriptions of the above embodiments are only used to help understand the methods and methods of the present invention. The core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation and the scope of application. In summary, the content of this specification should not be construed as a limitation of the present invention .

Claims

An image signal processing device, characterized by comprising a pixel classification module and at least two first image signal processing modules;

After the pixel classification module, at least two first image signal processing modules are connected in sequence;

The pixel classification module is used to classify each pixel in the input image, generate a number of classification information, and transmit the input image and a number of classification information to the first image signal processing module connected to it; The classification information is used to characterize the image characteristics of the pixel;

The first image signal processing module is configured to receive the image input by the pixel classification module or the previous first image signal processing module and the plurality of classification information; based on the plurality of classification information, the pixels in the input image Perform a corresponding image processing operation, and transmit the generated image and the plurality of classification information to the next first image signal processing module.
The device according to claim 1, wherein the classification information is expressed in any of the following ways: a hash value or a hash vector.
The device according to claim 1, wherein the first image signal processing module stores a correspondence between classification information and image processing operations;

The first image signal processing module is further configured to use the plurality of classification information as an index to obtain an image processing operation corresponding to each pixel in the input image from the correspondence relationship.
The device according to claim 3, wherein the corresponding relationship is stored in a hash table format;

The hash table includes one or more key-value pair relationships; wherein the classification information is used as a hash key, and the image processing operation is stored as a key-value pair relationship of the hash value.
The apparatus according to claim 1, wherein the image processing operation includes filters of different scales and different types of image processing functions.
The device of claim 1, wherein:

The pixel classification module is specifically configured to: for each pixel in the input image, classify based on the pixel and the neighboring pixels of the pixel, and generate classification information of the pixel.
The device according to claim 1, wherein the pixel classification module is connected to an image sensor, and the image sensor is used to collect an image and transmit it to the pixel classification module.
The device according to claim 1, wherein the device further comprises a second image signal processing module or a plurality of second image signal processing modules connected in sequence;

The pixel classification module is connected to the image sensor through the one or more second image signal processing modules to receive the image transmitted by the image sensor through the one or more second image signal processing modules.
The device according to claim 7 or 8, wherein the device further comprises the down-sampling module;

The down-sampling module is used to perform one or more down-sampling processing on the image collected by the image sensor, and transmit the generated large-scale image to the pixel classification module connected to it.
9. The device according to claim 9, wherein the image input to the pixel classification module comprises: an image currently collected by the image sensor and a large-scale image collected last time after down-sampling processing.
The device according to claim 10, wherein the device further comprises a storage module;

The pixel classification module is further configured to store some classification information corresponding to the large-scale image in the storage module;

The first image signal processing module is further configured to obtain the plurality of classification information from the storage module, and perform corresponding image processing operations on the input image based on the plurality of classification information.
The device according to claim 1, wherein the first image signal processing module comprises at least one or more of the following:

Dead pixel correction module, black level correction module, shadow correction module, white balance correction module, demosaicing module, color correction module, brightness adjustment module, noise reduction module and sharpening module.
The device according to claim 1, wherein the image feature includes at least one or more of the following:

Edge strength feature, edge direction feature, flat area feature, texture feature, color feature, isolated point feature, and domain feature.
An image signal processing method, characterized in that it is applied to an image signal processing device, the image signal processing device includes a pixel classification module and at least two first image signal processing modules; the method includes:

In the pixel classification module, each pixel in the input image is classified to generate a number of classification information; the classification information is used to characterize the image characteristics of the pixel;

In the first image signal processing module, the image input by the pixel classification module or the previous first image signal processing module and the plurality of classification information are received, and based on the plurality of classification information, the pixels in the input image Perform a corresponding image processing operation, and transmit the generated image and the plurality of classification information to the next first image signal processing module.
The method according to claim 13, wherein the classification information is expressed in any of the following ways: a hash value or a hash vector.
The method according to claim 13, wherein the image feature includes at least one or more of the following:

Edge strength feature, edge direction feature, flat area feature, texture feature, color feature, isolated point feature, and domain feature.
The method according to claim 13, wherein the performing corresponding image processing operations on the pixels in the input image based on the plurality of classification information comprises:

Using the several classification information as an index, the image processing operation corresponding to each pixel in the input image is obtained from the pre-stored correspondence relationship, and the image processing operation is executed; the correspondence relationship represents the relationship between the classification information and the image processing operation Correspondence.
The method according to claim 17, wherein the corresponding relationship is stored in a hash table format;

The hash table includes one or more key-value pair relationships; wherein the classification information is used as a hash key, and the image processing operation is stored as a key-value pair relationship of the hash value.
The method according to claim 13, wherein the image processing operation includes filters of different scales and different types of image processing functions.
The method according to claim 13, wherein the classifying each pixel in the input image to generate a number of classification information comprises:

For each pixel in the input image, classification is performed based on the pixel and the neighboring pixels of the pixel, and classification information of the pixel is generated.
The method according to claim 13, wherein the input image is obtained from an image sensor.
The method according to claim 21, further comprising:

One or more down-sampling processing is performed on the image collected by the image sensor to generate a large-scale image.
The method according to claim 22, wherein the images to be classified include: the image currently collected by the image sensor and the large-scale image collected last time after the down-sampling process.
The method according to claim 23, further comprising:

A number of classification information corresponding to the large-scale image is stored, so that the first image signal processing module obtains the plurality of stored classification information, and performs a corresponding image processing operation on the input image based on the plurality of classification information.
The method according to claim 13, wherein the first image signal processing module comprises at least one or more of the following:

Dead pixel correction module, black level correction module, shadow correction module, white balance correction module, demosaicing module, color correction module, brightness adjustment module, noise reduction module and sharpening module.
A camera, characterized in that it comprises:

shell;

The lens assembly is arranged inside the housing;

An image sensor, arranged inside the housing, for sensing light passing through the lens assembly and generating an electrical signal; and,

The image signal processing device according to any one of claims 1 to 13.
A movable platform, characterized in that it comprises:

Body

A power system installed in the body and used to provide power to the movable platform; and,

And the camera of claim 26.
The movable platform according to claim 27, wherein the movable platform includes an unmanned aerial vehicle, an unmanned vehicle, and an unmanned ship.