CN115170409A

CN115170409A - Image processing method, apparatus, device, readable storage medium and program product

Info

Publication number: CN115170409A
Application number: CN202210689289.7A
Authority: CN
Inventors: 王超君; 胥立丰; 刘欢
Original assignee: Beijing Eswin Computing Technology Co Ltd
Current assignee: Beijing Eswin Computing Technology Co Ltd
Priority date: 2022-06-16
Filing date: 2022-06-16
Publication date: 2022-10-11

Abstract

The embodiment of the application provides an image processing method, an image processing device, readable storage medium and program product, wherein the method comprises the following steps: acquiring the positions of any two adjacent pixel points in the target image on the target image; determining transformation processing information based on the format of the target image and the positions of any two adjacent pixel points on the target image; based on the conversion processing information, carrying out conversion processing on the windows corresponding to any two adjacent pixel points to obtain converted windows; a window corresponding to any two adjacent pixel points is an area formed by a plurality of pixel points; carrying out image processing on the transformed window to obtain pixel points corresponding to any two adjacent pixel points after image processing; and based on the transformation processing information, correspondingly processing the pixel points after the image processing so as to enable the pixel points after the image processing to be used for displaying the target image.

Description

Image processing method, apparatus, device, readable storage medium and program product

Technical Field

The present application relates to the field of computer technologies, and in particular, to an image processing method, an apparatus, a device, a readable storage medium, and a program product.

Background

In the prior art, applications using a pair of RGB + X images, such as RGB + NIR (infrared light), RGB + W (white light), etc., have appeared in the field of computer vision and image processing. Due to the advantages of compactness, low cost and the like, the single sensor captures the RGB + X Image to become an ideal choice, wherein the RAW4X4 format is the most common, the pixel points arranged in the RAW4X4 format are single-channel pixel points, each single-channel pixel point only stores one channel in the RGBX, the single-channel pixel points cannot be directly used for Image display and the like, the single-channel pixel points are processed by an ISP (Image Signal Processing) module, and the single-channel pixel points can be used for Image display and the like; the ISP module may be an image processing module in an image processing Chip or SoC (System on Chip). The ISP module processes a single-channel pixel, and generally needs to use a peripheral pixel of the pixel as reference information, and the ISP module sequentially scans (from top to bottom and from left to right) the position of each pixel in the image in a sliding window manner.

Due to the characteristics of the RAW4X4 format, the color of the center point of the sliding window changes as the sliding window moves, and there are four colors of R (Red ), G (Green ), B (Blue, blue) and X. Because the processing modes for different colors are different, the color of the pixel point is judged at each step of algorithm processing, and different processing algorithms are called. When the algorithm is complex, the algorithm complexity is greatly increased, for example, when an ASIC (Application Specific Integrated Circuit) hardware is implemented, the judgment and processing logic greatly increases the hardware resource consumption, thereby resulting in low image processing efficiency.

Disclosure of Invention

The present application provides an image processing method, an image processing apparatus, an image processing device, an image processing apparatus, a computer-readable storage medium, and a computer program product, which are used to solve the problem of how to improve image processing efficiency.

In a first aspect, the present application provides an image processing method, including:

acquiring the positions of any two adjacent pixel points in the target image on the target image;

determining transformation processing information based on the format of the target image and the positions of any two adjacent pixel points on the target image;

based on the conversion processing information, carrying out conversion processing on windows corresponding to any two adjacent pixel points to obtain converted windows; the window corresponding to any two adjacent pixel points is an area formed by a plurality of pixel points, the plurality of pixel points comprise any two adjacent pixel points, and the any two adjacent pixel points are positioned in the center of the window corresponding to any two adjacent pixel points;

performing image processing on the transformed window to obtain pixel points corresponding to any two adjacent pixel points after image processing;

and based on the transformation processing information, correspondingly processing the pixel points after the image processing so as to enable the pixel points after the image processing to be used for displaying the target image.

In one embodiment, determining transformation processing information based on the format of the target image and the positions of any two adjacent pixel points on the target image includes:

determining a row index and a column index corresponding to any two adjacent pixel points based on the row and the column of any two adjacent pixel points on the target image;

determining state serial numbers in a preset state table corresponding to any two adjacent pixel points based on the format, the row index and the column index of the target image;

and determining the conversion processing information corresponding to the state sequence number.

In one embodiment, determining the transformation processing information corresponding to the state sequence number includes:

and determining mirror image turning information and anticlockwise rotation information corresponding to the state serial number and information whether any pixel point in any two adjacent pixel points is a preset pixel point.

In one embodiment, the transformation processing information includes mirror image flipping information and counterclockwise rotation information, and based on the transformation processing information, the transformation processing is performed on the window corresponding to any two adjacent pixel points, including at least one of:

if the mirror image overturning information indicates that the mirror image overturning is carried out on the window corresponding to any two adjacent pixel points, carrying out mirror image overturning on the window corresponding to any two adjacent pixel points;

and if the anticlockwise rotation information indicates that the window corresponding to any two adjacent pixel points is anticlockwise rotated, the window corresponding to any two adjacent pixel points is anticlockwise rotated.

In one embodiment, one of the two pixels at the center position of the transformed window is a G pixel, the other of the two pixels is any one of an R pixel and a B pixel, and the upper side of the G pixel is an X pixel; the upper side is used for representing that the G pixel points are located in the Nth line of the converted window, the X pixel points are located in the (N-1) th line of the converted window, the G pixel points and the X pixel points are located in the same column of the converted window, and N is a positive integer.

In one embodiment, the image processing is performed on the transformed window to obtain each image-processed pixel point corresponding to any two adjacent pixel points, and the method includes:

and carrying out image processing on the converted window through a first image algorithm aiming at the G pixel point, a second image algorithm aiming at the R pixel point and the B pixel point and a third image algorithm aiming at the X pixel point to obtain pixel points which correspond to any two adjacent pixel points and are subjected to image processing.

In one embodiment, the conversion processing information includes mirror image turning information, counterclockwise rotation information, and information on whether any one of any two adjacent pixels is a predetermined pixel, the predetermined pixel is a B pixel, and the pixel after each image processing is correspondingly processed based on the conversion processing information, including at least one of:

if the mirror image turning information indicates that the mirror image turning is carried out on the window corresponding to any two adjacent pixel points, the value of one image-processed pixel point in each image-processed pixel point is exchanged with the value of the other image-processed pixel point in each image-processed pixel point;

if the anticlockwise rotation information indicates that the window corresponding to any two adjacent pixel points is anticlockwise rotated, determining one of the image-processed pixel points as a G pixel point and the other one of the image-processed pixel points as any one of a B pixel point, an R pixel point and an X pixel point;

and if any one of any two adjacent pixel points is a B pixel point, interchanging the value of a B channel and the value of an R channel in the three-channel values of the pixel points after the image processing.

In a second aspect, the present application provides an image processing apparatus comprising:

the first processing module is used for acquiring the positions of any two adjacent pixel points in the target image on the target image;

the second processing module is used for determining conversion processing information based on the format of the target image and the positions of any two adjacent pixel points on the target image;

the third processing module is used for carrying out conversion processing on the windows corresponding to any two adjacent pixel points based on the conversion processing information to obtain converted windows; a window corresponding to any two adjacent pixel points is an area formed by a plurality of pixel points, the plurality of pixel points comprise any two adjacent pixel points, and the any two adjacent pixel points are positioned in the center of the window corresponding to the any two adjacent pixel points;

the fourth processing module is used for carrying out image processing on the converted window to obtain pixel points corresponding to any two adjacent pixel points after image processing;

and the fifth processing module is used for correspondingly processing the pixel points after the image processing based on the transformation processing information so as to enable the pixel points after the image processing to be used for displaying the target image.

In a third aspect, the present application provides an electronic device, comprising: a processor, a memory, and a bus;

a bus for connecting the processor and the memory;

a memory for storing operating instructions;

and the processor is used for executing the image processing method of the first aspect of the application by calling the operation instruction.

In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program for executing the image processing method of the first aspect of the present application.

In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, performs the steps of the image processing method of the first aspect of the present application.

The technical scheme provided by the embodiment of the application at least has the following beneficial effects:

acquiring the positions of any two adjacent pixel points in the target image on the target image; determining transformation processing information based on the format of the target image and the positions of any two adjacent pixel points on the target image; based on the conversion processing information, carrying out conversion processing on the windows corresponding to any two adjacent pixel points to obtain converted windows; the window corresponding to any two adjacent pixel points is an area formed by a plurality of pixel points, the plurality of pixel points comprise any two adjacent pixel points, and the any two adjacent pixel points are positioned in the center of the window corresponding to any two adjacent pixel points; carrying out image processing on the transformed window to obtain pixel points corresponding to any two adjacent pixel points after image processing; based on the transformation processing information, correspondingly processing the pixel points after the image processing so as to enable the pixel points after the image processing to be used for displaying the target image; therefore, according to the image format (the format of the target image) and the row and column where two pixel points at the center of the window are located (any two adjacent pixel points are located on the target image), the window is converted into a fixed state (the converted window), after algorithm processing (image processing), corresponding output pixel points are selected (the pixel points after the image processing are correspondingly processed based on conversion processing information) so as to reduce judgment and logic processing required in the algorithm, therefore, the consumption of ASIC hardware resources and the implementation cost are reduced, the data throughput is high, and the efficiency of the image processing is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the description of the embodiments of the present application will be briefly described below.

Fig. 1 is a schematic diagram of an architecture of an image processing system according to an embodiment of the present application;

fig. 2 is a schematic flowchart of an image processing method according to an embodiment of the present application;

FIG. 3 is a schematic diagram of image processing provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of image processing provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of image processing provided by an embodiment of the present application;

FIG. 6 is a diagram illustrating image processing according to an embodiment of the present application;

FIG. 7 is a schematic diagram of image processing provided by an embodiment of the present application;

fig. 8 is a schematic flowchart of another image processing method according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application;

fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

Embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the embodiments set forth below in connection with the drawings are exemplary descriptions for explaining technical solutions of the embodiments of the present application, and do not limit the technical solutions of the embodiments of the present application.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that the terms "comprises" and/or "comprising," when used in this specification in connection with embodiments of the present application, specify the presence of stated features, information, data, steps, operations, elements, and/or components, but do not preclude the presence or addition of other features, information, data, steps, operations, elements, components, and/or groups thereof, as embodied in the art. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. The term "and/or" as used herein indicates at least one of the items defined by the term, e.g., "a and/or B" indicates either implementation as "a", or implementation as "B", or implementation as "a and B".

It is understood that in the specific implementation of the present application, the data related to image processing is referred to, when the above embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use and processing of the related data need to comply with relevant laws and regulations and standards in relevant countries and regions.

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

The research of the related technology finds that:

(1) One clock handles one pixel: processing a pixel point by one clock, namely taking a window area with the size of HxW (5 x 5) by taking each pixel of the image as a center, and processing the center pixel in the window; there are four possibilities for each clock-input center pixel: r, G, B and X; if the relationship between the central pixel and its neighborhood pixels is considered, there are eight possibilities: gr, gb, grx, gbx, R, B, xr and Xb (the right side of G is the R pixel point is marked as Gr, the right side of G is the B pixel point is marked as Gb, the right side of G is X, the lower side of G is R is marked as Grx, the B is marked as Gbx, the lower right corner of X is R is marked as Xr, the B is marked as Xb); the treatment method comprises the following steps: the processing modes of each color are different, the Gr, gb, grx, gbx, R, B, xr and Xb do not divide the R and B pixel point types, four processing algorithms are realized, and algorithm switching is required according to the pixel point colors; the problems are as follows: the data throughput is low, one clock can only process one pixel point, and four sets of hardware logics and corresponding processing algorithms need to be realized when the hardware is realized.

(2) One clock handles two pixels: processing two pixel points by one clock, namely taking a window area with the size of HxW (5 x 6) for every two pixels of the image by taking the two pixels as the center, and processing the two pixels at the center in the window; the center two pixels of each clock input have six possibilities: GR, GB, GX, RG, BG, and XG; if the relationship between the central two pixel points and the neighborhood pixel points is considered, eight possibilities exist: grR, gbB, grX, gbX, RGb, BGr, XGr, and XGb (the case where the right side or lower side of G is an R pixel is denoted as Gr, and the case where the right side or lower side of G is a B pixel is denoted as Gb); the treatment method comprises the following steps: processing two pixel points (P0 and P1) simultaneously; p0 has R, G, B and X, and does not have R and B pixel point types, so three possibilities are provided, and three processing algorithms need to be realized; when P0 is G, P1 has two additional possibilities, and two processing algorithms need to be realized; the problems are as follows: when the hardware is implemented, four processing algorithms need to be implemented at every two points, and corresponding color judgment and switching logic need to be implemented, so that the hardware resources are doubled.

In order to solve the problems in (1) and (2), in the image processing method provided in the embodiment of the present application, according to an image format (a format of a target image), a row and a column where two pixel points in a center of a window are located (where any two adjacent pixel points are located on the target image), the window is transformed to a fixed state (a transformed window), and after algorithm processing (image processing), corresponding output pixel points are selected (based on transformation processing information, corresponding processing is performed on the pixel points after each image processing), so as to reduce judgment and logic processing required in the algorithm, thereby reducing ASIC hardware resource consumption and implementation cost, and improving image processing efficiency, and data throughput is high.

The embodiment of the application relates to an image processing method provided by an image processing system, and relates to the fields of computer vision, image processing and the like.

For better understanding and description of the embodiments of the present application, some technical terms referred to in the embodiments of the present application will be briefly described below.

RGB: the RGB color scheme is a color standard in the industry, and various colors are obtained by changing three color channels of red (R), green (G) and blue (B) and superimposing the three color channels on each other; RGB represents the color of the three channels red, green, and blue.

RAW: RAW is an unprocessed and uncompressed format; the RAW image is RAW data obtained by converting the captured light source signal into a digital signal by the image sensor.

The technical scheme provided by the embodiment of the application relates to an image processing technology, and the technical scheme of the application is explained in detail by specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.

In order to better understand the scheme provided by the embodiment of the present application, the scheme is described below with reference to a specific application scenario.

In an embodiment, fig. 1 shows an architecture diagram of an image processing system to which the embodiment of the present application is applied, and it is understood that the image processing method provided by the embodiment of the present application may be applied to, but is not limited to, the application scenario shown in fig. 1.

In the present example, as shown in fig. 1, the architecture of the image processing system in this example may include, but is not limited to, the image processing apparatus 10, the terminal 20, and the network 30. The image processing apparatus 10 and the terminal 20 may interact with each other through the network 30. The image processing apparatus 10 may be an in-vehicle apparatus, a security camera, or the like.

The image processing apparatus 10 acquires a target image by a sensor; the image processing device 10 determines the format of the target image and the position of any two adjacent pixel points in the target image on the target image through a chip (for example, an image processing chip or an image processing module in SoC), wherein any two adjacent pixel points are single-channel pixel points; the image processing device 10 determines conversion processing information based on the format of the target image and the positions of any two adjacent pixel points on the target image through a chip; the image processing device 10 performs conversion processing on the windows corresponding to any two adjacent pixel points through the chip based on the conversion processing information to obtain converted windows; the window corresponding to any two adjacent pixel points is an area formed by a plurality of pixel points, the plurality of pixel points comprise any two adjacent pixel points, and the any two adjacent pixel points are positioned in the center of the window corresponding to any two adjacent pixel points; the image processing device 10 performs image processing on the transformed window through the chip to obtain pixel points corresponding to any two adjacent pixel points after image processing; the image processing device 10 performs corresponding processing on each image-processed pixel point based on the conversion processing information by the chip, so that each image-processed pixel point is used for displaying the target image; the target image is displayed through a display of the image processing device 10 (for example, an in-vehicle device), or a result of performing corresponding processing on each pixel point after image processing is sent to the terminal 20, and the target image is displayed through the display of the terminal 20.

It should be understood that the above description is only an example, and the present embodiment is not limited thereto.

The terminal includes, but is not limited to, a smart phone, a cell phone simulator, a tablet computer, a notebook computer, a digital broadcast receiver, an MID (Mobile Internet Devices), a PDA (personal digital assistant), an intelligent voice interaction device, an intelligent home appliance, a vehicle-mounted terminal, and the like.

Such networks may include, but are not limited to: a wired network, a wireless network, wherein the wired network comprises: a local area network, a metropolitan area network, and a wide area network, the wireless network comprising: bluetooth, wi-Fi, and other networks that enable wireless communication. The determination may also be based on the requirements of the actual application scenario, and is not limited herein.

Referring to fig. 2, fig. 2 shows a flowchart of an image processing method provided in an embodiment of the present application, where the method may be executed by any electronic device, such as an image processing device, as an alternative implementation, the method may be executed by the image processing device, and for convenience of description, in the following description of some alternative embodiments, the image processing device will be taken as an example of a main body for executing the method. As shown in fig. 2, an image processing method provided in an embodiment of the present application includes the following steps:

s201, the position of any two adjacent pixel points in the target image on the target image is obtained.

Specifically, as shown in fig. 3, in the RAW4x4 format, there may be eight permutation combinations with 4x4 pixel blocks as a group, and the eight permutation combinations are GRGBX, GBGRX, XRGBG, XBGRG, RGBGX, BGRGX, XGRGB and XGBGR, respectively; wherein R, G, B and X are respectively: r (Red ), B (Blue ), G (Green, green), and X (X may represent infrared, white, etc.). The target image may be arranged in any one of the eight arrangements.

Specifically, the format of the target image may be any one of eight permutation and combination manners shown in fig. 3; any arrangement combination mode corresponds to eight states of the two central pixels of the window shown in fig. 4, wherein the eight states are GrR, gbB, grX, gbX, RGb, BGr, XGr and XGb; here, the case where the right side or lower side of G is an R pixel is denoted as Gr, the case where the right side or lower side of G is a B pixel is denoted as Gb, and the window size is HxW, for example, 5x6. Any two adjacent pixel points in the target image are single-channel pixel points.

S202, determining transformation processing information based on the format of the target image and the positions of any two adjacent pixel points on the target image.

The conversion processing information is used for indicating the position conversion processing of any two adjacent pixel points. Specifically, the conversion processing information includes mirror image turning information, counterclockwise rotation information, information on whether any one of any two adjacent pixel points is a predetermined pixel point, and the like.

S203, based on the conversion processing information, converting the windows corresponding to any two adjacent pixel points to obtain converted windows; the window corresponding to any two adjacent pixel points is an area formed by a plurality of pixel points, the plurality of pixel points comprise any two adjacent pixel points, and the any two adjacent pixel points are positioned in the center of the window corresponding to the any two adjacent pixel points.

Specifically, the size of the window shown in fig. 4 is HxW, for example, 5x6, and the window is a region composed of a plurality of pixel points, and the size of the region is 5x6.

And S204, carrying out image processing on the transformed window to obtain pixel points corresponding to any two adjacent pixel points after image processing.

Specifically, the pixel points after image processing may be single-channel pixel points, and the pixel points after image processing may also be multi-channel pixel points.

And S205, based on the transformation processing information, performing corresponding processing on the pixel points after the image processing so as to enable the pixel points after the image processing to be used for displaying the target image.

Specifically, based on the transformation processing information, the pixel points of the processed images are correspondingly processed, that is, the positions of the pixel points of the processed images are adjusted, so that the target image can be correctly displayed to the user.

In the embodiment of the application, the positions of any two adjacent pixel points in the target image on the target image are obtained; determining transformation processing information based on the format of the target image and the positions of any two adjacent pixel points on the target image; based on the conversion processing information, carrying out conversion processing on the windows corresponding to any two adjacent pixel points to obtain converted windows; the window corresponding to any two adjacent pixel points is an area formed by a plurality of pixel points, the plurality of pixel points comprise any two adjacent pixel points, and the any two adjacent pixel points are positioned in the center of the window corresponding to any two adjacent pixel points; performing image processing on the transformed window to obtain pixel points corresponding to any two adjacent pixel points after image processing; based on the transformation processing information, correspondingly processing the pixel points after the image processing so as to enable the pixel points after the image processing to be used for displaying the target image; therefore, according to the image format (the format of the target image) and the row and column where two pixel points at the center of the window are located (any two adjacent pixel points are located on the target image), the window is converted into a fixed state (the converted window), after algorithm processing (image processing), corresponding output pixel points are selected (the pixel points after the image processing are correspondingly processed based on conversion processing information) so as to reduce judgment and logic processing required in the algorithm, therefore, the consumption of ASIC hardware resources and the implementation cost are reduced, the data throughput is high, and the efficiency of the image processing is improved.

In one embodiment, the determining the transformation processing information based on the format of the target image and the position of any two adjacent pixel points on the target image includes steps A1-A3:

step A1, based on the row and the column of any two adjacent pixel points on the target image, determining the row index and the column index corresponding to any two adjacent pixel points.

Specifically, a behavior cy of two adjacent pixels on the target image, for example, a target image with a size of 100 × 100, a behavior 98 of two adjacent pixels; the row index corresponding to two adjacent pixel points is row _ id, and formula (1) for calculating the row index row _ id is as follows:

row_id＝cy％4 (1)

wherein, cy%4 represents the remainder of dividing cy by 4, and row _ id takes the value of 0, 1, 2 or 3.

The adjacent two pixels are in a row cx on the target image, for example, the target image with the size of 100x100, and the column of the adjacent two pixels is 24. Since one clock processes two pixels, cx is an even number, for example, a target image having a size of 100x100 has 0 th row to 99 th row (total 100 rows), and has 0 th column, 2 nd column, 6 th column, 8 th column … and 98 th column (total 50 columns). The column index corresponding to two adjacent pixel points is col _ id, and formula (2) for calculating the column index col _ id is as follows:

col_id＝(cx％4)/2 (2)

wherein cx%4 represents a remainder obtained by dividing cx by 4, for example, when cx is 6, 6% = 4=2, (6%4)/2=2/2=1, that is, the column index col _ id is 1; the value of col _ id is 0 or 1.

And step A2, determining state sequence numbers in a preset state table corresponding to any two adjacent pixel points based on the format, the row index and the column index of the target image.

Specifically, the format of the target image may be any one of the eight arrangement combinations shown in fig. 3, which are GRGBX, GBGRX, XRGBG, XBGRG, RGBGX, BGRGX, XGRGB, and XGBGR, respectively; any one of the permutation and combination modes corresponds to eight states of the central two pixels of the window shown in fig. 4, wherein the eight states are GrR, gbB, grX, gbX, RGb, BGr, XGr and XGb.

As shown in fig. 4, the mirror image inversion is performed on the four states (RGb, BGr, XGr, and XGb) of the first row (row 1) with the central axes of the two central pixels (two adjacent pixels) as the mirror image axis, and the four states of the first row after the inversion are converted into four states (GrR, gbB, grX, and GbX) of the corresponding column of the second row (row 2); after the last column is omitted from GrX of the third column (row 2, col 3) of the second row and GbX of the fourth column (row 2, col 4) of the second row, the two states (GrX and GbX) of the third column (col 3) and the fourth column (col 4) after rotation are rotated by 90 degrees counterclockwise around the G pixel point, and the two states (GrR and GbB) of the first column (col 1) and the second column (col 2) are converted. Thus, the eight states are switched to two states (GrR and GbB) for the first column (row 2, col 1) of the second row and the second column (row 2, col 2) of the second row by mirror inversion and counterclockwise rotation; and finally, distinguishing GrR and GbB by whether the two central pixel points contain B pixels.

The method comprises the following steps that inv can be used for representing whether mirror image overturning is executed or not, rot can be used for representing whether anticlockwise rotation is executed or not by 90 degrees, and blue can be used for representing whether two central pixel points contain B pixels or not; the image overturning method comprises the following steps that inv is 1 for representing that image overturning is executed, inv is 0 for representing that image overturning is not executed, rot is 1 for representing that anticlockwise rotation is executed for 90 degrees, rot is 0 for representing that anticlockwise rotation is not executed for 90 degrees, blue is 1 for representing that two central pixel points contain B pixels, and blue is 0 for representing that two central pixel points do not contain B pixels. The 3-bit data composed of inv, rot and blue can be used to represent the eight states shown in fig. 4, and the correspondence between inv, rot, blue and eight states is shown in table 1:

table 1: correspondence between inv, rot, blue and eight states

Status (Status)	inv	rot	blue	Window (fig. 4)
					0	0	0	0	row2，col1
1	0	0	1	row2，col2
					2	0	1	0	row2，col3
3	0	1	1	row2，col4
					4	1	0	0	row1，col1
5	1	0	1	row1，col2
					6	1	1	0	row1，col3
7	1	1	1	row1，col4

Where 0, 1, 2, 3, 4, 5, 6, and 7 in Status are Status numbers. row denotes the row in which the eight states are located in fig. 4, col denotes the column in which the eight states are located in fig. 4, for example, row2 denotes the second row, col1 denotes the first column. The state is 3-bit data, the high order of the 3-bit data stores the information of inv, the middle order of the 3-bit data stores the information of rot, and the low order of the 3-bit data stores the information of blue.

Since the RAW4x4 format uses 4x4 pixel blocks as a group and processes two pixel points in one clock, a matrix (Mat) of four rows and two columns can be used to store all states of a group of pixel blocks when every two pixels are used as central pixels. For eight RAW4x4 formats of RGBX images, eight matrices of four rows and two columns need to be stored, and these eight matrices are represented as the state lookup table (preset state table) as shown below:

TABLE 2 State lookup Table

For example, the matrix corresponding to GRGBX in the state lookup table has four rows and two columns; four rows are respectively represented by 4 row indexes, a row index 0 represents a 0 th row, a row index 1 represents a1 st row, a row index 2 represents a2 nd row, and a row index 3 represents a3 rd row; the two columns are respectively represented by 2 column indexes, wherein the column index 0 represents the 0 th column, and the column index 1 represents the 1 st column; GRGBX corresponds to

matrix elements

0, 1, 2, 3, 4, 5, 6, and 7 as state indexes. For another example, based on the format GRGBX, the row index 0, and the column index 0 of the target image, it is determined that the state sequence number in the state lookup table corresponding to two adjacent pixel points is 0, that is, the element 0 in the 0 th row and the 0 th column in the matrix corresponding to GRGBX.

For certain sensors, there is a certain RAW4x4 format, one sensor corresponds to one image format in fig. 3, i.e. one target image corresponds to one RAW4x4 format, so that, according to the sensor, the corresponding matrix can be selected from eight matrices in the state lookup table, where GRGBX: pattern =0, gbgrx: pattern =1, xrgbg: pattern =2, xbgrg: pattern =3, rgbgx: pattern =4, bgrgx: pattern =5,xgrgb: pattern =6,xgbgr: pattern =7.

And step A3, determining the conversion processing information corresponding to the state sequence number.

Specifically, for example, based on the format GRGBX, the row index 0, and the column index 0 of the target image, it is determined that the state sequence number in the state lookup table corresponding to two adjacent pixel points is 0, that is, element 0 in row 0 and column 0 in the matrix corresponding to GRGBX; from table 1, inv, rot, and blue corresponding to the state number 0 are all 0, that is, the state number 0 corresponds to GrR.

Specifically, values of inv, rot and blue corresponding to the state sequence numbers are determined according to the table 1; for example, if the state serial number is 0, it is determined that the values of inv, rot, and blue are all 0; for another example, if the state number is 6, it is determined that inv takes a value of 1, rot takes a value of 1, and blue takes a value of 0.

In one embodiment, the transformation processing information includes mirror image turning information and counterclockwise rotation information, and the transformation processing is performed on the window corresponding to any two adjacent pixel points based on the transformation processing information, and includes at least one of:

Specifically, as shown in fig. 5, if inv =1, the window is mirror-flipped. As shown in fig. 6, if rot =1, the window is rotated counterclockwise by 90 degrees.

In one embodiment, one of the two pixels at the center position of the transformed window is a G pixel, the other of the two pixels is any one of an R pixel and a B pixel, and the upper side of the G pixel is an X pixel; the upper side is used for representing that the G pixel points are in the Nth row of the transformed window, the X pixel points are in the N-1 th row of the transformed window, the G pixel points and the X pixel points are in the same column of the transformed window, and N is a positive integer.

Specifically, as shown in fig. 7, if blue =1, there is a B pixel in the center two pixels. And carrying out corresponding transformation on the window according to the values of inv and rot, wherein the left pixel point is a G pixel point, the right pixel point is an R pixel point or a B pixel point, and the pixel point on the upper side of the G pixel point is an X pixel point after the transformation.

In one embodiment, the image processing on the transformed window to obtain each image-processed pixel point corresponding to any two adjacent pixel points includes:

and performing image processing on the converted window through a first image algorithm aiming at the G pixel point, a second image algorithm aiming at the R pixel point and the B pixel point and a third image algorithm aiming at the X pixel point to obtain pixel points which correspond to any two adjacent pixel points and are subjected to image processing.

Specifically, algorithm processing (image processing) is performed on the transformed window, and only a processing algorithm (a first image algorithm) of G pixels is required to be applied to left-side pixel points of two central pixel points, a processing algorithm (a second image algorithm) of R pixels or B pixels is required to be applied to right-side pixel points of the two central pixel points, and a processing algorithm (a third image algorithm) of X pixels is required to be applied to upper-side pixel points of the left-side pixel points of the two central pixel points; thus, the logic for judging and processing the pixel type and position under different RAW4x4 formats and different states is omitted. The first image algorithm, the second image algorithm and the third image algorithm may be the same algorithm, for example, an image processing algorithm such as DPC (dead Pixel Correction), demosaic (demosaicing), and the like.

In one embodiment, the transformation processing information includes mirror image turning information, counterclockwise rotation information, and information about whether any one of any two adjacent pixels is a predetermined pixel, the predetermined pixel is a B pixel, and the image-processed pixels are correspondingly processed based on the transformation processing information, including at least one of:

Specifically, two image-processed pixel points are selected from the image-processed pixel points as output, and the output two image-processed pixel points are marked as P0 and P1. If blue =1 and the image processing algorithm is Demosaic, the value of the R channel and the value of the B channel in the three channel values of the pixel point after the same image processing are exchanged; and if blue =0, the value of the R channel and the value of the B channel in the three channel values of the pixel point after the same image processing are not interchanged. If the image processing algorithm is not Demosaic, namely the image processing algorithm is other algorithms, the output is represented as a B pixel point. If rot =1, then P1 is the upper side pixel point of the left side pixel point in the two center pixel points of the output; if rot =0, P1 is the right pixel point of the two center pixel points of the output, and P0 is always the left pixel point of the two center pixel points of the output. If inv =1, the values of the P0 and P1 outputs are interchanged; if inv =0, the values output by P0 and P1 are not interchanged.

The application of the embodiment of the application has at least the following beneficial effects:

according to the image format and the rows and columns of the two pixel points in the center of the window, the window is converted to a fixed state, and after algorithm processing, corresponding output pixel points are selected to reduce judgment and logic processing required in the algorithm, so that ASIC hardware resource consumption and implementation cost are reduced, data throughput is high, and image processing efficiency is improved.

In order to better understand the method provided by the embodiment of the present application, the following further describes the scheme of the embodiment of the present application with reference to an example of a specific application scenario.

In a specific application scenario embodiment, for example, an image processing scenario, referring to fig. 8, a processing flow of an image processing method is shown, and as shown in fig. 8, the processing flow of the image processing method provided in the embodiment of the present application includes the following steps:

s801, inputting an image to be processed into an image processing apparatus.

Specifically, each pixel point location in the image is scanned sequentially (top to bottom, left to right) in a sliding window fashion. And for every two adjacent pixel points, taking a window area with the size of HxW by taking the two adjacent pixel points as the center, and processing the two adjacent pixel points in the window. For the pixel points on the image boundary, the pixel points can exceed the image area when the window area is taken, filling processing is needed, and the filling processing is willing to adopt a mirror image mode.

S802, the image processing device constructs a state lookup table.

Specifically, the first index of the state lookup table represents the image format, the second index and the third index of the state lookup table represent the values of inv, rot and blue in the 4x4 pixel block, and the 4x4 pixel block has two pixel points with every two adjacent pixel points as the center. For example, the state number status = rgbx [ pattern ] [ row _ id ] [ col _ id ], pattern of 0, row \uid of 0, col \uid of 0, rgbx [0] [0] [0] indicates that the image format is GRGBX, the center two pixels are GrR, inv =0, rot =0, and blue =0.

And S803, the image processing equipment determines the state (state serial number) through a state lookup table based on the image format and the rows and columns of two adjacent pixel points in the image.

Specifically, based on the image format, the value of the first index of rgbx is calculated; for example, if the input format is GRGBX, pattern =0. Calculating the value of a second index and the value of a third index of the rgbx through a formula (1) and a formula (2) based on the rows and the columns of two adjacent pixel points; for example, cy =0, cx =0, then row _ id =0, col \uid =0. Searching the state: status = rgbx [ pattern ] [ row _ id ] [ col _ id ]; for example, pattern =0, row \uid =0, col \uid =0, then status = rgbx [0] [0] [0] =0.

S804, the image processing equipment carries out corresponding conversion on the window according to the state to obtain the converted window.

Specifically, if inv =1, mirror image flipping is performed on the window; if inv =0, the window is not mirror flipped. If rot =1, after discarding the last column of the window, rotating the window by 90 degrees counterclockwise; if rot =0, no transformation is done.

And S805, the image processing equipment performs algorithm processing (image processing) on the converted window to obtain pixel points after each image is processed.

And S806, the image processing equipment selects an output pixel point from the pixel points after the image processing according to the state.

Specifically, two image-processed pixel points are selected from the image-processed pixel points as output, and the output two image-processed pixel points are marked as P0 and P1. If blue =1 and the image processing algorithm is Demosaic, the value of the R channel and the value of the B channel in the three channel values of the pixel point after the same image processing are exchanged; and if blue =0, the value of the R channel and the value of the B channel in the three channel values of the pixel point after the same image processing are not interchanged. If the image processing algorithm is not Demosaic, that is, the image processing algorithm is other algorithms, the output is represented as a B pixel point. If rot =1, P1 is an upper pixel point of a left pixel point of the two output center pixel points; if rot =0, P1 is the right pixel point of the two center pixel points of the output, and P0 is always the left pixel point of the two center pixel points of the output. If inv =1, the values output by P0 and P1 are interchanged; if inv =0, the values output by P0 and P1 are not interchanged.

An image processing apparatus is further provided in the embodiment of the present application, and a schematic structural diagram of the image processing apparatus is shown in fig. 9, and the image processing apparatus 90 includes a first processing module 901, a second processing module 902, a third processing module 903, a fourth processing module 904, and a fifth processing module 905.

The first processing module 901 is configured to obtain positions of any two adjacent pixel points in the target image on the target image;

a second processing module 902, configured to determine transformation processing information based on a format of the target image and positions of any two adjacent pixel points on the target image;

a third processing module 903, configured to perform transformation processing on a window corresponding to any two adjacent pixel points based on the transformation processing information, so as to obtain a transformed window; the window corresponding to any two adjacent pixel points is an area formed by a plurality of pixel points, the plurality of pixel points comprise any two adjacent pixel points, and the any two adjacent pixel points are positioned in the center of the window corresponding to any two adjacent pixel points;

a fourth processing module 904, configured to perform image processing on the transformed window to obtain pixel points after image processing corresponding to any two adjacent pixel points;

a fifth processing module 905, configured to perform corresponding processing on the pixel points after the image processing based on the transformation processing information, so that the pixel points after the image processing are used for displaying the target image.

In an embodiment, the second processing module 902 is specifically configured to:

In an embodiment, the transformation processing information includes mirror flipping information and counterclockwise rotation information, and the third processing module 903 is specifically configured to perform at least one of the following:

if the mirror image overturning information indicates that the mirror image overturning is carried out on the windows corresponding to any two adjacent pixel points, carrying out mirror image overturning on the windows corresponding to any two adjacent pixel points;

In one embodiment, one of the two pixels at the center position of the converted window is a G pixel, the other of the two pixels is any one of an R pixel and a B pixel, and the upper side of the G pixel is an X pixel; the upper side is used for representing that the G pixel points are in the Nth row of the transformed window, the X pixel points are in the N-1 th row of the transformed window, the G pixel points and the X pixel points are in the same column of the transformed window, and N is a positive integer.

In an embodiment, the fourth processing module 904 is specifically configured to:

In an embodiment, the conversion processing information includes mirror image turning information, counterclockwise rotation information, and information about whether any one of two adjacent pixels is a predetermined pixel, where the predetermined pixel is a B pixel, and the fifth processing module 905 is specifically configured to execute at least one of:

and if any one of any two adjacent pixel points is a B pixel point, interchanging the value of a B channel and the value of an R channel in the three channel values of the pixel points after the image processing.

acquiring the positions of any two adjacent pixel points in the target image on the target image; determining transformation processing information based on the format of the target image and the positions of any two adjacent pixel points on the target image; based on the conversion processing information, carrying out conversion processing on the windows corresponding to any two adjacent pixel points to obtain converted windows; the window corresponding to any two adjacent pixel points is an area formed by a plurality of pixel points, the plurality of pixel points comprise any two adjacent pixel points, and the any two adjacent pixel points are positioned in the center of the window corresponding to any two adjacent pixel points; carrying out image processing on the transformed window to obtain pixel points corresponding to any two adjacent pixel points after image processing; based on the transformation processing information, correspondingly processing the pixel points after the image processing so as to enable the pixel points after the image processing to be used for displaying the target image; therefore, according to the image format (the format of the target image) and the row and the column (the positions of any two adjacent pixels on the target image) of the two pixels in the center of the window, the window is converted to a fixed state (the converted window), after algorithm processing (image processing), corresponding output pixels are selected (corresponding processing is performed on the pixels after the image processing based on conversion processing information) so as to reduce judgment and logic processing required in the algorithm, and therefore, the consumption of ASIC hardware resources and the implementation cost are reduced, the data throughput is high, and the efficiency of image processing is improved.

An embodiment of the present application further provides an electronic device, a schematic structural diagram of the electronic device is shown in fig. 10, and an electronic device 4000 shown in fig. 10 includes: a processor 4001 and a memory 4003. Processor 4001 is coupled to memory 4003, such as via bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, and the transceiver 4004 may be used for data interaction between the electronic device and other electronic devices, such as transmission of data and/or reception of data. It should be noted that the transceiver 4004 is not limited to one in practical applications, and the structure of the electronic device 4000 is not limited to the embodiment of the present application.

The Processor 4001 may be a CPU (Central Processing Unit), a general-purpose Processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other Programmable logic device, a transistor logic device, a hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. The processor 4001 may also be a combination that performs a computational function, including, for example, a combination of one or more microprocessors, a combination of a DSP and a microprocessor, or the like.

Bus 4002 may include a path that carries information between the aforementioned components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus 4002 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 10, but this is not intended to represent only one bus or type of bus.

The Memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical Disc storage, optical Disc storage (including Compact Disc, laser Disc, optical Disc, digital versatile Disc, blu-ray Disc, etc.), a magnetic Disc storage medium, other magnetic storage devices, or any other medium that can be used to carry or store a computer program and that can be Read by a computer, without limitation.

The memory 4003 is used for storing computer programs for executing the embodiments of the present application, and is controlled by the processor 4001 to execute. The processor 4001 is used to execute computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

Among them, electronic devices include but are not limited to: an image processing apparatus, and the like.

The embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, can implement the steps of the foregoing method embodiments and corresponding content.

Embodiments of the present application further provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps and corresponding contents of the foregoing method embodiments can be implemented.

Based on the same principle as the method provided by the embodiment of the present application, the embodiment of the present application also provides a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in any of the alternative embodiments of the present application.

It should be understood that, although each operation step is indicated by an arrow in the flowchart of the embodiment of the present application, the implementation order of the steps is not limited to the order indicated by the arrow. In some implementation scenarios of the embodiments of the present application, the implementation steps in the flowcharts may be performed in other sequences as desired, unless explicitly stated otherwise herein. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on an actual implementation scenario. Some or all of these sub-steps or stages may be performed at the same time, or each of these sub-steps or stages may be performed at different times, respectively. In a scenario where execution times are different, an execution sequence of the sub-steps or the phases may be flexibly configured according to requirements, which is not limited in the embodiment of the present application.

The foregoing is only an optional implementation manner of a part of implementation scenarios in the present application, and it should be noted that, for those skilled in the art, other similar implementation means based on the technical idea of the present application are also within the protection scope of the embodiments of the present application without departing from the technical idea of the present application.

Claims

1. An image processing method, comprising:

acquiring the positions of any two adjacent pixel points in a target image on the target image;

determining transformation processing information based on the format of the target image and the positions of the any two adjacent pixel points on the target image;

based on the conversion processing information, converting the windows corresponding to any two adjacent pixel points to obtain converted windows; the window corresponding to any two adjacent pixel points is an area formed by a plurality of pixel points, the plurality of pixel points comprise any two adjacent pixel points, and the any two adjacent pixel points are positioned at the center of the window corresponding to the any two adjacent pixel points;

and correspondingly processing the pixel points after the image processing based on the transformation processing information so as to enable the pixel points after the image processing to be used for displaying the target image.

2. The method according to claim 1, wherein determining transformation processing information based on the format of the target image and the positions of the two arbitrary adjacent pixels on the target image comprises:

determining a row index and a column index corresponding to any two adjacent pixel points based on the row and the column of the any two adjacent pixel points on the target image;

determining state sequence numbers in a preset state table corresponding to any two adjacent pixel points based on the format of the target image, the row index and the column index;

and determining the transformation processing information corresponding to the state sequence number.

3. The method according to claim 2, wherein the determining the transformation processing information corresponding to the state sequence number comprises:

and determining mirror image turning information and anticlockwise rotation information corresponding to the state serial number and information whether any pixel point of any two adjacent pixel points is a preset pixel point.

4. The method according to claim 1, wherein the transformation processing information includes mirror image flipping information and counterclockwise rotation information, and the transforming the window corresponding to any two adjacent pixel points based on the transformation processing information includes at least one of:

and if the anticlockwise rotation information indicates that the windows corresponding to any two adjacent pixel points are anticlockwise rotated, carrying out anticlockwise rotation on the windows corresponding to any two adjacent pixel points.

5. The method according to claim 1, wherein one of the two pixels at the center position of the transformed window is a G pixel, the other of the two pixels is any one of an R pixel and a B pixel, and an upper side of the G pixel is an X pixel; the upper side is used for representing that the G pixel points are located on the Nth line of the transformed window, the X pixel points are located on the N-1 th line of the transformed window, the G pixel points and the X pixel points are located on the same column of the transformed window, and N is a positive integer.

6. The method according to claim 5, wherein the image processing the transformed window to obtain each image-processed pixel point corresponding to any two adjacent pixel points comprises:

and performing image processing on the transformed window through a first image algorithm aiming at the G pixel point, a second image algorithm aiming at the R pixel point and the B pixel point and a third image algorithm aiming at the X pixel point to obtain pixel points after image processing corresponding to any two adjacent pixel points.

7. The method according to claim 1, wherein the transformation processing information includes mirror image flipping information, counterclockwise rotation information, and information on whether any one of the two adjacent pixels is a predetermined pixel, the predetermined pixel is a B pixel, and the corresponding processing is performed on the processed pixels of each image based on the transformation processing information, and includes at least one of:

if the mirror image turning information indicates that the mirror image turning is performed on the window corresponding to any two adjacent pixel points, exchanging the value of one image-processed pixel point of the image-processed pixel points with the value of the other image-processed pixel point of the image-processed pixel points;

if the anticlockwise rotation information indicates that the window corresponding to any two adjacent pixel points is anticlockwise rotated, determining that one of the image-processed pixel points is a G pixel point and the other of the image-processed pixel points is any one of a B pixel point, an R pixel point and an X pixel point;

and if any one of the two random adjacent pixel points is the B pixel point, interchanging the value of the B channel and the value of the R channel in the three-channel values of the pixel points after the image processing.

8. An image processing apparatus characterized by comprising:

the first processing module is used for acquiring the positions of any two adjacent pixel points in a target image on the target image;

the second processing module is used for determining transformation processing information based on the format of the target image and the positions of any two adjacent pixel points on the target image;

the third processing module is used for carrying out conversion processing on the windows corresponding to any two adjacent pixel points based on the conversion processing information to obtain converted windows; the window corresponding to any two adjacent pixel points is an area formed by a plurality of pixel points, the plurality of pixel points comprise any two adjacent pixel points, and the any two adjacent pixel points are positioned at the center of the window corresponding to the any two adjacent pixel points;

9. An electronic device comprising a memory, a processor and a computer program stored on the memory, characterized in that the processor executes the computer program to implement the steps of the method of any of claims 1-7.

10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1-7 when executed by a processor.