JP6978720B2 - Image processing device - Google Patents

Description

The present invention relates to an image processing apparatus.

Patent Document 1 discloses an image processing apparatus. The pixels of the image sensor are assigned to the pixels of the display. When the pixels do not have a one-to-one correspondence, the RGB values are associated with the pixels of the display according to the interpolation process.

International Publication No. 2008/139757

For example, when the number of pixels of the display is smaller than the number of pixels of the image pickup element, when a plurality of pixels of the image pickup element are assigned to one pixel of the display, the plurality of RGB information is combined into one pixel. Most of the RGB information is discarded. As a result, the image becomes blurry.

An object of the present invention is to provide an image processing device capable of substantially increasing the resolution of a display even when the number of pixels of the display is smaller than the number of pixels of the image pickup element.

One embodiment of the present invention comprises an image pickup device having a plurality of pixels that output an R information signal, a G information signal, and a B information signal designated by one coordinate value as information of one pixel, and the output one pixel. The R information signal and the G information are attached to the R element, the G element, and the B element on the display, which are present for each of the receiving unit for receiving the information and the pixel on the image sensor and individually assigned the coordinate values. The present invention relates to an image processing apparatus including a signal and a coordinate conversion unit for assigning the B information signal.

Generally, in a display, one pixel is formed by a combination of an R element, a G element, and a B element. Colors are reproduced by combining RGB. Therefore, instead of managing the R element, G element, and B element in one pixel with one coordinate value, the coordinate values are individually assigned to the R element, G element, and B element in one pixel in the image. It is possible to substantially increase the resolution of the display in a part where the color is weak or when there is not much color information in the image at night or the like. The principle of increasing the resolution will be described later.

The coordinate conversion unit sets coordinates for each of the R information signal, the G information signal, and the B information signal according to the coordinate conversion, and the plurality of R information signals having the set coordinate values, and the plurality of the above. R information for interpolating between B information signals and a plurality of G information signals and having new coordinates to cause the R element corresponding to the coordinates of the display to emit light, and the coordinates of the display. It may be provided with an interpolation unit that generates G information for causing the G element corresponding to the above to emit light and B information for causing the B element corresponding to the coordinates of the display to emit light.

It is desirable that the number of pixels in the horizontal direction of the image received by the receiving unit is larger than the number of pixels in the horizontal direction of the image output from the interpolation unit. This is because even if the present invention is applied, the resolution finally displayed on the display cannot be increased beyond the resolution of the image received by the receiving unit. It is a well-known fact that the resolution improves as the number of pixels increases.

The coordinate conversion unit may receive the rotation angle signal for specifying the rotation angle of the display and specify the coordinate value according to the specified rotation angle. Even if the arrangement of the R element, the G element, and the B element changes according to the rotation of the display, the image can be displayed according to the change. The discomfort of the image can be avoided.

The display device can include the image processing device as described above. In the display device, the resolution of the display can be substantially increased without increasing the number of pixels in the display.

As described above, according to the disclosed image processing apparatus, it is possible to display an image having a resolution higher than the resolution determined by the number of pixels of the display.

It is a conceptual diagram which shows the whole image of the display device, that is, the display unit which concerns on one Embodiment. It is a flowchart which shows the processing operation of an image processing apparatus roughly. It is a conceptual diagram which shows the whole image of the display device, that is, the display unit which concerns on a modification. It is a schematic diagram explaining that the resolution is improved by image processing. It is a photograph which shows the image generated by this embodiment and the image generated by a conventional method. It is a comparison diagram of the image generated by the conventional method and the image generated by this invention.

Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. First, as a precondition of the present invention, it will be described that the number of pixels in the horizontal direction of the image pickup element 24 is larger than the number of pixels in the horizontal direction of the display panel 12.

FIG. 1 schematically shows an overall image of a display device according to an embodiment of the present invention. The display unit 11 includes a display panel 12 and an image processing device 13 that is connected to the display panel 12 and supplies a display image signal to the display panel 12. The display panel 12 has a plurality of pixels 14, 14 ... That constitute the screen. Pixels 14, 14 ... Are arranged in the vertical direction Vc and the horizontal direction Hr of the screen.

Each pixel 14 of the display panel 12 is formed by one R (red) element 15, one G (green) element 16 and one B (blue) element 17. The R element 15 emits light in red. The G element 16 emits green light. The B element 17 emits light in blue. Here, the R element 15, the G element 16, and the B element 17 are arranged in order from the left to the right of the screen, but the order of the arrangement is not limited to this order. An electric signal is supplied to the individual elements 15, 16 and 17 according to the brightness. Coordinate values on the display 12 are assigned to each of the elements 15, 16 and 17.

The image processing device 13 is included in the reception unit 21 that sequentially receives the information of one pixel including the R information signal, the G information signal, and the B information signal output from the image pickup element 24 together with the coordinate values, and the information of the received one pixel. The R information signal, the G information signal, and the R information signal are assigned to the coordinate conversion unit 22 that allocates the R information signal, the G information signal, and the R information signal to the R element 15, the G element 16, and the B element 17, which constitute the pixel 14 of the specific coordinates of the display unit 12, respectively. It includes an encoder 23 that generates a signal to be transmitted to the display 12 based on the R information signal, the G information signal, and the B information signal.

The image sensor 24 is connected to the receiving unit 21. For example, a CCD (charged coupled device) image sensor or a CMOS (complementary metal oxide semiconductor) image sensor may be used for the image pickup element 24. The image pickup device 24 is incorporated in an input device 25 such as a camera. A lens 26 is combined with the image sensor 24. Here, for example, a wide-angle lens or a fisheye lens is used for the lens 26. The image sensor 24 outputs one R information signal, one G information signal, and one B information signal from each pixel on the image sensor. The receiving unit 21 receives these R information signals, G information signals, and B information signals.

The input device 25 may be a decoder that receives a video signal from another device, or may be a reader that reads an image or a moving image from a recording medium such as an SD card. In addition, the input device 25 may include a converter that converts a YUV signal into an RGB signal.

The receiving unit 21 includes an image memory 28. The image memory 28 may be composed of an SRAM (static random access memory) or an SDRAM (synchronous dynamic random access memory). In the image memory 28, information on the R element 15 (luminance value), information on the G element 16 (luminance value), and information on the B element 17 (luminance value) for each individual pixel on the image sensor 24 specified by the coordinate values. Is remembered.

The coordinate conversion unit 22 includes a coordinate calculation unit 29 and an interpolation unit 31. The coordinate calculation unit 29 sets the brightness of the element corresponding to the coordinates (x, y) on the display 12 to the coordinates (X, Y) output from the individual pixels on the image sensor 24 and the R information signal with the brightness. , G information signal or B information signal . The coordinate values of the R element, the G element, and the B element on the display 12 exist individually. In the present invention, the coordinates in the x direction will be described as integers in 1/3 increments rather than discrete values. Assuming that the coordinates of a certain G element are (x, y), the coordinates of the R element adjacent to the left side in the x direction are (x-1 / 3, y), and the coordinates of the B element adjacent to the right side in the x direction are The coordinates are (x + 1/3, y). Further, here, in addition to the above-mentioned addition of the coordinates, image conversion to, for example, a bird's-eye view image may be performed. Since the method of converting to a bird's-eye view image is a known technique, the description thereof is omitted here. The coordinate calculation unit 29 may be configured by a general-purpose CPU (central processing unit) or FPGA (field programmable gate array). For example, a ROM (read-only memory) is connected to the CPU or FPGA. A software program executed by the CPU or FPGA is stored in the ROM. Alternatively, the coordinate calculation unit 29 may be configured by an ASIC (Integrated Circuit for Specific Applications) that executes a dedicated calculation. In addition, the coordinate calculation unit 29 may be composed of a ROM for storing the calculation result and a ROM reader. Alternatively, the coordinate calculation unit 29 may be composed of a ROM and a ROM reader that stores a part of the calculation results, and an interpolation circuit that calculates the remaining coordinates from the ROM and the ROM reader by interpolation.

The coordinates of the pixel that outputs the R information signal on the image pickup element 24, the coordinates of the pixel that outputs the G information signal and the R information signal , and the R element, the G element, and the B element on the display 12 to which the G information signal is input. The relationship between the coordinates is expressed by the following equation. Here, (Xr, Yr) is the coordinate of the pixel on the image sensor 24 referred to for determining the brightness of the R element on the display 12, and (Xg, Yg) is the brightness of the G element on the display 12. The coordinates of the pixels on the image sensor 24 referred to for determining, and (Xb, Yb) are the coordinates of the pixels on the image sensor 24 referred to for determining the brightness of the element B on the display 12. Each is shown. Xr, Yr, Xg, Yg, Xb, Yb take integer values. For the specific processing of the function f and the function g, prepare a table in which the coordinates of the element on the display 12 corresponding to the coordinates of the pixels on the image sensor 24 are predetermined, and refer to the table to obtain the coordinates. The process of performing conversion is common.

The interpolation unit 31 performs interpolation processing in order to improve the resolution of the enlarged image portion when partially enlarging the image due to the above-mentioned bird's-eye view conversion or the like. To describe an example of the interpolation processing between R elements, the R information of the coordinates between the two R elements is newly added based on the luminance and coordinate information of the two R elements adjacent to the horizontal Hr stored in the image memory 28. The brightness is the average value of the two R elements. The interpolation process is not limited to the above-mentioned horizontal method, but may be generated by a weighted average from the distance between the coordinates of a plurality of surrounding R information and the coordinates of the newly generated R information and the respective brightness, or linear interpolation or buy. Cubic, Ranchos and other interpolation techniques may be used.

Next, the operation of this embodiment will be described. In the display panel 12, the origin of the two-dimensional coordinates is set in one of the four corners. The x-axis is set in the horizontal direction of the screen from the origin, and the y-axis is set in the vertical direction. Coordinate values are individually set for each of the R element 15, the G element 16, and the B element 17. A new coordinate value (x, y) is given by the coordinate conversion unit 29 in order to assign an information signal with a coordinate value (X, Y) output from each pixel of the image pickup element 24 to a specific element on the display 24. Based on the result, the element on the display 24 emits light.

As shown in FIG. 2, 0 is input to the coordinates x and y of the element on the display 12 stored in the coordinate calculation unit 29 in steps S1 and S2. In step S3, the coordinate calculation unit 29 has the coordinate values of the specific pixels on the image pickup element 24 corresponding to x = 0-1 / 3 and y = 0 based on the function f and the function g that perform the coordinate conversion and the above-mentioned bird's-eye view conversion. (Xr, Yr) is calculated. In step S4, the interpolation unit 31 acquires the R information of the coordinate values (Xr, Yr) and the R information of the coordinate values around the coordinate values (Xr, Yr), and performs the above-mentioned interpolation processing. These R information may be taken in from the image memory 28. The R information calculated by interpolation is output in step S5.

In step S6, the coordinate conversion unit 29 uses the coordinate values (Xg, Yg) of the specific pixel on the image pickup device 24 corresponding to x = 0 and y = 0 based on the function f and the function g that perform image conversion such as the above-mentioned bird's-eye view conversion. ) Is calculated. In step S7, the interpolation unit 31 acquires the G information of the coordinate values (Xg, Yg) and the G information of the coordinate values around it, and performs the above-mentioned interpolation processing. These G information may be taken in from the image memory 28. The G information calculated by interpolation is output in step S8.

In step S9, the coordinate calculation unit 29 uses the coordinate values (Xb) of the specific pixels on the image sensor 24 corresponding to x = 0 + 1/3 and y = 0 based on the function f and the function g that perform image conversion such as the above-mentioned bird's-eye view conversion. , Yb) is calculated. In step S10, the interpolation unit 31 acquires the B information of the coordinate values (Xb, Yb) and the B information of the coordinate values around the coordinate values (Xb, Yb), and performs the above-mentioned interpolation processing. The B information may be taken in from the image memory 28. The B information calculated by interpolation is output in step S11.

Here, the pixel existing at the coordinates of x = 0-1 / 3 and y = 0 on the display 12 is the R element, and the pixel existing at the coordinates of x = 0 and y = 0 is the G element and x. The pixel existing at = 0 + 1/3 and y = 0 is a B element. The luminance given to each is the luminance of a specific pixel on the image sensor 24 as described in the above step, and the luminance of this one pixel is the value of the R information signal and the B information signal possessed by one pixel. The total value or the average value of the value of and the value of the G information signal may be used.

In step S12, pixels 14 adjacent to each other in the Hr direction are specified on the screen of the display panel 12. The processing of steps S3 to S12 is repeated until the maximum horizontal coordinate value (wise) is reached in step S13. When the completion of the processing of the coordinate calculation unit 29 and the interpolation unit 31 is recognized in the horizontal direction in step S13, the processing operation shifts to step S14. The position of the pixel 14 is shifted by one in the vertical direction. If it is determined in step S15 that the processing operation has not been completed, the processing operation returns to step S2. Subsequently, the processing of the coordinate calculation unit 29 and the interpolation unit 31 is similarly performed in the horizontal direction. When the maximum coordinate value (height) in the vertical direction is recognized in step S15, a new frame is acquired. In this way, the image is reproduced.

Next, by applying the above-mentioned method, the principle of increasing the resolution when the saturation of the entire image is low at a part of the image where the saturation is low, or at night, etc. will be described with an example. FIG. 4 is a schematic diagram showing that an image captured by an image sensor having 1920 x 1080 pixels is displayed as an image on a 640 x 360 display. Since the improvement in resolution according to the present invention is exhibited only in the horizontal direction, the pixels in the y direction are omitted. FIG. 4A is a part of an image captured by an image sensor of 1920x1080 pixels, FIG. 4B is a part of an image displayed on a 640x360 display by a conventional method, and FIG. 4C is shown in FIG. Is a part of an image displayed on a 640x360 display to which the present invention is applied.

Conventionally, as shown in FIGS. 4A and 4B, an image captured by an image sensor having 1920 pixels in the horizontal direction is displayed on a display having 640 pixels in the horizontal direction. For example, an average of the brightness and saturation of three adjacent pixels of the image pickup device is applied as the brightness value and saturation value of one pixel of the display. The horizontal resolution is 1/3 of that captured by the image sensor. In the present invention, as shown in FIGS. 4A and 4C, the brightness value of one pixel of the image sensor (R information signal value, G information signal value, B information signal total value or average value). Is applied as the brightness value of the R element in one pixel of the display, and the next pixel of the image sensor is sequentially applied to the G element, the next pixel is applied to the B element, and so on, 1: 1. , Horizontal resolution is maintained.

One pixel of the image pickup element is composed of an R element, a G element, and a B element, and the total value of the respective luminances is the luminance of one pixel. By applying the sum of the brightness values of the red color captured by the R element of the image pickup element, the green color captured by the G element, and the blue color captured by the B element to the R element of the display, the element of the display emits red light. .. When such a method is carried out, no matter what color is captured by one pixel of the image pickup element, it is displayed as red on the display, so that the color of the image displayed on the display is captured by the image pickup element. Even if the image is locally significantly different from the color or the image captured by the image pickup element is close to black and white, it is considered that the image is displayed as an irregular color on the display. However, as shown in FIG. 5 (A), when the image to which the present invention is applied is viewed as a whole image, such a color defect is not seen as compared with the comparative example according to the conventional method of FIG. 5 (B). The inventor has experimentally found that no match occurs. The applicant submits the color image in the property submission form as a reference material that color inconsistency does not occur.

FIG. 6A is a graph showing an image of a chart displayed on a display by the above-mentioned conventional method and a change in brightness of one line L in the horizontal direction. FIG. 6B is an image and a graph to which the present invention is applied. On one line L on the display, there is a resolution chart in which black-and-white stripes gradually narrow from left to right (the lateral spatial frequency gradually increases from left to right). do. In the graph of the conventional method shown in FIG. 6A, it can be seen that the amplitude of the luminance following the change in color black gradually decreases toward the right, and the change in luminance is not reproduced. On the other hand, in the graph of the method of the present invention shown in FIG. 6B, it can be seen that the decrease in the amplitude of the luminance is smaller than that of the conventional method. That is, the change in brightness is reproduced. As a result, it can be seen that the resolution is higher than that of the conventional method.

As shown in FIG. 3, the rotation angle specifying unit 37 may be connected to the coordinate conversion unit 22. For example, the gyro sensor 38 incorporated in the display unit 11 may be connected to the rotation angle specifying unit 37. The gyro sensor 38 detects the orientation of the display panel 12 based on gravity. The gyro sensor 38 detects the rotation angle of the display panel 12 and outputs a rotation angle signal. The rotation angle signal specifies the rotation angle of the display panel 12. The rotation angle specifying unit 37 receives the rotation angle signal and replaces the coordinate values quoted in the function f and the function g according to the rotation angle of the display panel 12. Specifically, when the image conversion of the R pixel is described, assuming that the rotation angle θ, Xr'= f (x− (1/3) × COSθ, y− (1/3) × SINθ), Yr'= g (x +). (1/3) × COSθ, y + (1/3) × SINθ), and introduce a trigonometric function in the coordinate values quoted for each function. Even if the arrangement of the R element 15, the G element 16, and the B element 17 changes according to the rotation of the display panel 12, the image can be displayed according to the change. The discomfort of the image can be avoided.

The coordinate conversion unit 22 includes an interpolation unit 31 that performs interpolation for each R information signal, G information signal, or B information signal. Even if there is a discrepancy between the coordinate values of the R element 15, G element 16 and B element 17 and the coordinate values specified by the position information, the image is reproduced on the screen as accurately as possible according to the interpolation of each RGB. Can be

11 ... Display device (display unit), 12 ... Display (display panel), 13 ... Image processing device, 14 ... Pixel, 15 ... R element, 16 ... G element, 17 ... B element, 21 ... Receiver, 22 ... Coordinates Conversion unit, 28 ... image memory, 31 ... interpolation unit.

Claims (4)

An image sensor having a plurality of pixels that outputs an R information signal, a G information signal, and a B information signal specified by one coordinate value as information of one pixel, and an image sensor.
A receiving unit that receives the output information of one pixel, and
The R information signal, the G information signal, and the B information signal are assigned to the R element, the G element, and the B element on the display, which exist for each of the pixels on the image pickup element and are individually assigned coordinate values. Equipped with a coordinate conversion unit
The coordinate conversion unit is
Coordinates are set for each of the R information signal, the G information signal, and the B information signal according to the coordinate conversion, and a plurality of the R information signals having the set coordinate values, a plurality of the B information signals, and a plurality of the B information signals are set. The R information for causing the R element corresponding to the coordinates of the display to emit light by interpolating each of the G information signals of the above and having new coordinates, and the G element corresponding to the coordinates of the display. An image processing apparatus including an interpolating unit that generates G information for causing light emission and B information for causing B element corresponding to the coordinates of the display to emit light. In the image processing apparatus according to claim 1, the number of pixels in the horizontal direction of the image received by the receiving unit is larger than the number of pixels in the horizontal direction of the image output from the interpolation unit. Device. In the image processing apparatus according to claim 1 or 2, the coordinate conversion unit receives a rotation angle signal for specifying the rotation angle of the display, and designates the coordinate value according to the specified rotation angle. Characteristic image processing device. A display device comprising the image processing device according to any one of claims 1 to 3.

Images