JP2017083533A - Image processing device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To avoid failures of display picture images.SOLUTION: According to an embodiment, an image processing device includes: a first buffer; an image processing unit; and a second buffer. The first buffer is configured to store input images in a first operation period, third operation period and fourth operation period successive to the third operation period. The second buffer is configured to store processed images to be output from the image processing unit in the first operation period, second operation period occupying in between the first operation period and the third operation period, and a fourth operation period, and store a first buffer image to be output from the first buffer in the third operation period. An output control unit is configured to output a second buffer image stored in the second buffer to a display device. The first buffer is configured to store processed images fed back from the second buffer in the second operation period.SELECTED DRAWING: Figure 1

Description

  Embodiments relate to image processing.

  For example, a display device such as a TV (TV receiver) or a PC (Personal Computer) monitor can display an image subjected to various image processing (typically filter processing). Such image processing can be implemented using software or hardware. Comparing the two, there is a problem that the software implementation has a low processing speed, and the hardware implementation has a problem that the circuit scale is large.

  In general, the image processing applied to a given image is only a combination of some of the various image processing. Therefore, for example, by implementing image processing using a programmable device such as FPGA (Field Programmable Gate Array), the circuit scale can be saved. In other words, by reconfiguring a part of a programmable device as needed (partial reconfiguration) or reconstructing all (reconfiguration), various combinations of image processing can be performed on a smaller circuit scale. Can be implemented in hardware. According to such partial reconfiguration technology, there is a possibility that various image processing equivalent to the case where an expensive large-scale FPGA is used may be implemented using an inexpensive medium-scale FPGA.

  However, execution of partial reconfiguration requires a certain amount of time, and its length depends on the circuit scale of the target area. Therefore, if the circuit size of the target area is sufficiently small, even if the partial reconfiguration is performed during the non-display (blanking) period of the image (frame), it is in time for the start of the display (active) period of the next image. Can be made. On the other hand, if the circuit scale of the target region is increased, if the partial reconfiguration is performed during the blanking period, the partial reconfiguration may not be completed by the start of the next image display period. In this case, since the image processing cannot be performed on the next image, the display video is broken.

  For example, a progressive image having an active area of 1920 (pixels) × 1080 (lines) as defined in Code 16 of p13 Table 2 of Non-Patent Document 1, an image called “1080p (full HD or 2K)” is used. Think. The blanking area of a 1080p image is 2200 (pixels) x 45 (lines), and the pixel (pixel) frequency is 148.5 MHz.

  Therefore, the blanking period of 1080p image is only 0.667ms. With a configuration clock frequency of 33 MHz from a 16-bit parallel flash memory of BPI (Byte Peripheral Interface), the circuit scale that can be configured in this short time is about 350,000 (350K) bits. On the other hand, the scale of the filter circuit is often in units of 1,000,000 (1000K) bits. If the configuration speed (bit width x configuration clock frequency) can be increased by 5 to 10 times, display video failure can be avoided. However, increasing the configuration speed has limitations and technical difficulties, and is not realistic. Absent.

  Thus, a display image failure avoiding method or apparatus that is technically easy and feasible is desired.

JP 2014-165651 A

A DTV Profile for Uncompressed High Speed Digital Interfaces: CEA-861-D

  An object of the embodiment is to avoid failure of a display image.

  According to the embodiment, the image processing apparatus includes at least a first buffer, an image processing unit, and a second buffer. The first buffer stores the input image in the first operation period, the third operation period, and the fourth operation period following the third operation period, and the first operation period and the third operation The filtered image fed back from the second buffer is stored in the second operation period that occupies the period. In the first operation period, the second operation period, and the fourth operation period, the image processing unit has performed the filter process by performing the filter process on the first buffer image stored in the first buffer. Generate an image. The second buffer stores the filtered image output from the image processing unit in the first operation period, the second operation period, and the fourth operation period, and from the first buffer in the third operation period. The output first buffer image is stored.

1 is a block diagram illustrating an image processing apparatus according to a first embodiment. 3 is a flowchart illustrating image processing performed by the image processing apparatus in FIG. 1. FIG. 6 is a diagram for explaining an outline of an operation of an image processing apparatus according to a comparative example. FIG. 3 is a diagram for explaining the outline of the operation of the image processing apparatus according to the first embodiment. FIG. 3 is a diagram for explaining an outline of operations of the image processing apparatus according to the first embodiment and the image processing apparatus according to the comparative example.

  Hereinafter, embodiments will be described with reference to the drawings. Hereinafter, the same or similar elements as those already described are denoted by the same or similar reference numerals, and redundant description is basically omitted. The following explanation focuses on the case where the image processing unit becomes temporarily unusable mainly due to execution of partial reconfiguration, but the image processing unit is temporarily disabled due to other factors. The embodiment can also be applied to cases where it is possible.

(First embodiment)
As illustrated in FIG. 1, an image processing apparatus according to the first embodiment includes an input control unit 101, a multiplexer (MUX) 102, a first buffer 103, an image delay unit 104, and an image processing unit. 105, MUX 106, second buffer 107, and output control unit 108.

  The input control unit 101 receives an input image output from an external video output device (not shown) (for example, a video decoder, a hard disk recorder, a disk player, a TV, a PC, a portable terminal, a game machine, etc.). The input control unit 101 receives the input image via the MUX 102 in the first operation period, the third operation period, and the fourth operation period that follows the third operation period. It is stored in the buffer 103.

  The MUX 102 has a first input terminal, a second input terminal, a control terminal, and an output terminal. The MUX 102 receives an input image from the input control unit 101 via the first input terminal. The MUX 102 receives a feedback image described later from the second buffer 107 via the second input terminal. The MUX 102 receives the first selection control signal from the switching control unit 109 via the control terminal. The output terminal of the MUX 102 is connected to the first buffer 103.

  The MUX 102 selects either one of the two input images according to the level of the first selection control signal and guides it to the output terminal. The first selection control signal has a first level (for example, a high level) in the first operation period, the third operation period, and the fourth operation period, and includes the first operation period and the third operation period. In the second operation period that occupies the period between the first and second operation periods, the second operation period has a second level (for example, a low level). The MUX 102 guides the input image to the first buffer 103 if the first selection control signal has the first level. On the other hand, the MUX 102 guides the feedback image to the first buffer 103 if the first selection control signal has the second level.

  The first buffer 103 stores the image output from the MUX 102. Specifically, in the first operation period, the third operation period, and the fourth operation period, the first buffer 103 stores an input image. On the other hand, in the second operation period, the first buffer 103 stores the feedback image. Note that the second operation period can be referred to as a feedback period because feedback to the first buffer 103 is performed. An image stored in the first buffer 103 (hereinafter referred to as a first buffer image) is read out in a timely manner and guided to the image delay unit 104 or the image processing unit 105. The first buffer 103 may be a frame buffer, for example.

  In the present embodiment, the first buffer image (input image or feedback image) is stored in the first buffer 103 as a whole (at least one frame), and then the image delay unit 104 or the image processing unit 105. To be output.

  As will be described later, the image delay unit 104 gives a delay of substantially the same time as the processing delay of the image processing unit 105 to the first buffer image, and guides it to the MUX 106. That is, the image delay unit 104 outputs the first buffer image or the image processing unit from the output of the first buffer image in the first buffer 103 to the input of the MUX 106 even during a period in which the image processing unit 105 cannot be used. It plays the role of keeping the time until the processed image (typically the filtered image) arrives at 105 constant. In order to facilitate understanding in FIG. 1, the first buffer 103 or the second buffer 107 and the image delay unit 104 are illustrated as separate blocks, but the first buffer 103 or the second buffer 107 is illustrated. The function of the image delay unit 104 can also be included in this embodiment, and this embodiment can be applied to any configuration.

  The image processing unit 105 receives the first buffer image from the first buffer 103 in the first operation period, the second operation period, and the fourth operation period. On the other hand, the image processing unit 105 cannot be used in the third operation period. The image processing unit 105 generates a filtered image by performing image processing on the first buffer image. This image processing has a processing delay corresponding to several lines to several tens of lines. The image processing unit 105 outputs the filtered image to the MUX 106.

  In the present embodiment, it is assumed that the image processing unit 105 is mounted using a device capable of partial reconfiguration (or reconfiguration). Then, it is possible to change the filter processing performed by the image processing unit 105 when the switching control unit 109 performs partial reconfiguration. As will be described later, the image processing apparatus in FIG. 1 may be disabled even if an unusable period (corresponding to the third operation period) of the image processing unit 105 occurs due to partial reconfiguration or other factors. The failure of the display image can be avoided. Therefore, by setting only one partial area that can include at least one combination without providing two or more partial reconfiguration areas (partial areas: PA) according to the number of combinations of filters, The image processing unit 105 can be mounted using an inexpensive medium-scale FPGA without sacrificing quality.

  In summary, the image processing unit 105 performs the first image processing in the first operation period and the second operation period, undergoes partial reconfiguration in the third operation period, and in the fourth operation period. Second image processing different from the first image processing can be performed. That is, the second operation period corresponds to a preparation period for partial reconfiguration. Further, the third operation period includes a partial reconfiguration execution period.

  Image processing that can be constructed in the image processing unit 105 includes, for example, image rotation, image color change, image enlargement or reduction, and detection and extraction of a predetermined pattern (eg, edge, white line, etc.) from the image. Alternatively, the processing may include enhancement processing, addition of a predetermined image effect (for example, smoothing), and the like, but is not limited thereto.

  The MUX 106 has a first input terminal, a second input terminal, a control terminal, and an output terminal. The MUX 106 receives the filtered image from the image processing unit 105 via the first input terminal. The MUX 106 receives the first buffer image from the first buffer 103 via the image delay unit 104 via the second input terminal. The MUX 106 receives the second selection control signal from the switching control unit 109 via the control terminal. The output terminal of the MUX 106 is connected to the second buffer 107.

  The MUX 106 selects either one of the two input images according to the level of the second selection control signal and guides it to the output terminal. The second selection control signal has a first level (for example, a high level) in the first operation period, the second operation period, and the fourth operation period, and a second level ( For example, Low level). The MUX 106 guides the filtered image to the second buffer 107 if the second selection control signal has the first level. On the other hand, the MUX 106 guides the first buffer image to the second buffer 107 if the second selection control signal has the second level.

  The second buffer 107 stores the image output from the MUX 106. Specifically, in the first operation period, the second operation period, and the fourth operation period, the second buffer 107 stores a (undisplayed) filtered image. On the other hand, in the third operation period, the second buffer 107 stores the first buffer image (filtered image that has been displayed in the past). An image stored in the second buffer 107 (hereinafter referred to as a second buffer image) is read by the output control unit 108 in a timely manner. Further, the second buffer image is returned to the MUX 102 as the feedback image. The second buffer 107 may be a line buffer, for example. In the present embodiment, the second buffer image (first buffer image or filtered image) is stored in the second buffer 107 after a part (at least one line) is stored in the output control unit 108. To be output.

  The output control unit 108 reads the second buffer image from the second buffer 107 and outputs it to an external video display device (for example, a television receiver, a PC monitor, etc.) not shown in a timely manner. The image output by the output control unit 108 forms a display image by the display device.

  The switching control unit 109 is a mode switch for instructing a change in image processing from a central processing unit (for example, a microcomputer, CPU, GPU), a host device (for example, a PC, a workstation) or a user interface (for example, a remote controller) (not shown). Receive a signal. The switching control unit 109 determines whether or not the image processing unit 105 can currently provide the desired filter processing indicated by the mode switching signal.

  If the image processing unit 105 cannot currently provide the desired filter processing, the third operation period starts after the second operation period corresponding to the preparation period. That is, the switching control unit 109 performs partial reconfiguration on the image processing unit 105 so as to provide desired image processing, and sends the first selection control signal and the second selection control signal having appropriate levels to the MUX 102. And MUX 106 respectively. Further, when the partial reconfiguration of the image processing unit 105 is completed, the switching control unit 109 supplies the first selection control signal and the second selection control signal having the first level to the MUX 102 and the MUX 106, respectively. (Start of the fourth operation period).

The image processing apparatus in FIG. 1 performs image processing as illustrated in FIG.
If it is not during the preparation period before the image processing unit 105 becomes unusable (that is, corresponds to the first operation period, the second operation period, or the fourth operation period) (step S201), the input control unit 101 stores the input image in the first buffer 103 via the MUX 102 (step S202). Subsequently, the first buffer image stored in the first buffer 103 in step S202 is subjected to filter processing by the image processing unit 105 (step S203). The filtered image generated in step S203 is stored in the second buffer 107 (step S204), and is output to a display device (not shown) by the output control unit 108 (step S205).

  On the other hand, if it is during the preparation period before the image processing unit 105 becomes unusable (that is, corresponds to the second operation period) (step S201), the input image is stored in the first buffer at this time. Even when the input image is output, the image processing unit 105 cannot be used (enters the third operation period). Therefore, the second buffer image stored in the second buffer 107 (which corresponds to the filtered image generated immediately before the image processing unit 105 becomes unusable) is used as the first feedback image. The first buffer image stored in the buffer 103 (step S206) and stored in the first buffer in step 206 is subjected to delay processing by the image delay unit 104 (step S207). Then, the image delayed in step S207 is stored again in the second buffer 107 (step S208), and is output to a video display device (not shown) by the output control unit 108 (step S205).

  Here, the effect of the image processing apparatus according to the present embodiment will be described in comparison with the comparative example with reference to FIGS. 3, 4, and 5. The image processing apparatus according to this comparative example can implement various image processing by reconfiguring the partial area as necessary. However, special measures are taken against the failure of the display video during the reconfiguration. Not. For example, the image processing apparatus according to this comparative example may be obtained by deleting the MUX 102 and the MUX 106 from the image processing apparatus of FIG. In order to facilitate understanding, the input timing and output timing of the image are matched in FIG. 5, but in practice, they may not be matched.

  In the image processing apparatus according to the comparative example, before the partial reconfiguration occurs (first operation period and second operation period), the image subjected to the first image processing (F5 ′ and F6 in FIG. 5). ') Can be sequentially output to the display device (Step 1 in FIG. 3). However, during the occurrence of partial reconfiguration (third operation period), the image processing unit (partial area) becomes unusable and a filtered image is not generated, so input to the second buffer is interrupted. (Step 2 in FIG. 3). That is, the display target image is not supplied to the display device, and the display video is broken. After the partial reconfiguration is completed (fourth operation period), the image processing apparatus can sequentially output the image subjected to the second image processing (F8 ″ in FIG. 5) to the display device ( Step 3 in FIG. 3).

  As a modification, it is also possible to supply the input image stored in the first buffer as it is to the display device during the partial reconfiguration. However, according to this modified example, after the image subjected to the first image processing (F6 ′ in FIG. 5) is displayed, the image subjected to the second image processing (F8 ″ in FIG. 5). An image that has not been subjected to any image processing (F7 in FIG. 5) is displayed suddenly until is displayed. Therefore, the display image changes rapidly in a short period of time, which may give the viewer a feeling of strangeness.

  On the other hand, before the partial reconfiguration occurs (first operation period and second operation period), the image processing apparatus in FIG. 1 performs an image subjected to the first image processing (F5 ′ and FIG. 5). F6 ′) can be sequentially output to a display device (not shown).

  1 starts the second buffer image (first image processing immediately before entering the third operation period) when the second operation period preceding the third operation period starts. The filtered image that corresponds to F6 ′ in FIG. 5 is not only output to a display device (not shown) but also saved as a feedback image to the first buffer 103 via the MUX 102 (FIG. 4). Step 1). Then, the image processing apparatus repeats the feedback image until the third operation period ends through the path of the first buffer 103 → the image delay unit 104 → the MUX 106 → the second buffer 107 → the output control unit 108. (Step 2 in FIG. 4). Note that the third operation period can also be referred to as a repeat period because the feedback image is output repeatedly.

  When the third operation period ends and the fourth operation period starts, the image processing apparatus in FIG. 1 outputs the filtered image (F8 ″ in FIG. 5) subjected to the second image processing to the second image processing apparatus. Are output to the display device (Step 3 in FIG. 4).

  That is, the display device connected to the image processing device in FIG. 1 repeats and displays the filtered image (F6 ′ in FIG. 5) generated last in the second operation period immediately before in the third operation period. . Therefore, in this display device, the filtered image that has been subjected to the second image processing (F8 ″ in FIG. 5) without causing the viewer to be aware that the image processing unit 105 is temporarily not functioning. Can be displayed smoothly (pseudo seamless playback).

  As described above, the image processing apparatus according to the first embodiment has the filtered image generated immediately before the image processing unit becomes temporarily unavailable due to partial reconfiguration or other factors. Is output repeatedly. Then, the display device connected to the image processing device makes the viewer almost unaware that the image processing unit is temporarily not functioning (that is, while avoiding the failure of the display image), and then again. The display of the filtered image generated by the usable image processing unit can be started smoothly. Therefore, although this image processing apparatus has only one partial area, it is equivalent to an image processing circuit that requires a large-scale FPGA using an inexpensive medium- and small-scale FPGA without sacrificing the quality of the displayed video. Various functions can be implemented. Due to such advantages, this image processing apparatus is particularly suitable for an internal device or an external device of a product that requires video display such as a low-end or middle-end TV, PC, player or the like.

  The various functional units described in the above embodiments may be realized by using hardware or software. For example, dedicated hardware for realizing a specific function may be used, or dedicated software may be combined with general-purpose hardware such as a processor.

  At least a part of the processing of each of the above embodiments can also be realized by using a general-purpose computer as basic hardware. A program for realizing the above processing may be provided by being stored in a computer-readable recording medium. The program is stored in the recording medium as an installable file or an executable file. Examples of the recording medium include a magnetic disk, an optical disk (CD-ROM, CD-R, DVD, etc.), a magneto-optical disk (MO, etc.), and a semiconductor memory. The recording medium may be any recording medium as long as it can store the program and can be read by the computer. The program for realizing the above processing may be stored on a computer (server) connected to a network such as the Internet and downloaded to the computer (client) via the network.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

101 ... Input control unit 102 ... MUX
103: First buffer 104: Image delay unit 105: Image processing unit 106: MUX
107 ... Second buffer 108 ... Output control unit 109 ... Switch control unit

Claims (4)

A first buffer for storing an input image in a first operation period, a third operation period, and a fourth operation period following the third operation period;
Stored in the first buffer in the first operation period, the second operation period that occupies between the first operation period and the third operation period, and the fourth operation period. An image processing unit that generates a processed image by performing image processing on the first buffer image;
Stores processed images output from the image processing unit in the first operation period, the second operation period, and the fourth operation period, and outputs them from the first buffer in the third operation period A second buffer for storing the first buffer image to be stored,
The first buffer stores a processed image fed back from the second buffer in the second operation period.
Image processing device.   The image processing unit performs first image processing on the first buffer image in the first operation period and the second operation period, and performs the first buffer image in the fourth operation period. The image processing apparatus according to claim 1, wherein second image processing different from the first image processing is performed on the image processing apparatus.   The image processing apparatus according to claim 1, further comprising a switching control unit that performs partial reconfiguration of the image processing unit in the third operation period. Storing an input image in the first buffer in a first operation period, a third operation period, and a fourth operation period following the third operation period;
Stored in the first buffer in the first operation period, the second operation period that occupies between the first operation period and the third operation period, and the fourth operation period. An image processing unit performs image processing on the first buffer image being generated to generate a processed image;
Storing the processed image output from the image processing unit in the first operation period, the second operation period, and the fourth operation period in a second buffer;
Storing the first buffer image output from the first buffer in the third operation period in the second buffer;
An image processing method comprising: feeding back a processed image from the second buffer and storing the processed image in the first buffer in the second operation period.