JP2014072846A - Image composition device and image composition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image composition device and an image composition method capable of suppressing an access frequency to a memory in which image data in image composition is stored.SOLUTION: An image processor 12a executes processing for compositing first and second Bayer signals of a Bayer format by an image composition part 22. A conversion processing part 71 executes processing for converting a pixel value of a second target pixel on the basis of the pixel value of peripheral pixels to be the same as a color corresponding to a first target pixel among peripheral pixels of the second target pixel in the case where colors corresponding to each of the first and second target pixels to be an object of composition processing of the first and second Bayer signals are different.

Description

  The technology disclosed in the present application relates to an image composition device and an image composition method for combining other image data with image data input by an image input device.

  2. Description of the Related Art Conventionally, an imaging apparatus such as a digital camera processes image data (including moving images) acquired by an imaging element such as a CCD (Charge Coupled Device) image sensor by a plurality of processing units and stores the processed data in a memory. The imaging apparatus temporarily stores in the memory the image data processed when each processing unit appropriately processes the image data.

  In recent years, such an imaging apparatus is expected to have a configuration in which the access frequency is suppressed because the memory access frequency increases as the number of pixels increases and the frame rate increases, and the bandwidth and power consumption in memory access increase. It is rare.

  On the other hand, some imaging devices have a function called an on-screen display (OSD) that superimposes and displays other images such as menu display and various setting information when displaying image data on a display unit, for example. . In addition, in such an image synthesis technique, a technique called augmented reality (AR) that superimposes and displays image data such as a computer graphics (CG) image in real time on a captured image has attracted attention. Yes.

  For example, as an imaging device, Bayer-format image data (hereinafter referred to as “Bayer signal”) acquired by the imaging device is converted into YCbCr-format image data (hereinafter referred to as “YCbCr signal”), and the converted YCbCr signal Some of them synthesize other images (for example, Patent Document 1).

JP 2009-100029 A

  In the imaging apparatus having the above-described image composition function, for example, demosaicing processing (pixel interpolation processing) is performed on the Bayer signal as preprocessing for converting the Bayer signal into image data of another data format (for example, YCbCr signal). Apply. In this demosaicing process, a process of interpolating the pixel values of the other two pixels (G (green) and B (blue) pixels) with respect to one Bayer-type pixel (for example, R (red) pixel) is executed. As a result, the amount of data per pixel increases in the image data after interpolation processing. Accordingly, the memory access frequency increases in the image composition using the interpolated image data.

  The technology disclosed in the present application has been proposed in view of the above problems, and provides an image composition device and an image composition method capable of suppressing the access frequency to a memory in which image data is stored in image composition. For the purpose.

  An image composition device according to a technique disclosed in the present application indicates composition position information including information on a pixel position for combining a first image signal input from an image input device and a second image signal stored in a memory. And a color corresponding to the first pixel of interest between the first pixel of interest of the first image signal and the second pixel of interest of the second image signal based on the combined position information, When the color corresponding to the second target pixel of the second image signal is different, the pixel value of the peripheral pixel having the same color as the color corresponding to the first target pixel among the peripheral pixels of the second target pixel is set. A conversion processing unit that sets a pixel value of the second target pixel based on the pixel value, and a pixel of the second target pixel that is set by the conversion processing unit to the pixel value of the first target pixel based on the combined position information Image composition to synthesize values and generate third image signal And, equipped with a.

  In addition, the image composition method according to the technique disclosed in the present application includes composite position information including information on a pixel position for combining the first image signal input from the image input device and the second image signal stored in the memory. And a color corresponding to the first target pixel with respect to the second target pixel of the second image signal to be combined with the first target pixel of the first image signal based on the combination position information. , When the color corresponding to the second target pixel is different, the second pixel of interest based on the pixel value of the peripheral pixel having the same color as the color corresponding to the first target pixel among the peripheral pixels of the second target pixel. A pixel value of two target pixels is set, and based on the combination position information, the set second target pixel is combined with the first target pixel of the first image signal to generate a third image signal.

  According to the image synthesizing apparatus and the image synthesizing method according to the technique disclosed in the present application, it is possible to suppress the access frequency to the memory in which the image data in the image synthesis is stored.

1 is a block diagram illustrating an imaging apparatus according to a first embodiment. The block diagram which shows the state of the image data in the image composition process of 1st Embodiment. The figure for demonstrating the 1st and 2nd Bayer signal of 1st Embodiment. The flowchart of the image composition process of 1st Embodiment. The block diagram which shows the state of the image data in the image composition process of 2nd Embodiment. The block diagram of the conversion process part of 2nd Embodiment. The figure for demonstrating the 1st and 2nd Bayer signal of 2nd Embodiment. The figure for demonstrating calculation of the pixel value of a 2nd attention pixel. The figure for demonstrating calculation of the pixel value of a 2nd attention pixel. The figure for demonstrating calculation of the pixel value of a 2nd attention pixel. The figure for demonstrating calculation of the pixel value of a 2nd attention pixel. The figure for demonstrating calculation of the pixel value of a 2nd attention pixel. The figure for demonstrating calculation of the pixel value of a 2nd attention pixel. The figure for demonstrating calculation of the pixel value of a 2nd attention pixel. The figure for demonstrating calculation of the pixel value of a 2nd attention pixel. The figure for demonstrating calculation of the pixel value of a 2nd attention pixel. The figure for demonstrating calculation of the pixel value of a 2nd attention pixel. The figure for demonstrating calculation of the pixel value of a 2nd attention pixel. The figure for demonstrating calculation of the pixel value of a 2nd attention pixel. The block diagram which shows the state of the image data in the image composition process of 3rd Embodiment.

(First embodiment)
With reference to FIG. 1, the structure of the imaging device 10 of 1st Embodiment is demonstrated.
An imaging device 10 illustrated in FIG. 1 is a digital camera, for example, and includes an imaging unit 11, an image processing device 12, a memory 13, an external storage medium 15, and a display unit 16.

  The imaging unit 11 includes an imaging element 11a and an A / D conversion circuit 11b. The image sensor 11a is, for example, a CCD image sensor, and receives incident light through a color filter in which color filter components {red (R), green (G), and blue (B)} are arranged in a Bayer array. The image sensor 11a converts incident light received through the color filter into an image signal and outputs the image signal to the A / D conversion circuit 11b. The A / D conversion circuit 11b converts an RGB format imaging signal input from the imaging element 11a into image data (Bayer signal) and outputs the image data to the image processing device 12. The bit width of each pixel value in the Bayer signal is 12 bits, for example.

  The image processing device 12 performs image processing and data format conversion processing on the Bayer signal input from the imaging unit 11. The image processing device 12 temporarily stores image data (Bayer signal or the like) in the memory 13 at a predetermined stage of each process. Therefore, the memory 13 functions as a working memory. The memory 13 is, for example, an SDRAM (Synchronous Dynamic Random Access Memory).

  Further, the image processing apparatus 12 stores the image data after image processing stored in the memory 13 in the external storage medium 15 or outputs it to the display unit 16. The external storage medium 15 is an SD memory card (registered trademark), for example. The display unit 16 is, for example, a liquid crystal display (LCD).

Next, the internal configuration of the image processing apparatus 12 will be described.
The image processing apparatus 12 includes a preprocess processing unit 21, an image composition unit 22, an image processing unit 23, a codec processing unit 24, a central processing unit (CPU) 26, an arbitration circuit 27, a storage medium interface circuit 28, and a display interface circuit. 29, and these circuits are connected to each other by a shared bus 31. The shared bus 31 has an address bus, a control bus, and a data bus.

  The preprocess processing unit 21 performs preprocessing such as white balance adjustment, gain adjustment, defect signal correction, and the like on one frame of the Bayer signal input from the imaging unit 11. The preprocess processing unit 21 stores the processed Bayer signal (hereinafter referred to as “first Bayer signal”) in a predetermined area of the memory 13.

  The image synthesizing unit 22 reads the first Bayer signal of one frame preprocessed by the preprocess processing unit 21 from the memory 13 and outputs another Bayer signal (character data for menu display or CG image to the first Bayer signal). Etc.) is superimposed. In the imaging apparatus 10 of the present embodiment, image data to be combined with a first Bayer signal input from the imaging unit 11 via the preprocess processing unit 21 (hereinafter referred to as “second Bayer signal”) is stored in a memory 13 in a Bayer format. Stored in The second Bayer signal is stored in, for example, a non-volatile storage medium (for example, ROM) (character data for menu display, etc.), stored in the external storage medium 15 or calculated by the CPU 26 (CG image, etc.), and a predetermined process. In step, the data is read out to the memory 13 by the CPU 26. The image composition unit 22 stores the synthesized Bayer signal (hereinafter referred to as “third Bayer signal”) in the memory 13.

  The image processing unit 23 reads out one frame of the third Bayer signal synthesized by the image synthesizing unit 22 from the memory 13, performs a demosaicing process or the like on the third Bayer signal, and executes a process of converting the signal into a YCbCr signal. The image processing unit 23 stores the converted YCbCr signal in the memory 13.

  The codec processing unit 24 reads the YCbCr signal converted by the image processing unit 23 from the memory 13, encodes the YCbCr signal by a predetermined method (here, JPEG method), and stores the encoded image data in the memory 13. Store.

  The CPU 26 is a control device that performs overall control of the operation of each circuit, and executes analysis of each image data, setting of necessary information based on the analysis result to each circuit, data writing / reading control, and the like. Further, the CPU 26 outputs data (hereinafter referred to as “compositing position information”) CI indicating to which part of the first Bayer signal the second Bayer signal is to be synthesized to the image synthesizing unit 22. To do. The image synthesis unit 22 performs a synthesis process of the first and second Bayer signals based on the synthesis position information CI.

  Each processing unit (including the image synthesis unit 22) outputs a signal for requesting the right to use the shared bus 31 to the arbitration circuit 27 in order to write or read data in the memory 13. The arbitration circuit 27 sets the right to use the arbitration circuit 27 for each processing unit in response to the request signal.

  The storage medium interface circuit 28 executes data input / output with respect to the external storage medium 15. The CPU 26 reads out the JPEG image data converted by the codec processing unit 24 from the memory 13 and stores it in the external storage medium 15 via the storage medium interface circuit 28. The display interface circuit 29 converts the image data transferred from the memory 13 into data in a format that can be supported by the display unit 16 and outputs the data to the display unit 16.

Next, the configuration of the image processing unit 23 will be described with reference to FIG.
As illustrated in FIG. 2, the image processing unit 23 includes an interpolation processing unit 41, a gamma conversion unit 42, an outline enhancement unit 43, and a format conversion unit 44.

  The interpolation processing unit 41 performs demosaicing processing (pixel interpolation processing) on the third Bayer signal combined by the image combining unit 22. In the demosaicing process, for example, image data (hereinafter, “RGB signal”) obtained by interpolating pixel values of other peripheral pixels (G and B pixels in the case of R pixels) for each pixel of the third Bayer signal. Generated). The RGB signal has a data width of 36 bits per pixel when, for example, a Bayer signal having a data width of 12 bits per pixel is interpolated, that is, compared to the data amount of the third Bayer signal before processing. The data amount is tripled.

  The gamma conversion unit 42 performs a process for matching device characteristics (gamma characteristics) of the display unit 16 or the like, that is, so-called gamma correction. The contour emphasizing unit 43 performs processing such as contour (edge) enhancement and noise removal using a spatial filter in order to improve the clearness of the entire image. The format conversion unit 44 executes processing for converting RGB signals into YCbCr signals.

Next, composition of image data will be described with reference to FIG.
For example, a process for combining the second Bayer signal 52 with the first Bayer signal 51 shown in FIG. 3 will be described. For convenience of explanation, it is assumed that the first Bayer signal 51 is image data in which 12 vertical and 12 horizontal pixels are arranged in one frame. Further, the second Bayer signal 52 is assumed to be image data (for example, CG image) in which 6 pixels in the vertical direction and 6 pixels in the horizontal direction are arranged.

  Note that the characters shown in each pixel of the first and second Bayer signals 51 and 52 indicate the color filter components {red (R), green (G), blue (B)} of the color filter corresponding to each pixel. A portion indicated by hatching in the second Bayer signal 52 indicates a colored pixel. In addition, the hatched portion of the second Bayer signal 52 indicates a pixel that is colorless (transparent), that is, a pixel in which the pixel value of the first Bayer signal 51 is a combined value when superimposed on the first Bayer signal 51. Yes.

  The image composition unit 22 sequentially performs composition processing on each pixel of the first Bayer signal 51. For example, the image composition unit 22 moves a pixel 62 (hereinafter referred to as “first target pixel”) 62 to be processed by the first Bayer signal 51 from the pixel position (1, 1) on the first line to the pixel position (1, 2). ) Is performed while moving one pixel at a time (in the right direction in FIG. 3) in order. In addition, the image composition unit 22 performs the same processing on each line in the order of 1 line to 12 lines. At this time, the image composition unit 22 similarly moves the pixel position of the processing target pixel (second attention pixel) 63 of the second Bayer signal 52 when the pixel position of the first attention pixel 62 becomes the position to be synthesized. The process of combining the first and second target pixels 62 and 63 is executed.

  First, the case of the arrow 61a shown in FIG. 3 will be described. The case indicated by the arrow 61a indicates a case where the second target pixel 63 is the R pixel at the pixel position (1, 1) when the first target pixel 62 is the R pixel at the pixel position (1, 3). The synthesis process in this case will be described with reference to FIG. The CPU 26 (see FIG. 2) outputs the combined position information CI to the image combining unit 22 (Step 101 in FIG. 4). The combined position information CI includes information such as the position of the pixel to be combined and the size of the image data to be combined. For example, the CPU 26 obtains information on the position at which the synthesis is started (in this case, the pixel position (1, 3) of the first Bayer signal 51) and the number of pixels (6 × 6) of the second Bayer signal 52, as the synthesis position information CI. To the image composition unit 22. Note that the content of the combined position information CI is changed as appropriate according to the size and pixel values of the first and second Bayer signals 51 and 52.

  Next, the image composition unit 22 reads the first Bayer signal 51 and the second Bayer signal 52 of one frame from the memory 13 (Step 102). The image composition unit 22 temporarily stores the first and second Bayer signals 51 and 52 read from the memory 13 in a buffer (not shown) included in the image composition unit 22.

  Next, the image composition unit 22 determines whether or not the first target pixel 62 is a pixel position where the composition process is performed based on the composition position information CI (step 103). For example, if the first line of the first Bayer signal 51 is based on the combined position information CI, the image composition unit 22 counts the number of pixels next to the second Bayer signal 52 from the pixel position (1, 3) where image composition starts. Up to “six” (pixel positions (1, 8)) are determined as composite positions.

  If the image synthesis unit 22 determines that the first target pixel 62 is not the pixel position to be synthesized, the pixel value of the first target pixel 62 in the first Bayer signal 51 is the pixel of the processing result (third Bayer signal). It outputs as a value (step 104). The image synthesizing unit 22 performs the process of step 104 while moving the position of the first target pixel 62 one by one until the pixel position where the first target pixel 62 is combined (in this case, the pixel position (1, 3)). Repeatedly.

Next, when the first target pixel 62 reaches the pixel position (1, 3), the image composition unit 22 reads the second target pixel 63 of the second Bayer signal 52 from the buffer (step 105).
Next, the image composition unit 22 determines whether the second target pixel 63 is colored or transparent (colorless) (step 106). For example, the image composition unit 22 determines that a specific pixel value (all bit values are “1” or all pit values are “0”, etc.) as transparent. In addition, for example, a bit value (α value: “1” if colored, “0” if transparent) is set for each pixel value of the second Bayer signal 52, and image synthesis is performed. The unit 22 may make a determination based on this bit value.

  If the image composition unit 22 determines in step 106 that the pixel value of the second target pixel 63 is colored, the image composition unit 22 outputs the pixel value of the second target pixel 63 as a third Bayer signal (step 107). If the image composition unit 22 determines that the pixel value of the second target pixel 63 is transparent, the image composition unit 22 outputs the pixel value of the first target pixel 62 as a third Bayer signal (step 108). In this way, the image composition unit 22 outputs the pixel value to be output as the third Bayer signal according to the pixel value of the second target pixel 63 of the second Bayer signal 52, which of the first and second target pixels 62 and 63. Execute the process of outputting while switching between.

  Next, the image composition unit 22 determines whether or not the processing for the second Bayer signal 52 is finished, that is, whether or not the second target pixel 63 is the pixel position (6, 6) of the second Bayer signal 52 (step). 109). The image composition unit 22 determines, for example, by comparing the number of pixels (6 × 6 = 36) of the second Bayer signal 52 with the number of processes in step 106 based on the composite position information CI.

  If it is determined in step 109 that the processing for the second Bayer signal 52 has not ended, the processing from step 103 is repeatedly executed. When it is determined in step 109 that the processing for the second Bayer signal 52 has been completed, the pixel value of the first Bayer signal 51 is output as the processing result up to the last pixel (pixel position (12, 12)) of one frame. (Step 110).

  For example, in the case illustrated in FIG. 3, the image composition unit 22 causes the pixel value (pixel) of the first Bayer signal 51 until the first pixel of interest 62 reaches the pixel position (2, 5) of the first Bayer signal 51. (Pixel value corresponding to position (1,1) to pixel position (2,4)) is output as the third Bayer signal. Further, for example, when the first target pixel 62 reaches the pixel position (2, 5), the image composition unit 22 replaces the pixel value of the first target pixel 62 with the pixel value of the second target pixel 63 (pixel position (2, 2, 3) is output as a processing result.

  The image composition unit 22 repeatedly executes the above-described processing on the first Bayer signal 51 read from the memory 13 in units of frames, and stores the third Bayer signal as a processing result in the memory 13. The third Bayer signal stored in the memory 13 is read by, for example, the CPU 26 and displayed on the display unit 16.

  Note that the position where the color (R, G, B) of the first target pixel 62 matches the color of the second target pixel 63 can be set in the composite position information CI, as indicated by arrows 61b and 61c in FIG. For example, the arrow 61c is a case where the R pixel at the pixel position (1, 5) of the first Bayer signal 51 and the R pixel at the pixel position (1, 1) of the second Bayer signal 52 are overlapped. A case where the colors of the first target pixel 62 and the second target pixel 63 do not match will be described in the second embodiment.

As described above, according to the first embodiment, the following effects are obtained.
(1) In the image processing device 12, the second Bayer signal 52 to be combined with the first Bayer signal 51 input from the imaging unit 11 is stored in the memory 13 in the Bayer format. The image synthesis unit 22 synthesizes the first Bayer signal 51 and the second Bayer signal 52 based on the synthesis position information CI input from the CPU 26 and outputs it as a third Bayer signal. The interpolation processing unit 41 performs demosaicing processing on the third Bayer signal. In such a configuration, the synthesizing process can be executed using the first and second Bayer signals 51 and 52 before the demosaicing process.

Here, the first and second Bayer signals 51 and 52 have a smaller data amount per pixel than image data (YCbCr signal or the like) after demosaicing processing used for conventional image synthesis. Thereby, the amount of data transferred once to the memory 13 in the image composition process can be reduced.
Further, in the configuration in which the signal after the demosaicing process is used for the synthesis process as in the prior art, for example, a process corresponding to the first Bayer signal 51 of the present embodiment needs to be converted into a YCbCr signal and stored in the memory. Become. In addition, it is necessary to read out from the memory both signals corresponding to the first and second Bayer signals 51 and 52 of the present embodiment (both YCbCr signals and the like).

  On the other hand, the image composition unit 22 of the present embodiment can execute the access process for the memory 13 in the composition process only by the reading process of the second Bayer signal 52. As a result, the frequency of access to the memory 13 can be suppressed by reducing the amount of data transferred to and from the memory 13 in the image composition process and reducing the number of read / write operations.

(Second Embodiment)
FIG. 5 is a block diagram of the second embodiment. In the first embodiment, the color (R, G, B) corresponding to the first target pixel 62 of the first Bayer signal 51 matches the color corresponding to the second target pixel 63 of the second Bayer signal 52. The synthesis process is illustrated. On the other hand, in the second embodiment, an example of the synthesis process when the colors of the pixels do not match is illustrated. Note that in the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

  As illustrated in FIG. 5, the image processing apparatus 12 a includes a conversion processing unit 71. The conversion processing unit 71 is connected to each circuit by the shared bus 31 (see FIG. 1). As a preprocessing for the second Bayer signal 52 input to the image composition unit 22, the conversion processing unit 71 inputs image data including the second target pixel 63 and outputs the second target pixel according to the color of the first target pixel 62. A process of converting 63 pixel values is executed.

  As illustrated in FIG. 6, the conversion processing unit 71 includes a register unit 72, a control processing unit 73, a buffer unit 74, and a pixel value conversion unit 75. The register unit 72 includes three registers R1 to R3.

  The register R1 stores a value indicating an address where the second Bayer signal 52 in the memory 13 is stored. The register R2 stores a value indicating the pixel type of the first and second Bayer signals 51 and 52. The pixel type here is, for example, a color (R, G, B) corresponding to each pixel position (1, 1) of the first and second Bayer signals 51, 52. In addition, when the color corresponding to the pixel position (1, 1) is a G pixel, the register R2 determines whether the G pixel in the line where the R pixel is arranged or the G pixel in the line where the B pixel is arranged. The value shown is also included (see FIG. 7). The register R3 stores a value indicating the above-described combined position information CI.

  In the register unit 72, the values of the registers R1 to R3 are set by the CPU 26. The control processing unit 73 executes control to read out a predetermined number of pixels of the second Bayer signal 52 from the address of the register R1 in the memory 13 with respect to the buffer unit 74. The data for the predetermined number of pixels is data including the second pixel of interest 63, and is, for example, a partial image of 3 in the vertical direction and 3 in the horizontal direction shown in FIG. The pixel value conversion unit 75 executes a process of converting pixel values by sequentially inputting the partial images input to the buffer unit 74 based on the control of the control processing unit 73.

  The control processing unit 73 outputs the values of the registers R2 and R3 to the pixel value conversion unit 75. The pixel value conversion unit 75 selects an arithmetic circuit that converts the pixel value of the second pixel of interest 63 based on the values of the registers R2 and R3. The pixel value conversion unit 75 inputs the pixel value of the partial image supplied from the buffer unit 74 according to the selected arithmetic circuit. Then, the pixel value conversion unit 75 executes a process of setting the pixel value calculated by the arithmetic circuit as the pixel value of the second target pixel 63 and outputting it to the image composition unit 22.

Here, there are the following four combinations of colors of the first and second target pixels 62 and 63 in the synthesis process of the first and second Bayer signals 51 and 52.
(A) The second target pixel 63 of the R pixel is combined with the first target pixel 62 of the R pixel (arrow 64 in FIG. 7, first embodiment).
(B) The second target pixel 63 of the B pixel is combined with the first target pixel 62 of the R pixel (see arrows 65a and 65b in FIG. 7 and FIGS. 8 to 11).
(C) The second target pixel 63 of the G pixel in the line in which the R pixel is arranged is combined with the first target pixel 62 of the R pixel (see arrows 66a and 66b in FIG. 7 and FIGS. 12 to 15).
(D) The second target pixel 63 of the G pixel on the line where the B pixel is arranged is combined with the first target pixel 62 of the R pixel (see FIGS. 16 to 19).
Below, it demonstrates in order of said (b)-(d). In the case of (a) (when the colors match, the partial pixel data 81 in FIG. 7) is the same process as in the first embodiment, and thus the description thereof is omitted. Moreover, the character shown by each pixel of FIGS. 8-19 shows a pixel value. Further, the central pixel of each of the first and second Bayer signals 51 and 52 in FIGS. 8 to 19 indicates the first and second target pixels 62 and 63.

In the case of (b) In the case of (b), the image processing apparatus 12a converts the second Bayer signal 52 by the conversion processing unit 71 (arrow 65a) as shown in FIG. 7, and converts the second Bayer after the conversion. A process (arrow 65b) for combining the signal 52a with the partial pixel data 82 of the first Bayer signal 51 is executed.

  More specifically, the first and second Bayer signals 51 and 52 are regularly arranged with a color corresponding to each pixel in a Bayer array. The values of the registers R2 and R3 include the pixel type of the first and second Bayer signals 51 and 52 (the color of the pixel position (1, 1)) and the combined position information CI (the pixel on which the Bayer signals 51 and 52 are superimposed). Information on location). Accordingly, the pixel value conversion unit 75 determines in which color the first and second Bayer signals 51 and 52 are superimposed based on the values of the registers R2 and R3, that is, which of the above (a) to (b). It can be detected whether this is the case. As a result, the pixel value conversion unit 75 can alternatively select an arithmetic circuit corresponding to cases (a) to (b) and convert the pixel value of the second target pixel 63.

  FIG. 8 shows a case where the first target pixel 62 is an R pixel and the second target pixel 63 is a B pixel in the case of (b). The pixel value of the first target pixel 62 is R11, and the pixel value of the second target pixel 63 is B11. In this case, the pixel value conversion unit 75 executes a process of calculating the pixel value r11 set for the second target pixel 63 from the peripheral pixels of the second target pixel 63. The pixel value conversion unit 75 outputs the pixel value r11 to the image synthesis unit 22, and the pixel value r11 is synthesized by the image synthesis unit 22 into R11 of the first target pixel 62 (first Bayer signal 51).

For example, the pixel value r11 is expressed by the following arithmetic expression using pixel values of the peripheral pixels of the second target pixel 63 (upper left R00, upper right R02, lower left R20, lower right R22).
r11 = (R00 + R02 + R20 + R22) / 4
That is, the pixel value of the peripheral pixel having the same color (the same color on the Bayer array) as the color of the first target pixel 62 among the peripheral pixels adjacent to the second target pixel 63 to the pixel value of the second target pixel 63 after conversion. Set the average value.

Similarly, when the first target pixel 62 shown in FIG. 9 is a B pixel (pixel value B11) and the second target pixel 63 is an R pixel (pixel value R11), the converted pixel value b11 is calculated as follows: It is expressed by a formula.
b11 = (B00 + B02 + B20 + B22) / 4
When the first target pixel 62 (pixel value G11) and the second target pixel 63 (pixel value G11) shown in FIGS. 10 and 11 are the same color, the pixel value G11 of the second target pixel 63 is converted as it is. Is set as the pixel value.

  Here, the pixel value conversion unit 75 calculates the pixel value of the second target pixel 63 from the pixel values of the surrounding pixels as described above. In other words, when the corresponding peripheral pixel is colored, a colored value is set for the second target pixel 63. In view of this, in the second Bayer signal 52 of the present embodiment, transparent pixels are set around colored pixels as shown in FIG. Then, the pixel value conversion process is performed on the second Bayer signal 52, so that the range of colored pixels of the converted second Bayer signal 52a is expanded as compared with that before the conversion. In other words, the conversion process of the pixel value of the second Bayer signal 52 is performed in a more appropriate range by setting the transparent pixel in consideration of the position of the colored pixel after the conversion with respect to the second Bayer signal 52. be able to.

  Regarding the pixel value conversion process, a predetermined value is set when the corresponding peripheral pixel is not set. Specifically, for example, when the R pixel at the pixel position (1, 3) of the second Bayer signal 52 is set as the second target pixel 63, the four pixels around the second target pixel 63 as shown in FIG. There is no pixel corresponding to two (B00, B02) of the two B pixels. In such a case, transparent pixel values (for example, all pit values are “0”) are set as predetermined values for the pixel values B00 and B02. Further, for example, the pixel value at the opposite position may be set with the second target pixel 63 (in this case, the pixel value R11) interposed therebetween (B22 if B00, B20 if B02).

  Then, in the case of (b), the pixel value conversion unit 75 executes a process of selecting an arithmetic circuit that performs the above arithmetic process. The first and second Bayer signals 51 and 52 have pixel colors regularly arranged, and are sequentially input to the arithmetic circuit in which the pixel value of the second target pixel 63 is selected by the pixel value conversion unit 75. The pixel value is converted by.

  For example, as shown in FIG. 7, in the case of (b) (arrows 65a and 65b), the partial pixel data 82 at the synthesis position of the first Bayer signal 51 is B, G, B, ... The pixel colors are arranged in this order. On the other hand, in the second Bayer signal 52, pixel colors are arranged in the order of R, G, R,... On the first line. Therefore, the pixel value conversion unit 75 can execute the conversion process for the first line by alternately inputting the pixel value of the second pixel of interest 63 to the arithmetic circuits corresponding to the processing contents shown in FIGS. 9 and 11. .

  Further, for example, in the partial pixel data 82, pixel colors are arranged in the order of G, R, G,... On the second line. On the other hand, in the second Bayer signal 52, pixel colors are arranged in the order of G, B, G,... On the second line. Therefore, the pixel value conversion unit 75 can execute the conversion process for the second line by alternately inputting the pixel value of the second pixel of interest 63 to the arithmetic circuit corresponding to each of the processing contents shown in FIGS. . In this way, the pixel value conversion unit 75 sets the average value calculated from the peripheral pixels as the pixel value of the second target pixel 63 and outputs the average value to the image composition unit 22. The image composition unit 22 synthesizes (replaces) the converted second target pixel 63 input from the pixel value conversion unit 75 with the first target pixel 62 of the first Bayer signal 51.

In the case of (c) In the case of (c), the image processing apparatus 12a converts the second Bayer signal 52 by the conversion processing unit 71 (arrow 66a) as shown in FIG. A process of combining the signal 52b with the first Bayer signal 51 (arrow 66b) is executed.

FIG. 12 shows a case where the first target pixel 62 is an R pixel and the second target pixel 63 is a G pixel (G pixel on the line where the R pixel is arranged) in the case of (c). The pixel value r11 set for the second target pixel 63 is expressed by the following arithmetic expression using the pixel values (left R10, right R12) of the peripheral pixels of the second target pixel 63.
r11 = (R10 + R12) / 2

Similarly, when the first pixel of interest 62 shown in FIG. 13 is a B pixel (pixel value B11) and the second pixel of interest 63 is a G pixel (pixel value G11), the converted pixel value b11 is expressed by the following equation: It is represented by
b11 = (B10 + B12) / 2

Similarly, when the first target pixel 62 shown in FIG. 14 is a G pixel (G pixel on the line where the R pixel is arranged) and the second target pixel 63 is an R pixel (pixel value R11), the pixel after conversion The value g11 is expressed by the following arithmetic expression.
g11 = (G01 + G10 + G12 + G21) / 4
The case where the first pixel of interest 62 shown in FIG. 15 is the G pixel (pixel value G11) of the B line and the second pixel of interest 63 is the B pixel (pixel value B11) is the same as in FIG. Is omitted.
In this way, the pixel value conversion unit 75 sets the average value of the pixel values of the peripheral pixels corresponding to the color of the pixel of the first target pixel 62 as the pixel value of the second target pixel 63 even in the case of (c). To do.

Case (d) FIG. 16 shows a case where the first pixel of interest 62 is an R pixel and the second pixel of interest 63 is a G pixel (G pixel on the line where the B pixel is arranged) in the case of (d). Yes. The pixel value r11 set for the second target pixel 63 is expressed by the following arithmetic expression using the pixel values of the peripheral pixels of the second target pixel 63 (upper R01, lower R21).
r11 = (R01 + R21) / 2

Similarly, when the first target pixel 62 shown in FIG. 17 is a B pixel (pixel value B11) and the second target pixel 63 is a G pixel (pixel value G11), the converted pixel value b11 is expressed by the following equation. It is represented by
b11 = (B01 + B21) / 2

Similarly, when the first target pixel 62 shown in FIG. 18 is a G pixel (G pixel on the line where the R pixel is arranged) and the second target pixel 63 is a B pixel (pixel value B11), the converted pixel The value g11 is expressed by the following arithmetic expression.
g11 = (G01 + G10 + G12 + G21) / 4
In addition, when the 1st attention pixel 62 shown in FIG. 19 is G pixel (G pixel of the line in which B pixel is arrange | positioned) and the 2nd attention pixel 63 is R pixel (pixel value R11), it is the same as that of the case of FIG. Therefore, the description is omitted.
In this way, the pixel value conversion unit 75 sets the average value of the pixel values of the peripheral pixels according to the color of the pixel of the first target pixel 62 as the pixel value of the second target pixel 63 even in the case of (d). To do.

As described above, according to the second embodiment, the following effects can be obtained.
(1) The conversion processing unit 71 of the image processing device 12a uses the first of the peripheral pixels of the second pixel of interest 63 when the colors of the color filters corresponding to the first and second pixels of interest 62 and 63 are different. A process of calculating the pixel value of the second target pixel 63 based on the pixel values of peripheral pixels having the same color as the color corresponding to the target pixel 62 is executed. As a result, it is possible to combine the first and second Bayer signals 51 and 52 when the colors of the first and second target pixels 62 and 63 are different. That is, Bayer-type image composition processing that can suppress the access frequency to the memory 13 can be processed without being limited by the pixel position.

  (2) The conversion processing unit 71 sets a pixel adjacent to the second target pixel 63 as a peripheral pixel, and sets an average value of pixel values of the peripheral pixel as a pixel value of the second target pixel 63. Thus, by calculating the pixel value from the pixels adjacent to the second target pixel 63, the pixel value of the second target pixel 63 after conversion can be calculated with high accuracy.

  (3) In the register unit 72, the values of the registers R1 to R3 are set by the CPU 26. The control processing unit 73 performs control to output a partial image including the second target pixel 63 from the memory 13 to the buffer unit 74 based on the value of the register R1. The control processing unit 73 outputs the values of the registers R1 and R3 to the pixel value conversion unit 75, and the pixel value conversion unit 75 inputs the partial image from the buffer unit 74 based on the values of the registers R1 and R3. Processing for converting the pixel value of the pixel of interest 63 is executed. In such a configuration, the conversion processing unit 71 that can be easily controlled by the CPU 26 can be configured.

  (4) The values of the registers R2 and R3 include information on the pixel types of the first and second Bayer signals 51 and 52 and the combined position information CI. The pixel value conversion unit 75 can detect the manner in which the first and second Bayer signals 51 and 52 are superimposed based on the values of the registers R2 and R3. As a result, the pixel value conversion unit 75 can alternatively select an arithmetic circuit corresponding to each case (in the case of (a) to (d)), and efficiently executes the pixel value conversion processing. It becomes possible.

  (5) The pixel value conversion unit 75 calculates the pixel value after the conversion of the second target pixel 63 from the pixel values of the peripheral pixels having the same color as the first target pixel 62 among the adjacent peripheral pixels. Based on this, in the second Bayer signal 52, transparent pixels are set around colored pixels (see FIG. 7). Thereby, the conversion process of the pixel value of the 2nd Bayer signal 52 can be implemented in a more suitable range.

(Third embodiment)
FIG. 20 is a block diagram of the third embodiment. The third embodiment has a configuration in which an inverse gamma conversion unit 77 is added to the configuration of the second embodiment. Here, the data output from the imaging unit 11 is a value proportional to the amount of incident light. Accordingly, the first Bayer signal 51 has a linear value. On the other hand, the image data including the second Bayer signal 52 stored in the memory 13 is often non-linear data that has been subjected to gamma correction by the gamma conversion unit 42.

  In the image processing apparatus 12b of the third embodiment, as shown in FIG. 20, an inverse gamma conversion unit 77 that performs inverse gamma correction on the second Bayer signal 52 read from the memory 13 into the conversion processing unit 71 is provided. . The inverse gamma conversion unit 77 executes a process of performing inverse gamma correction that cancels the gamma correction previously applied to the second Bayer signal 52.

As described above, according to the third embodiment, the following effects are obtained.
(1) The inverse gamma conversion unit 77 executes a process of inputting the second Bayer signal 52 from the memory 13, performing reverse gamma correction, and outputting the result to the conversion processing unit 71. Thereby, for example, when a third Bayer signal is displayed on the display unit 16, a difference in luminance caused by device characteristics between a portion corresponding to the first Bayer signal 51 and a portion corresponding to the second Bayer signal 52, etc. Can be reduced.

  The imaging unit 11 is an example of an image input device, the image processing device 12 is an example of an image composition device, the memory 13 is an example of a memory, the CPU 26 is an example of a control unit, and the first Bayer signal 51 is a first image. As an example of the signal, the second Bayer signal 52 is an example of the second image signal, and the third Bayer signal is an example of the third image signal.

Needless to say, the technology disclosed in the present application is not limited to the above-described embodiments, and various modifications and changes can be made without departing from the scope of the invention.
For example, in the second and third embodiments, the conversion processing unit 71 obtains the pixel value of the second target pixel 63 using all the pixel values of the corresponding peripheral pixels. On the other hand, you may make it calculate using arbitrary pixel values.
In addition, although the conversion processing unit 71 uses a pixel adjacent to the second pixel of interest 63 for the calculation, the conversion processing unit 71 is not limited to this, and other adjacent pixels may be used. In this case, the number of pixels of the partial image (image centered on the second target pixel 63) input from the memory 13 to the buffer unit 74 is appropriately changed.

  In each of the above embodiments, the image composition unit 22 may be configured to synthesize a plurality of second Bayer signals (characters, menu displays, CG images, etc.) with the first Bayer signal 51. Moreover, it is good also as a structure which performs a synthetic | combination process in multiple times.

  In each of the above embodiments, a plurality of pixels constituting the image sensor 11a are arranged in a Bayer array via RGB color filters, and the first Bayer signal 51 is handled as Bayer format data. Any data before the interpolation processing (RAW data) may be used, and data of another arrangement may be used. For example, a honeycomb arrangement may be used. Also, other arrangements (double Bayer arrangement etc.) may be used for the Bayer arrangement. In this case, the data format of the second Bayer signal 52 is appropriately changed according to the data format of the first Bayer signal 51.

Moreover, it is good also as a structure which abbreviate | omitted the conversion process part 71 in the structure of the said 3rd Embodiment. In other words, the inverse gamma conversion unit 77 may be added to the configuration of the first embodiment.
Further, the image composition unit 22 may use the Bayer signal output from the A / D conversion circuit 11b as the first Bayer signal. In this case, the pre-process processing unit 21 can be omitted.

  Moreover, it is good also as a structure provided with two or more imaging parts 11. FIG. In this case, it is good also as a structure provided with the image processing apparatus 12 corresponding to each imaging part 11, respectively.

Hereinafter, various aspects of the technology disclosed in the present application will be summarized as additional notes.
(Appendix 1)
A control unit for instructing combined position information including information on a pixel position for combining the first image signal input from the image input device and the second image signal stored in the memory;
Based on the combined position information, a color corresponding to the first target pixel between the first target pixel of the first image signal and the second target pixel of the second image signal, and the second image signal When the color corresponding to the second target pixel is different from the peripheral pixel of the second target pixel, the second target pixel is based on the pixel value of the peripheral pixel having the same color as the color corresponding to the first target pixel. A conversion processing unit for setting the pixel value of the target pixel;
An image combining unit that combines the pixel value of the second target pixel set by the conversion processing unit with the pixel value of the first target pixel based on the combined position information to generate a third image signal;
An image composition device comprising:
(Appendix 2)
The conversion processing unit sets a pixel adjacent to the second pixel of interest as the peripheral pixel, calculates an average value of pixel values of the peripheral pixels, and sets the pixel value of the second pixel of interest;
The image synthesizing device according to supplementary note 1, wherein:
(Appendix 3)
The conversion processing unit
A register unit whose value is set by the control unit;
A control processing unit that performs control based on a value set in the register unit;
A buffer unit for inputting a partial image of the second image signal including the second target pixel from the memory based on the control of the control processing unit;
Based on the value set in the register unit input from the control processing unit, the partial image is input from the buffer unit, and the pixel value of the second target pixel is calculated and set from the peripheral pixels of the partial image. A pixel value converter to
The image synthesizing device according to appendix 1 or 2, further comprising:
(Appendix 4)
In the register unit, a value including the pixel type of the first and second image signals and the combined position information is set by the control unit,
The pixel value conversion unit includes a plurality of arithmetic circuits, and the first and second image signals are based on values including pixel types of the first and second image signals set in the register unit and the combined position information. The image synthesizing device according to any one of appendices 1 to 3, wherein a mode in which the second image signal is superimposed is detected, and an arithmetic circuit corresponding to the detection result is selected.
(Appendix 5)
The supplementary notes 1 to 4 further comprising an inverse gamma conversion unit that inputs the second image signal from the memory, performs inverse gamma correction on the second image signal, and outputs the second image signal to the conversion processing unit. The image synthesis device according to any one of the above.
(Appendix 6)
Instructing combined position information including information on pixel positions for combining the first image signal input from the image input device and the second image signal stored in the memory;
Based on the combined position information, for the second target pixel of the second image signal to be combined with the first target pixel of the first image signal, the color corresponding to the first target pixel, and the second When the color corresponding to the target pixel is different from the peripheral pixels of the second target pixel, the second target pixel is determined based on the pixel value of the peripheral pixel that is the same color as the color corresponding to the first target pixel. Set the pixel value,
Based on the combined position information, a pixel value of the first target pixel of the first image signal is combined with a pixel value of the set second target pixel to generate a third image signal.
An image composition method characterized by the above.
(Appendix 7)
The image according to any one of appendix 1 to appendix 5, wherein the second image signal is provided with pixels set with transparent pixel values around pixels set with colored pixel values. Synthesizer.
(Appendix 8)
A control unit for instructing combined position information including information on a pixel position for combining the first image signal input from the image input device and the second image signal stored in the memory;
Based on the combined position information, a color corresponding to the first target pixel between the first target pixel of the first image signal and the second target pixel of the second image signal, and the second image signal When the color corresponding to the second target pixel is different from the peripheral pixel of the second target pixel, the second target pixel is based on the pixel value of the peripheral pixel having the same color as the color corresponding to the first target pixel. A conversion processing unit for setting the pixel value of the target pixel;
An image combining unit that combines the pixel value of the second target pixel set by the conversion processing unit with the pixel value of the first target pixel based on the combined position information to generate a third image signal;
An image synthesizing circuit comprising:

11 Imaging unit (image input device)
12 Image processing device (image composition device)
13 Memory 22 Image composition unit 23 Image processing unit 41 Interpolation processing unit 26 CPU (control unit)
51 First Bayer signal (first image signal)
52 Second Bayer signal (second image signal)
62 First attention pixel 63 Second attention pixel 71 Conversion processing section 72 Register section 73 Control processing section 74 Buffer section 75 Pixel value conversion section CI Composite position information R1 to R3 registers

Claims (6)

A control unit for instructing combined position information including information on a pixel position for combining the first image signal input from the image input device and the second image signal stored in the memory;
Based on the combined position information, a color corresponding to the first target pixel between the first target pixel of the first image signal and the second target pixel of the second image signal, and the second image signal When the color corresponding to the second target pixel is different from the peripheral pixel of the second target pixel, the second target pixel is based on the pixel value of the peripheral pixel having the same color as the color corresponding to the first target pixel. A conversion processing unit for setting the pixel value of the target pixel;
An image combining unit that generates a third image signal by combining the pixel value of the second target pixel set by the conversion processing unit with the pixel value of the first target pixel based on the combined position information;
An image composition device comprising: The conversion processing unit sets a pixel adjacent to the second pixel of interest as the peripheral pixel, calculates an average value of pixel values of the peripheral pixels, and sets the pixel value of the second pixel of interest;
The image synthesizing device according to claim 1. The conversion processing unit
A register unit whose value is set by the control unit;
A control processing unit that performs control based on a value set in the register unit;
A buffer unit for inputting a partial image of the second image signal including the second target pixel from the memory based on the control of the control processing unit;
Based on the value set in the register unit input from the control processing unit, the partial image is input from the buffer unit, and the pixel value of the second target pixel is calculated and set from the peripheral pixels of the partial image. A pixel value converter to
The image synthesizing device according to claim 1 or 2, further comprising: In the register unit, a value including the pixel type of the first and second image signals and the combined position information is set by the control unit,
The pixel value conversion unit includes a plurality of arithmetic circuits, and the first and second image signals are based on values including pixel types of the first and second image signals set in the register unit and the combined position information. The image synthesizing apparatus according to claim 1, wherein a mode in which the second image signal is superimposed is detected, and an arithmetic circuit corresponding to the detection result is selected.   5. A reverse gamma conversion unit that inputs the second image signal from the memory, performs reverse gamma correction on the second image signal, and outputs the resultant signal to the conversion processing unit. The image composition device according to any one of the above. Instructing combined position information including information on pixel positions for combining the first image signal input from the image input device and the second image signal stored in the memory;
Based on the combined position information, for the second target pixel of the second image signal to be combined with the first target pixel of the first image signal, the color corresponding to the first target pixel, and the second When the color corresponding to the target pixel is different from the peripheral pixels of the second target pixel, the second target pixel is determined based on the pixel value of the peripheral pixel that is the same color as the color corresponding to the first target pixel. Set the pixel value,
Based on the combined position information, a third image signal is generated by combining the set second target pixel with the first target pixel of the first image signal.
An image composition method characterized by the above.

Images