JP7242235B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing apparatus and image processing method, and is particularly suitable for use in transmitting values indicating attributes of pixels.

For example, in processing frame data, there are cases where it is desired to inherit pixel attributes (hereinafter referred to as pixel attributes). For example, pixel attributes related to the state of pixels in RAW data captured by a digital camera include defective pixels (scratches), saturated pixels (poor sensitivity), functional pixels (parallax acquisition pixels for image plane phase difference AF), and the like. mentioned.
Pixel data to which a pixel attribute is to be added may not be valid data. When recording an image, the pixel values corresponding to the defective pixels, saturated pixels, and functional pixels need to be replaced with pixel values interpolated using the pixel values of the reference pixels surrounding the pixels in question.
The transmitted pixel attributes are used to know the state of the pixel of interest during image processing. For example, in the process of correcting a defective pixel, if the pixel of interest is a known flaw in the process and the pixel value of the pixel of interest is interpolated from surrounding reference pixels, the pixel value of the pixel of interest is set to the pixel attribute of the defective pixel in advance in the data path. replace it. Then, the pixel attribute of the pixel of interest is detected by decoding the pixel attribute in the blemish correction processing unit (see Patent Document 1).

A method of directly binding pixel attributes to data for each pixel has also been proposed (see Patent Document 2). For example, a 1-bit flag indicating that the pixel is damaged is added to the lower bits of the data. In this case, the flag bits are separated and held in the intermediate arithmetic processing, and the separated flag bits are merged after the arithmetic processing.

Japanese Unexamined Patent Application Publication No. 2010-21858 JP 2012-124795 A JP-A-2001-83407 JP 2017-107204 A JP 2017-156770 A

However, in the method of Patent Document 1, when another process is performed between the replacement with the pixel attribute and the detection of the pixel attribute, the pixel attribute needs to be saved/held in the intermediate process (arithmetic circuit). Become. In particular, in the method of replacing a pixel value with an arbitrary setting value as a pixel attribute, it is necessary to have a mechanism for canceling the same value as the setting value when the same value as the setting value appears in the intermediate calculation result. Therefore, there is a possibility that the circuit scale (cost) will increase. Moreover, there is a possibility that the number of man-hours for mounting the arithmetic circuit and the number of man-hours for verification will increase.
On the other hand, the method described in Patent Document 2 binds pixel attributes to data, so there is a risk that the data band will increase when transmitting pixel attributes.
As described above, the conventional technique has the problem that the value of the pixel attribute (the value indicating the attribute of the pixel) cannot be properly transmitted.
SUMMARY OF THE INVENTION The present invention has been made in view of such problems, and it is an object of the present invention to enable appropriate transmission of values indicating attributes of pixels.

The image processing apparatus of the present invention converts data in which a value indicating a pixel attribute is expressed in a first form to a second form in which the value indicating the pixel attribute is expressed in a second form different from the first form. a first conversion means for converting into data, and converting data in which a value indicating a pixel attribute is expressed in the second form into data in which a value indicating the attribute of the pixel is expressed in the first form. a first transmission means for transmitting the data expressed in the first form to a first transmission line; and the data expressed in the second form in the first transmission and a second transmission means for transmitting to a second transmission path different from the path, wherein the first form assigns a value corresponding to the attribute of the pixel as the pixel value of the pixel in the pixel data The second form is a form in which a value indicating the attribute of the pixel is added to the pixel data, or a form in which a part of the pixel data is replaced with a value corresponding to the attribute of the pixel. characterized by being

According to the present invention, values indicating attributes of pixels can be appropriately transmitted.

It is a figure which shows the structure of an image processing apparatus. FIG. 2 is a diagram showing the configuration of a reconfigurable circuit; FIG. FIG. 10 is a diagram showing a first pixel attribute form; FIG. 10 is a diagram showing a first example of a second pixel attribute form; FIG. 4 is a diagram illustrating a first example of a configuration of a first conversion unit; FIG. FIG. 10 is a diagram illustrating a first example of a configuration of a second conversion unit; FIG. 4 is a diagram showing the internal configuration of an ALU; FIG. FIG. 4 illustrates a pulse generation circuit; 4 is a flowchart showing processing for creating a distance map; 4 is a flow chart showing processing when obtaining a Y value (luminance value). FIG. 10 is a flowchart showing processing when performing correlation calculation between parallax images; FIG. FIG. 10 is a diagram showing a second example of a second pixel attribute form; FIG. 10 is a diagram showing a second example of the configuration of the first conversion unit; FIG. 10 is a diagram illustrating a second example of the configuration of the second conversion unit;

In each of the following embodiments, a case where processing is completed through a reconfigurable circuit in a hardwired data path will be exemplified. A hardwired (fixed circuit) is a logical configuration (instruction) realized by hardware. A reconfigurable circuit realizes a logic configuration (instructions) through circuit reconfiguration (reconfigurable) by software.
In a reconfigurable circuit, many ALUs (arithmetic operation circuits) are laid out in submodules within the circuit. The reconfigurable circuit, in cooperation with the CPU, reads a large amount of frame data temporarily held in a volatile large-capacity storage unit (DRAM, etc.) at high speed, and performs desired arithmetic processing in combination with the ALU. . The reconfigurable circuit writes the frame data back to the storage unit after the arithmetic processing is executed. After reconfiguring the combination of ALUs by software, the reconfigurable circuit again performs arithmetic processing on the frame data in the storage unit using the reconfigured combination of ALUs.

There are access requests to the storage unit from various processing units in the system, including the reconfigurable circuit. Therefore, they must be processed within the tolerance band. Therefore, the system cannot be established unless an increase in the data bandwidth in each processing unit is avoided.
Therefore, it is conceivable to replace pixel data with pixel attributes. However, a reconfigurable circuit is a collection of ALUs, and in a form in which pixel data is replaced with pixel attributes, a mechanism for saving and holding the pixel attributes is required for each ALU. Specifically, a mechanism for inheriting the pixel attribute input to the ALU to the subsequent ALU and a mechanism for preventing the ALU operation result from simulating the pixel attribute state are required. Therefore, the circuit scale (cost) of the entire reconfigurable circuit may increase. In addition, there is a possibility that the man-hours required for mounting each ALU and the man-hours required for verification may increase.

On the other hand, in the method of binding pixel attributes suitable for ALU to pixel data, for example, when flag bits are bound in addition to pixel data, the amount of information stored in the storage unit increases, and the data transfer band is compressed. . When the flag bit is replaced with the LSB (Least Significant Bit) of the original pixel data, the original transfer band and numerical value of the data do not change, but the resolution of the original pixel value is discarded by the flag bit in the entire data path. It will be.

Therefore, in the following embodiments, pixel attribute values are expressed in two mutually different forms (first pixel attribute form and second pixel attribute form). A first pixel attribute form is a form in which any other pixel value is assigned as a pixel attribute value to a pixel value of pixel data. The second pixel attribute format is a format in which the pixel attribute value is replaced with a partial bit value of the pixel value (for example, an LSB bit value), or a format in which the pixel attribute value is bound (added) to data as a flag. is. Then, pixel attributes are transmitted between the reconfigurable circuit and the outside in the first pixel attribute format (not in the second attribute format). Also, the transmission of pixel attributes within the reconfigurable circuit is performed in a second pixel attribute format (instead of in the first attribute format). Therefore, in the reconfigurable circuit, conversion from the first pixel attribute form to the second pixel attribute form and conversion from the second pixel attribute form to the first pixel attribute form are performed.

Embodiments will now be described in detail with reference to the accompanying drawings.
(First embodiment)
First, the first embodiment will be described.
FIG. 1 is a diagram showing an example of the configuration of an image processing apparatus. In the present embodiment, an example in which an image processing device is installed in an imaging device will be described. The imaging device is, for example, a digital still camera, a digital movie camera, an industrial camera, an in-vehicle camera, a medical camera, or the like. However, if the device has a function of performing image processing on the image captured by the imaging unit, the device equipped with the image processing device can be a mobile terminal (mobile phone, tablet terminal, etc.) or an information processing device such as a personal computer. can be Also, the image processing device is not limited to one mounted on the imaging device. For example, the image processing apparatus does not have to have an imaging section (the imaging unit 105 in the example shown in FIG. 1). In this case, the image processing device can communicate with the device having the imaging unit.

In FIG. 1, the imaging apparatus has a CPU 100, a ROM 101, a RAM 102, a recording medium 103, a reconfigurable circuit 104, an imaging unit 105, and an image processing circuit . The image processing apparatus is configured using, for example, a CPU 100, a ROM 101, a RAM 102, a recording medium 103, a reconfigurable circuit 104, and an image processing circuit .

ROM 101 is an example of a non-volatile storage medium. The RAM 102 is an example of a volatile storage medium and temporarily stores data. The RAM 102 is, for example, a large-capacity DRAM or the like. Both the ROM 101 and the RAM 102 are primary storage units of the CPU 100 (storage media directly accessed by the CPU 100). Setting information 1011 of the reconfigurable circuit 104 is stored in the ROM 101 . In addition to providing a work area to the CPU 100, the RAM 102 temporarily stores setting information 1011 of the reconfigurable circuit 104 and frame data to be subjected to image processing.

The recording medium 103 is an example of an auxiliary storage device. The recording medium 103 is, for example, a CF (compact flash (registered trademark)), an SD card, or the like. In this embodiment, the frame data 1031 stored in the recording medium 103 is expanded once in the RAM 102 and passed through the data path. If the access speed (write, read) of the recording medium 103 is high, there is no need to pass through the RAM 102 . However, when the recording medium 103 is a recording medium using memory technology such as NAND flash, rewriting (the action of erasing and writing) itself shortens the data retention life of the device. Therefore, developing the frame data 1031 once in the RAM 102 avoids shortening the retention life of the data. The ROM 101, RAM 102, and recording medium 103 each have a control unit for writing and reading data.

Storage media have been developed that are non-volatile and also suitable for primary storage. For example, there is a memory that uses a "phase change memory" as a storage element technology and an "ovonic switch" as a cell selection technology. Such a memory is a memory that improves the drawbacks of the NAND flash and realizes an improvement in the number of rewrites (longer life) and an improvement in random access (shorter latency). Such a memory may be used as at least one of the ROM 101 , RAM 102 and recording medium 103 . In this embodiment, a case where an image processing apparatus is configured using the ROM 101, RAM 102, and recording medium 103 described above will be exemplified.

The imaging unit 105 has an optical system (lens, etc.), an imaging element (image sensor), an A/D conversion circuit, and the like. The imaging unit 105 captures an image and generates an image signal (digital signal) of the captured image. Incidentally, the optical system may be detachable from the imaging device.
The image processing circuit 106 performs image processing on the image signal. The image processing circuit 106 may include, for example, a signal processing circuit for sensor correction such as a shading correction circuit and a linearity correction circuit as a hardwired logic circuit configuration (submodule). The image processing circuit 106 may be connected in cascade before and after the reconfigurable circuit 104 .
A data bus 107 connects the CPU 100, ROM 101, RAM 102, recording medium 103, reconfigurable circuit 104, imaging unit 105, and image processing circuit 106 so that they can communicate with each other. For example, by using the data bus 107, a first transmission (transmission of data in the form of a first attribute) is performed on the first transmission line.

The digital circuit of the image processing apparatus of this embodiment is a synchronous circuit by a system clock (not shown), and in-phase transfer is established. Also, in this embodiment, the inter-module I/F inputs/outputs a status signal (valid status signal (not shown)) indicating that the data is valid in addition to the data signal. Depending on the system, the inter-module I/F may send a stop request signal for requesting stop of data transfer to the preceding stage circuit.

FIG. 2 is a diagram showing an example of the configuration of the reconfigurable circuit 104. As shown in FIG.
The reconfigurable circuit 104 has a first conversion section 1041 at its input section and a second conversion section 1042 at its output section. The ALU group 1043 is an ALU array in which a plurality of ALUs 10431 are selectively connectable. A selector (to be described later) connects between the first conversion unit 1041 and an arbitrary ALU 10431, between the second conversion unit 1042 and an arbitrary ALU 10431, and between arbitrary two points in the plurality of ALUs 10431. form a transmission path. For example, by using this transmission line, second transmission (transmission of data in the second attribute format) is performed on the second transmission line.

The first conversion unit 1041 will be described later with reference to FIG. The second conversion unit 1042 will be described later with reference to FIG. Also, the internal configuration of the ALU 10431 will be described later with reference to FIG.
The SRAM 1044 is an example of a memory (volatile storage medium) for temporarily storing the calculation result of any ALU 10431 . The control unit 1040 configures the circuit.

For example, the control unit 1040 rewrites register information existing in the first conversion unit 1041, the second conversion unit 1042, and the ALU group 1043, thereby changing/updating the entire arithmetic processing (processing function). In addition, the control unit 1040 sets the timing of changing/updating the register information in advance so that the register information can be changed/updated without the intervention of firmware. have a circuit. The setting of the timing of changing/updating the register information is realized by, for example, describing it as an instruction (instruction code) in the setting information 1011 .

A circuit update request can be realized by generating a pulse signal in the ALU 10431 and transmitting the pulse signal to the control unit 1040 .
The control unit 1040 decodes the instruction code described in the setting information 1011 and executes the instruction. As an instruction, in addition to a Write/Read instruction to a register for the ALU 10431, its own Wait instruction (waiting for completion notification, etc.) and Loop instruction (executing the same circuit processing multiple times, etc.) may be defined. These actions are well-known techniques similar to the instruction processing of the CPU, and detailed description thereof will be omitted. Control unit 1040 may issue an update status signal to inform ALU 10431 of register updates.

FIG. 3 is a diagram showing an example of the format of the first pixel attribute form. In this embodiment, the data width of the signal transmitted through the data bus 107 is defined as 16 bits (16 bits is exemplified). FIG. 3A shows that 16-bit width image data or pixel attribute values appear on the data bus 107 as pixel attribute information. For example, when the pixel value of a certain pixel is saturated and does not indicate a correct value, the pixel is considered to be a pixel with poor sensitivity, and a value indicating "saturated pixel" is assigned as the pixel attribute information of the pixel. Thus, pixel attribute information appears. The first form of pixel attribute is to replace the pixel data with the value of the pixel attribute. This embodiment exemplifies a case in which a value of 0 is assigned as a value of a pixel attribute indicating a "saturated pixel".

FIG. 3(b) illustrates the dynamic range of data values. A range of 1 to 65535 in the 16-bit data value is treated as an unsigned image data value, and a value of 0 is defined as a saturated pixel flag. That is, one value of the dynamic range is discarded from the whole (0 to 65535 values), and that portion is assigned to the pixel attribute information. Therefore, in the case of the first pixel attribute form, when the pixel value becomes 0, the pixel value is replaced with 1 value. Even if pixel values are replaced in this manner, development processing is performed later. Therefore, discarding a dynamic range of about one value does not pose a problem visually.

FIG. 4 is a diagram showing an example of the format of the second pixel attribute form. In this embodiment, the data width handled by the ALU group 1043 is defined as signed 17 bits (sign bit and pixel data 16 bits). The signed data takes into account offset clamp subtraction in the first conversion unit 1041, which will be described later.
FIG. 4A shows that the pixel data is signed 17-bit data (the most significant bit is the sign bit) and that the value of the pixel attribute bound to the pixel data is 1 bit. FIG. 4B shows that the dynamic range of pixel values is in the range of 65535 to -65536, and that the saturated pixel information attached to the pixel data has a value of 0 for a normal pixel and a value of 1 for the saturated pixel information. indicates that it is a saturated pixel. In FIG. 4(b), the "&" symbol between the data value and the saturated pixel information exemplifies that they are concatenated (in Verilog RTL description, etc.), and one signal on the logic circuit It exemplifies that it is treated as

FIG. 5 is a diagram showing an example of the configuration of the first conversion unit 1041. As shown in FIG.
A signal sig_500 is an input signal to the reconfigurable circuit 104 and corresponds to data to be processed (see FIG. 2). The input signal sig_500 takes the form of the first pixel attribute defined as shown in FIG. 3, and is transmitted as 16-bit data together with a valid status signal (not shown) from the preceding circuit.

Signal sig_504 is a reconfigurable register value, and is transferred from control unit 1040 shown in FIG. 2 together with an update status signal (not shown). The register value of the FF circuit 10411 is updated according to the update status instruction.
An FF circuit (flip-flop circuit) 10410 updates the value of the signal sig_501 to the value of the signal sig_500 at that time, depending on the state of the valid status signal (for example, at the rising edge of the clock in the High state). The FF circuit 10410 is a signal (implementation technology for LSI implementation) for improving the timing of the signal sig_500 and for mitigating wiring congestion. Similar to the FF circuit 10410, FIGS. 5 and 6 show FF circuits for improving timing and mitigating wiring congestion. These are necessary components for the configuration of a reconfigurable circuit that uses many selective switching units inside. However, the mounting (position, presence/absence, number, etc.) of the FF circuits is not limited here.

A FF circuit 10411 is implemented to hold the offset clamp value in the datapath. The bit width of the signal sig_505 may be the same as the bit width of the signal sig_500. For example, the bit width of the signal sig_505 is equivalent to 16 bits, and depending on the set black level for the image, the number of bits may be less than that. Image data from the image sensor does not have the original black level because offset components such as dark current are superimposed thereon. The FF circuit 10411 clamps the target value (the register setting value of the register 10411) as preprocessing so that the gain is not applied to this offset component when multiplication is performed during signal processing by the ALU.

This function can be canceled by setting the register value to 0 if already clamped data is being input. In this embodiment, the output of the adder (subtractor) 10415 is 17-bit signed data, and the MSB (Most-Significant Bit) of the signal sig_506 indicating the calculation result is treated as a sign bit.

Although the above-described clamp processing itself is not directly related to conversion processing in the form of pixel attributes, the implementation position as hardware is just this position. That is, the value of the pixel attribute of the data before clamping is detected, and a flag indicating the value of the pixel attribute is attached to the data after clamping. Therefore, the clamp processing is also included in the example. It is also assumed that the register value indicated by the signal sig_505 is set so that the addition (subtraction) result does not clip with the bit width of the signal sig_506. Alternatively, a positive/negative clip circuit (limiter) may be inserted at this position.

A signal sig_502 is a signal indicating a comparison value (reference value) in the comparator 10414 . The value of signal sig_502 is set to 0 in order to detect that signal sig_501 is 0 (value indicating "saturated pixel" shown in FIG. 3(a)). In FIG. 5, this value is degenerated as a fixed value, but if necessary, it may be set as a register value from the control unit 1040 like the signal sig_505.

A signal sig_503 is a flag signal indicating the saturation state of an arbitrary pixel. The signal sig_503 is for transmitting a value of 1 as a flag value indicating a saturated pixel when the first conversion unit 1041 receives a value of 0 as the first pixel attribute form. A signal sig_507 is a signal obtained by binding a clamped signal sig_506 and a signal sig_503 indicating a flag value. The data width of this signal is 18 bits in total as described with reference to FIG. The signal sig_507 is temporarily held in the FF circuit 10413 and output to the subsequent stage (ALU group 1043) as the signal sig_508. A signal sig_508 is a signal scattered within the ALU group 1043 (a signal with a large fan-out). The FF circuit 10413 is for improving the timing and mitigating wiring congestion.

FIG. 6 is a diagram showing an example of the configuration of the second conversion unit 1042. As shown in FIG.
A signal sig_601 is a signal output from any ALU 10431 . A signal sig_601 is the output of the ALU 10431 selected by the selector 10432 in the ALU group 1043 (as described above, 17-bit signed data and 1-bit saturation flag). All or some of the ALUs 10431 in the ALU group 1043 are connected to the selector 10432 .
The FF circuit 10421 is for improving the timing of the signal sig_601 and reducing wiring congestion. The output from the FF circuit 10421 is separated into an upper 17-bit signal sig_602 representing pixel data and a least significant bit signal sig_606 representing a saturated pixel flag. Signal sig — 602 is output to adder 10422 and signal sig — 606 is output to selector 10424 .

The FF circuit 10420 is a register that holds a signal sig_611 indicating a post-setup value for reproducing the set black level during signal processing for the signal sig_602. The register value of the FF circuit 104020 is supplied from the control unit 1040 (as signal sig_603) and updated as necessary during circuit configuration.
A signal sig_604 is a signal indicating the result of adding a signal sig_611 indicating a post-setup value for reproducing a set black level to a signal sig_602 indicating pixel data. The limiter circuit 10427 clips the value of the signal sig_604 to 0 when the signal sig_604 is still negative even after adding the post-setup value. Also, the limiter circuit 10427 clips the maximum value of the signal after the post-setup value is added to the 65535 value. The signal sig_612 output from the limiter circuit 10427 is assumed to be unsigned (positive) 16-bit data.

Comparator 10426 compares signal sig_612 and signal sig_610 output from limiter circuit 10427 . In this embodiment, in the first pixel attribute form, when the value of a signal is 0, the pixel indicated by that signal is a saturated pixel. Therefore, the value of the signal sig_610 is set to a fixed value of 0 here. Comparator 10426 outputs 1 to sig_605 when signal sig_612 output from limiter circuit 10427 is 0, and outputs 0 otherwise. In terms of circuitry, the signal sig_610 may be degenerated to a value of 0, but the value of the signal sig_610 may be used as the register set value from the control unit 1040 .

The selector 10423 outputs the value of the signal sig_612 as the signal sig_607 if the signal sig_605 output from the comparator 10426 is 0, and outputs the value of the signal sig_613 as the signal sig_607 if it is 1. In this embodiment, the value of the signal sig_613 is a fixed 1 value. As mentioned above, this is a workaround for assigning a value of 0 as a value indicating a saturated pixel, so that a pixel with a value of 0 as a result of the calculation by the ALU 10431 is replaced with a value of 1 to prevent a pseudo pixel attribute from being generated. is. When changing the value to be assigned to the signal sig_607 when the signal sig_605 output from the comparator 10426 is 1, the value of the signal sig_613 may be used as the register set value from the control unit 1040 .

The selector 10424 outputs the value of the signal sig_607 as the signal sig_608 if the value of the signal sig_606 is 0, and outputs the value of sig_610 as the signal sig_608 if it is 1. A signal sig_606 is a 1-bit status flag obtained by separating the value of the pixel attribute bound to the pixel data shown in FIG. 4 from the pixel data. Therefore, data with saturated pixel status is now replaced as the value of the pixel attribute. The FF circuit 10425 is for improving the timing and mitigating wiring congestion (to prevent the bus to the external module from becoming a critical path). A signal sig_609 is output from the FF circuit 10425 .

FIG. 7 is a diagram showing an example of the internal configuration of the ALU 10431. As shown in FIG.
The ALU group 1043 usually includes several tens to several hundreds of ALUs and is implemented with a considerable circuit scale, so power saving measures are required. The clock supplied to the ALU group 1043 may use a gated clock configuration so that it is turned on only when the valid status signal (not shown) described with reference to FIG. It is necessary to supply a clock considering the internal latency).

Signals sig_710 to 716 and sig_719 are circuit parameters distributed from the control unit 1040, and the values of these signals are held in FF circuits 713 to 719 and 721, respectively. Since the drawing becomes complicated, the update status of these FF circuits is not shown in the same manner as in FIGS. Also, the circuit update request to the control unit 1040 may be made by using several ALUs and by using a pulse generation circuit, which will be described later with reference to FIG.

In this embodiment, the ALU 10431 is assumed to be a two-input one-output arithmetic operation circuit. It is possible to independently adjust the delay amount of each of the two inputs of the ALU 10431, and the two inputs are set so that the phases are aligned immediately before the internal calculator. A signal sig_701 is a first input signal and is one of the output from the ALU in the previous stage and the output from the first conversion unit 1041 .
The selector 700 selects a signal from the input signal group including the signal sig_701 according to the set value of the FF circuit 716 and outputs it as the signal sig_702. The multi-stage delay element 701 is implemented by a shift register configuration in which FF circuits each having a data bit width are cascaded. The amount of delay is adjusted by setting in the FF circuit 715 at which stage the output of the FF circuit at this time is to be output to the subsequent stage.

ALU 10431 in this embodiment takes the form of a second pixel attribute. Therefore, the LSB (bit0) of signal sig_703 is a flag indicating the pixel attribute. The flag bit is merged with the signal sig_707 after the operation in the same phase state and input to the FF circuit 709 as the signal sig_708 without inputting it to the computing unit for transmission to the subsequent stage.
The LSB (bit 0) of signal sig_703, which is a flag indicating a pixel attribute, passes through AND element 710 and OR element 712 on the way. This involves masking the transmission of the signal indicating the pixel attribute by the setting value of the FF circuit 713 and merging (or replacing) the state of the pixel attribute of the other input signal (signal sig_704) of the ALU 10431 as necessary. This is to enable A signal sig_717 reflecting these states is merged with the signal sig_707 and input to the FF circuit 709 as the signal sig_708.

The signal sig_704 side, which is the other of the two input signals of the ALU 10431, is the same as the signal sig_701 side. As a different part from the signal sig_701 side, the selector 722 is set so that it can select the value of the accompanying pixel attribute instead of passing the signal value of the signal sig_706 to the calculator alone. The signal sig_718 may have the value of the pixel attribute given to the least significant bit, and the other bits may be degenerated with a value of 0, or the value of the pixel attribute may be expanded to all bits. If necessary, both of them may be used as options, and if unnecessary, the selector 722 itself may not be provided. The selector 722 can be switched by the value of the FF circuit 721 set by the signal sig_719 from the control unit 1040 .

The reason why the FF circuits do not receive the signals sig_701 and sig_704 is that the ALU 10431 and the first conversion unit 1041 output the signals sig_701 and sig_704 from the FF circuits. However, the FF circuits that receive the signals sig_701 and sig_704 may be placed in the ALU 10431 depending on the actual placement and routing of the semiconductor.

In this embodiment, there is no limitation on how the calculators 704 to 707 and 720 are actually mounted. The illustration in the block diagram shown in FIG. 7 is also an image. Circle marks in these calculators indicate adder-subtractors, multipliers, etc., triangles indicate comparators, etc., and logic elements indicate AND, OR, XOR circuits, and the like. However, these may be changed, added, or deleted as required at the time of implementation. Also, a mechanism for bit shifting may be provided after the calculator. In addition, a rounding mechanism (such as rounding off) may be provided for normalization by bit shift and reduction of calculation accuracy. The details of implementation of these ALU operations are not particularly limited.
A selector 708 selects the operation result of the arithmetic units 704 to 707 and 720 according to the set value of the FF circuit 719 and outputs it as a signal sig_707.
As described above, the signal sig_707 is temporarily held in the FF circuit 709 as the signal sig_708 merged with the signal sig_717 indicating the pixel attribute. The output from the FF circuit 709 is input to another ALU as a signal sig_709 and is also input to the second conversion unit 1042 . The output to the second conversion unit 1042 may be after selection within the ALU group 1043 as illustrated by the selector 10432 shown in FIG.
Also, the output from the FF circuit 709 is output to the control unit 1040 so that it can be used as a circuit update request signal. The output to the control unit 1040 may be after going through the selector 802 described later with reference to FIG. 8 in the ALU group 1043 .

FIG. 8 is a diagram showing an example of a pulse generation circuit configured by the ALU10431.
In FIG. 8, ALUs 10431a and 10431b exemplify only main functions of the functions of the ALU 10431 that can be obtained by giving parameter instructions to the ALU 10431 via registers. Small squares drawn inside the ALUs 10431a and 10431b indicate FF circuits. In particular, the squares on the input side indicate delay elements 701 (or 703).

The ALU 10431a constitutes a counter (integrating circuit). The ALU 10431a uses only one of the two inputs, and the other input in the arithmetic unit is the output of the FF circuit in the output stage of the ALU itself (the case where the ALU 10431a is the arithmetic unit 720 shown in FIG. 7 is exemplified). ). When the ALU group 1043 performs gated clock operation, a clock is generated for one cycle each time valid data is processed (not shown). Therefore, in such a configuration, by fixing the signal sig_801 to 1, it is possible to implement a counter that increments by 1 each time valid data is processed.

ALU 10431b constitutes a comparator. The ALU 10431b determines "True" when the count value of the ALU 10431a is equivalent to the signal sig_803, otherwise determines "False", and outputs a value of 1 only when it determines "True". When the signal sig_801 is continuously processed as valid data, the signal sig_804 is input to the selector 802 as a pulse, and when the selector 802 selects the signal sig_804, it is output to the control unit 1040 as the signal sig_805. be. In the example shown in FIG. 8, when the ALU 10431b (comparator) determines "True", the output is 1, and when the ALU 10431b determines "False", the output can be transferred to the subsequent stage as a 0-value flag status. If the signal sig_805 is only used as a control signal to the control unit 1040, the selector 802 may handle only the LSB (bit0).

The selector 700 and/or 702 of the ALU 10431 in this embodiment can use the register value of the multi-purpose register (not shown) of the control unit 1040 when selecting the signal to be input. The values of signals sig_801 and sig_803 can be multi-purpose register values set by control unit 1040 .
The upper limit of the integrated value in the ALU 10431a (counter) depends on the internal processing bit width. When processing the inside with signed 17 bits, the positive 65535 value is the upper limit of the integrated value. If it is necessary to integrate more than 65535 values, a plurality of counter configurations with ALUs 10431a and 10431b as one set should be prepared and connected in cascade, and pulses corresponding to the signal sig_804 should be counted by the subsequent counter.

In the gated clock configuration, the logic is preserved in the absence of the clock. Therefore, if the configuration in FIG. 8 remains as it is, the signal sig_804 (sig_805) may be kept at the value indicating "True". Therefore, if the circuit that receives the signal sig_804 (sig_805) is a circuit that receives a clock at any time, the AND (logical product) of the signal sig_804 (sig_805) and the cause of stopping the clock is used as a pulse for one clock cycle. There is a need. A circuit that receives the signal sig_804 (sig_805) is the control unit 1040 or the like. Further, since these configurations can be realized by known techniques, detailed description thereof will be omitted.

In this embodiment, the case where the ALU 10431 is configured as shown in FIG. 7 is exemplified. However, the components of ALU 10431 are diverse. ALUs with different specifications may exist in the ALU group 1043 of the reconfigurable circuit 104 . For example, the ALU may be a high-precision multiplier, one that outputs the comparison result of a comparator as a flag, or one that has a function of dividing over multiple cycles. There are no restrictions on types.

Also, in this embodiment, for the sake of simplicity of explanation, the case where input selection, delay adjustment, and calculation are implemented within one ALU 10431 has been exemplified. However, each may be arranged in the ALU group 1043 as a separate element, and the functional configuration of one ALU is not limited. Similarly, selectors such as the selector 10432 shown in FIG. 6 and the selector 802 shown in FIG. Although not shown, a selector that selectively connects an arbitrary ALU and the SRAM 1044 may be implemented as an I/F to the SRAM 1044 shown in FIG. Transmission of the pixel attribute via the SRAM 1044 is possible by storing and reading out the flag bit in the SRAM 1044 as a storage object. The address generation for accessing the SRAM 1044 may be performed by the ALU, or may be performed by providing a separate control unit. A write/read status signal to the SRAM 1044 may be generated by the ALU. The I/F between the ALU group 1043 and the SRAM 1044 includes data lines, address signals, status signals, and the like.

<Processing Example of Reconfigurable Circuit 104>
In the present embodiment, transmission of pixel attributes of saturated pixels when part of the process of forming a distance map using a defocus amount by a pupil division method using divided pixels of an image sensor is performed by the reconfigurable circuit 104. is exemplified. An image sensor having divided pixels is described in Patent Document 3, for example. Acquisition of a distance map corresponding to a parallax image is described in Patent Document 4, for example. For example, Japanese Patent Laid-Open No. 2002-200002 describes how to reduce the influence of saturated pixels on correlation calculation.

FIG. 9 is a flowchart showing an example of processing for detecting a defocus amount from a captured image (parallax image) and creating a distance map. As described in the publications mentioned above, the acquisition of parallax pixels from the imaging plane and the generation of the distance map itself are known techniques. Here, only the part of the processing that realizes the transmission of pixel attributes when the reconfigurable circuit 104 executes (a part of) the arithmetic processing for obtaining the distance map will be explained, and the details of the distance map generation itself will be explained. do not provide detailed explanations. Although the parallax image depends on the shape of the divided pixels, in this embodiment, two divided pixels such as vertical and horizontal eyes are exemplified, and the respective images are A image and B image.

In FIG. 9, the reconfigurable circuit 104 performs signal processing to acquire luminance values (hereinafter referred to as Y values) from parallax images (A image and B image, respectively) (steps S901 and S905). Next, the reconfigurable circuit 104 performs correlation calculation between parallaxes (step S902). Next, the reconfigurable circuit 104 estimates the defocus amount from the relative image shift amount between the parallaxes obtained by the correlation calculation in step S903 (step S903), and determines the in-plane defocus amount unevenness due to lens aberration. is corrected (step S904).

In this embodiment, the configuration of the ALU 10431 in the reconfigurable circuit 104 is switched in steps S901, S902, S903, and S904 of FIG. Therefore, in steps S901 (and S905), S902, S903, and S904, circuit information 901, 902, 903, and 904 relating to the configuration of ALU 10431 are used, respectively. Circuit information including circuit information 901 to 904 and control instructions (instruction code) to the control unit 1040 for completing the entire image processing are stored in the ROM 101 in a state compiled as one setting information 1011. Stored. When the device operates, it is necessary to transfer necessary information to the reconfigurable circuit 104 at high speed. Accordingly, the CPU 100 develops the setting information 1011 in the RAM 102 when the apparatus is started up (or at any preparatory stage). The circuit information 901 to 904 are appropriately read from the RAM 102 when a circuit update request including a request for initialization of the reconfigurable circuit 104 is made. It should be noted that the event itself for starting reading of the circuit information 901 to 904 may be executed by the CPU 100 instructing the control unit 1040 .

Based on the circuit information 901, the control unit 1040 in the reconfigurable circuit 104 distributes the register values as illustrated with reference to FIGS. The registers related to the circuit configuration of the reconfigurable circuit 104 are composed of a primary register (not shown) and a secondary register in the control unit 1040, so that the next circuit development can be performed while the current circuit is operating. Is possible. Incidentally, the secondary registers are the respective registers exemplified as the FF circuits in FIGS. This makes it possible to hide the processing time required for circuit update from the image processing. Based on the circuit information 901, the control unit 1040 develops the values in the primary registers, and then develops the values developed in the primary registers into the secondary registers mounted in each module. After expanding the primary register value to the secondary register, the control unit 1040 requests the RAM 102 to transfer the circuit information 902 in order to update the value of the primary register to the next circuit information. This series of operation instructions can be written in the setting information 1011 as an instruction code. An instruction code may be used for the sequence of expanding the circuit information 902 used in step S902 to the ALU group 1043 after the processing of steps S901 and S905.

At the completion of each step in FIG. 9, a pulse generation circuit such as that illustrated in FIG. For example, the circuit can be switched depending on the number of received pixels (including completion of frame processing). Processing events for pulse generation may be defined appropriately to meet system requirements. Further, the circuit update request to the control unit 1040 is not limited to the pulse signal (the level of the signal may be received).

In FIG. 9, examples of Y-value acquisition processing (steps S901 (and S905)) and correlation calculation processing (step S902) are illustrated in FIGS. 10 and 11, respectively, as processing for handling pixel attributes of saturated pixels. . Note that since the focus here is on handling pixel attributes of saturated pixels, detailed descriptions of the defocus amount estimation process (step S903) and the lens aberration correction process (step S904) will be omitted.

In the processing routine of FIG. 9, the step unit is a unit of configuration of the processing circuit, and the circuit information is appropriately updated so that the processing of the entire frame proceeds at each step. 10 and 11 each show one step (subroutine) in FIG. 9, exemplifying the processing performed by the circuit configured by the ALU 10431 and the first conversion unit 1041 and the second conversion unit 1042 as one step. are doing. Therefore, in FIGS. 10 and 11, a desired circuit path is applied so that arbitrary pixels (or arbitrary processing units such as blocks) are processed in each step.

FIG. 10 is a flow chart showing an example of the processing (steps S901 and S905) when obtaining the Y value. In this embodiment, in FIG. 1, frame data 1031 (RAW image/parallax image data) to be processed is expanded from the recording medium 103 to the RAM 102 and read out to the reconfigurable circuit 104 as an example. It is assumed that sensor correction (black level reproduction, shading correction, linearity correction, defect correction, etc.) has already been performed on the pixels of each of the A and B images of the parallax image data.

In step S1001, the reconfigurable circuit 104 performs saturation pixel processing. Specifically, the reconfigurable circuit 104 determines whether or not the signal value (pixel value) of each pixel has reached a predetermined saturation level (threshold). When the pixel reaches the saturation level, the reconfigurable circuit 104 determines that the pixel is a saturated pixel, and according to the second pixel attribute form (see FIG. 4B), the flag bit value is set to 1 value.

Next, in step S1002, the reconfigurable circuit 104 performs luminance shading correction (correction for the optical system). At this time, the effect of vignetting at a position where the image height is high and the crosstalk between divided pixels when the incident angle of the light flux to the microlens is large may be included. The image height is used as a parameter, but if horizontal shading correction is practically sufficient, the consumption of the ALU 10431 can be saved. The value of the pixel attribute indicating saturated pixels is preserved even after the process of step S1002.

Next, in step S1003, the reconfigurable circuit 104 generates a Y value by binning processing. For example, in the case of a RAW image acquired from an imaging device with a Bayer array, a lattice-shaped four-pixel unit is defined as the minimum unit of luminance value, and the reconfigurable circuit 104 outputs each color (R/Gr/Gb/B) Y value is obtained by adding the value of . At this time, if there is a pixel with a flag indicating a pixel attribute (the value of the pixel attribute is 1) in any of the pixels to be added, the pixel attribute value of the generated Y value is also 1. and

In this embodiment, a case of generating a Y value from a RAW image that has already been recorded on the recording medium 103 will be exemplified. Therefore, the Y value is not the resolution for AF use, but information to the extent that it can be attached to the image file and recorded for application use during development (use for grasping the position of the main subject by image recognition, etc.). It can be compressed into volume. One Y value may be generated from a plurality of Bayers while also performing image compression within the allowable range of the application.

Next, in step S1004, the reconfigurable circuit 104 converts the pixel attribute form (converts the value of the pixel attribute from the second pixel attribute form to the first pixel attribute form). This conversion is established when the data passes through the second conversion unit 1042 . The data output from the reconfigurable circuit 104 after processing in step S1004 is in the form of the first pixel attribute and is temporarily stored in the RAM 102. FIG.
The processing steps in FIG. 10 are performed for each parallax image (A image and B image in this embodiment) in steps S901 and S905.

FIG. 11 is a flowchart showing an example of processing (step S902) when performing correlation calculation between parallax images. The reconfigurable circuit 104 reads each of the Y value data of the A image and the B image that have undergone steps S901 and S905 from the temporarily stored RAM 102 . For AF applications, correlation calculation is performed for each AF frame. On the other hand, when acquiring the distance map, the image shift amount is derived for each block because the correlation calculation result of the entire image is acquired.

In step S1101, the reconfigurable circuit 104 performs pixel attribute format conversion (conversion of a pixel attribute value from a first pixel attribute format to a second pixel attribute format). The Y value data read out from the RAM 102 in arbitrary block units is in the first pixel attribute format. The reconfigurable circuit 104 converts the value of the pixel attribute into the second pixel attribute form in the first conversion unit 1041 .

Next, in step S1102, the reconfigurable circuit 104 applies a bandpass filter (hereinafter referred to as BPF) for band selection to the Y value data of the A and B images. The frequency band can be changed according to the imaging conditions. When shooting conditions are recorded as shooting information in a RAW image file, it is possible to select a frequency band according to the shooting conditions. For example, under shooting conditions where the S/N ratio is expected to be poor (high ISO shooting, etc.), it is preferable to apply a lower-pass filter.
A bandpass filter is a digital filter. The vertical taps of the band-pass filter are assigned by forming a plurality of line buffers in the SRAM 1044 in the reconfigurable circuit 104 (not shown). The SRAM 1044 may include a memory controller, and the ALU may generate addresses for accessing the SRAM 1044 as described above.

Next, in step S1103, the reconfigurable circuit 104 calculates an image shift amount for each block. A representative method for calculating the amount of image shift is a method based on template matching using SAD (Sum of Absolute Difference). For example, while shifting the B image block by one pixel (Y pixel) with respect to the A image block, the sum of the deviation amounts (difference values) is calculated. For each pixel of the A image in an arbitrary block, the amount of image deviation from the pixels of the B image in the block is obtained. Assuming that a parallax image obtained by an image sensor whose divided pixels are divided into left and right halves is processed, a horizontal correlation curve of an arbitrary block can be obtained.

Next, in step S1104, the reconfigurable circuit 104 performs an operation for sub-pixel estimation. This provides sub-pixel resolution. Sub-pixel estimation can be realized by executing an arithmetic expression derived from an existing fitting method such as equiangular straight line fitting or parabolic fitting on the correlation curve obtained in step S1103 by the ALU. The resolution of the distance map is determined by the size of blocks into which the frame data amount of the Y value input to this correlation calculation is divided. If the image shift amount is obtained by shifting 2000 horizontal pixels by about 20, the output is about 100 pixels. However, in this embodiment, the specifications in this area are not limited.

As an application of the pixel attribute according to this embodiment, there is a correlation calculation in step S1103. Saturated pixel values can distort the correlation results and cause an error in the estimated amount of image shift. Therefore, for example, during SAD, the reconfigurable circuit 104 can replace the pixel value of the target pixel with the pixel values of the pixels before and after the target pixel when the target pixel has a saturated pixel value. In addition, the reconfigurable circuit 104 can count the number of saturated pixels in the processing block and use the counted number of saturated pixels to determine the degree of reliability. The reconfigurable circuit 104 can make the distance map value obtained from a certain block unreliable when the count value of saturated pixels is equal to or greater than a predetermined number in that block.

For example, in an arbitrary ALU 10431, if the selector 722 illustrated in FIG. Numerical values can be accumulated.

If there is no ALU that covers all the steps illustrated in FIG. 11, the circuit information may be divided into steps to increase the number of configuration stages. For example, due to the capacity of the SRAM 1044, it may be necessary to temporarily store the data in the RAM 102 between steps S1102 and S1103 to interpose the configuration of the circuit. In this case, the reconfigurable circuit 104 temporarily converts the pixel attribute value of the saturated pixel into the first pixel attribute format via the second conversion unit 1042 after step S1102, and stores it in the RAM 102 again. Then, the reconfigurable circuit 104 receives from the RAM 102 the pixel attribute values of the saturated pixels converted into the first pixel attribute form, and converts them into the second pixel attribute form via the first conversion unit 1041 . Convert.

As described above, in the present embodiment, in the reconfigurable circuit 104 that passes through a large number of arithmetic circuits, pixel attribute values are bound to data as flags (second pixel attribute form). to transmit. Between the reconfigurable circuit 104 and the outside, pixel attribute values are transmitted in a form (first pixel attribute form) in which 0 is assigned to pixel values as pixel attribute values. Therefore, the reconfigurable circuit 104 has a circuit configuration that allows the first conversion unit 1041 to convert the first pixel attribute form into the second pixel attribute form. Also, the reconfigurable circuit 104 has a circuit configuration that allows the second conversion unit 1042 to convert the converted second pixel attribute form into the first pixel attribute form. Therefore, in a data path including a reconfigurable circuit including many ALUs and a primary storage unit, transmission of pixel attribute values can avoid an increase in bus bandwidth, reduce costs, and reduce design and verification man-hours. it becomes possible to Therefore, pixel attribute values can be transmitted appropriately according to the transmission environment.

The application targets of the first pixel attribute form and the second pixel attribute form are not limited to those described in this embodiment. For example, it is possible to apply the second pixel attribute format to a portion including a large number of arithmetic circuits other than reconfigurable circuits, and apply the first pixel attribute format to a portion that communicates between the arithmetic circuits and the outside. .
Also, pixel attributes are not limited to saturated pixels (poor sensitivity). For example, at least one of a defective pixel (flaw) and a functional pixel (parallax acquisition pixel for image plane phase difference AF) may be used as a pixel attribute instead of or in addition to a saturated pixel (poor sensitivity). A functional pixel is a pixel that outputs a signal that is used for a purpose different from that for forming a captured image for recording or display. When handling a plurality of pixel attributes, in the first pixel attribute form, two different pixel values corresponding to the pixel attributes are assigned as pixel attribute values. In the second pixel attribute form, the values of each pixel attribute are bound and treated as one signal.

(Second embodiment)
Next, a second embodiment will be described. In the first embodiment, as the second pixel attribute form, as illustrated in FIG. 4, a form in which signed 17-bit data as pixel data is bound with 1-bit data as a value indicating the value of the pixel attribute. explained with an example. On the other hand, in the present embodiment, the second pixel attribute mode will be described by taking as an example a mode in which the value of the pixel attribute is replaced with a part of the bit values in the image data.

When the image processing apparatus is equipped with a large-scale SRAM (a line buffer for multiple lines, etc.), configuring the system so that the status is stored in the SRAM while binding the data to the data may lead to an increase in cost. In this case, as the second pixel attribute form, the form in which a part of the pixel value is replaced with the status is more advantageous than the second pixel attribute form described in the first embodiment. However, replacing some of the effective bits of the image data with the status may lead to deterioration in calculation accuracy. Therefore, the second pixel attribute form of the present embodiment is effective, for example, for a path configuration that is mainly used without using pixel attribute values, or for a path configuration with a sufficient data bit width. As a path configuration with a sufficient bit width of data, for example, there is a path configuration in which calculations are performed up to the decimal point in order to obtain calculation accuracy during processing, and the decimal point is rounded at the output stage.

As described above, the present embodiment differs from the first embodiment mainly in the configuration and processing due to the difference in the form of the second pixel attribute. Therefore, in the description of the present embodiment, detailed description of the same parts as those of the first embodiment will be omitted.

FIG. 12 is a diagram showing an example of the format of the second pixel attribute form in this embodiment. FIG. 12(a) shows an example of handling of each bit of the original pixel data. FIG. 12(b) shows an example of a form in which the least significant bit (bit0) is replaced with the value of the pixel attribute as the second pixel attribute form.
FIG. 12A illustrates a case where the pixel data is signed 17-bit data. As can be said in the first embodiment, the bit width of data input/output to/from the reconfigurable circuit 104 does not need to match the bit width of data in the ALU 10431 . In addition, arithmetic processing is performed with high precision bits (for example, 20 bits) in the ALU 10431, and the bit width to be output from the reconfigurable circuit 104 in the second conversion unit 1042 (exit of the reconfigurable circuit 104) You can also round the data to .

FIG. 13 is a diagram showing an example of the configuration of the first conversion section 1300 in this embodiment. The first conversion unit 1300 illustrated in FIG. 13 replaces the first conversion unit 1041 illustrated in FIG. Of the parts illustrated in FIG. 13, detailed description of the same parts as those illustrated in FIG. 5 will be omitted.
In FIG. 13, signal sig_1301 is data in the form shown in FIG. 3(a). In a circuit configuration that does not require transmission of pixel attribute values, the FF circuit 1307 is set to 0 and the LSB of pixel data is passed through the selector 1305 .
The FF circuit 1301 is the same as the FF circuit 10410 in FIG. A signal sig_1302 is an offset clamp value and is held in the FF circuit 1302 . Signal sig_1302, FF circuit 1302, and adder (subtractor) 1303 are the same as signal sig_504, FF circuit 10411, and adder (subtractor) 10415 in FIG. 5, respectively. However, it is assumed that the bit width of the output data in FIG. 13 matches the setting in FIG. Signal sig_1308, comparator 1304, and signal sig_1304 are the same as signal sig_502, comparator 10414, and flag signal sig_503 in FIG. 5, respectively.

In this embodiment, an FF circuit 1307 and a selector 1305 are provided to replace the pixel attribute value with the least significant bit of the image data. As in the first embodiment, a case (similar to FIG. 3) in which a value of 0 is assigned as a value of the first pixel attribute form (a value indicating a saturated pixel) will be exemplified. Assuming that the signal sig_1308 is degenerated to a value of 0, the signal sig_1304 is a value of 1 when the signal sig_1301 is a value of 0. If the handling of the pixel attribute is valid, the control unit 1040 sets the FF circuit 1307 to 1 value. Accordingly, the selector 1305 selects the signal sig_1304 and outputs it as the signal sig_1305. On the other hand, when the pixel attribute is not used, the selector 1305 selects the least significant bit of the signal sig_1303 and outputs it as the signal sig_1305.

The signal sig_1306 is a signal that has the value of each bit of the signal sig_1303 as the value of the upper bits of the signal sig_1303 excluding bit0 and the value of the signal sig_1305 as the value of bit0, and is input to the FF circuit 1306 . The FF circuit 1306 is the same as the FF circuit 10413 illustrated in FIG. The output of the FF circuit 1306 becomes the signal sig_1307 output from the first conversion unit 1300 as it is.
In this embodiment, transmission of pixel attribute values can be turned ON/OFF by setting the FF circuit 1307 . Also in the first embodiment, similar control is possible by implementing a mask function that always sets the output of the comparator 10414 to 0 value.

FIG. 14 is a diagram showing an example of the configuration of the second conversion section 1400 in this embodiment. A second conversion unit 1400 illustrated in FIG. 14 replaces the second conversion unit 1042 illustrated in FIG. Of the parts illustrated in FIG. 14, detailed description of the same parts as those illustrated in FIG. 6 will be omitted.
14, FF circuits 1401, 1403, 1411, adder 1404, and limiter circuit 1405 are the same as FF circuits 10421, 10420, 10425, adder 10422, and limiter circuit 10427, respectively, illustrated in FIG. Comparator 1406 and selectors 1409 and 1410 are the same as comparator 10426 and selectors 10423 and 10424 shown in FIG. 6, respectively. However, it is assumed that the bit width of the input data in FIG. 14 matches the setting in FIG.

Signal sig_1401 is data in one of the forms shown in FIGS. 12(a) and 12(b). Here, the signal sig_1401 is assumed to be signed 17-bit data. The second pixel attribute format in this embodiment is as illustrated in FIG. 12B, and when the pixel attribute is valid, 1 value is set in the FF circuit 1402 (from the control unit 1040, the signal sig_1402 is transferred). AND element 1412 outputs a 0 value when the pixel attribute is valid. In this case, the LSB of the pixel data signal sig_1404 is fixed at 0 value. When the pixel attribute is invalid, the output of the AND element 1412 becomes equivalent to bit0 of the output of the FF circuit 1401, and all bits of the signal sig_1401 are treated as pixel data.

A signal sig_1407 and a signal sig_1408 are pixel data similar to the signal sig_604 and the signal sig_612 illustrated in FIG. 6, respectively.
A signal sig_1409 is a flag signal indicating that the pixel value of the operation result overlaps with the value of the pixel attribute defined in the first pixel attribute format. In this embodiment, a 0 value is assigned as a saturated pixel. For example, when the value of the signal sig_1410 is fixed at 0 and the pixel value in the comparator 1406 is 0, the value of the signal sig_1409 becomes 1 (this is the same as the signal sig_605 in FIG. 6). At this time, when the transmission of the pixel attribute value is invalid (the register value of the FF circuit 1402 is 1), the signal sig_1409 is masked by the AND element 1407 . As a result, the selector 1409 always selects the signal sig_1408 and outputs it as the signal sig_1412. When pixel attribute value transmission is enabled, the value of the signal sig_1409 is input as the selection signal of the selector 1409, and when the pixel is determined to be a saturated pixel, the 0 value, which is the operation result, is changed to the 1 value. This avoids generating spurious pixel attributes (which is similar to signal sig_607 in FIG. 6).

AND element 1408 is for masking the value of the pixel attribute. The AND element 1408 always sets the value of the signal sig_1415 to 0 when the transmission of pixel attribute values is disabled. The selector 1410 always selects the signal sig_1412 and outputs it as the signal sig_1413. When the register value of the FF circuit 1402 is 1 (transmission of the pixel attribute value is valid) and the pixel to be processed is a saturated pixel, the selector 1410 selects the signal sig_1410. The value of this signal sig_1410 is a 0 value and corresponds to the value of the pixel attribute in the first pixel attribute form. The selector 1410 outputs a signal sig_1413 in which the flag value of the least significant bit of the input signal sig_1401 is replaced with a pixel attribute value. The signal sig_1413 is held by the FF circuit 1411 and is output from the second conversion unit 1400 as the signal sig_1414.

In this embodiment, the FF circuit 1402 can turn ON/OFF the transmission of pixel attribute values. In the first embodiment (also in the circuit illustrated in FIG. 6), similar control is possible by implementing the FF circuit 1402 and mask functions corresponding to the AND elements 1407 and 1408 .
When using the second pixel attribute form in this embodiment, there is a mechanism for controlling whether the least significant bit is treated as a pixel or as a pixel attribute in generating the signal sig_708 in the ALU 10431 illustrated in FIG. is necessary. For example, it is a configuration like the FF circuit 1307 and the selector 1305 illustrated in FIG. Regarding the second pixel attribute form, the first embodiment is used for implementation in a device in which it is important to ensure the calculation accuracy, and the first embodiment is implemented in a device that does not normally use pixel attributes (pixel attributes are positioned as special uses). Therefore, the second embodiment should be selected.

Also in the second embodiment, various modifications described in the first embodiment can be adopted. When a plurality of pixel attributes are handled in the second embodiment, in the second pixel attribute form, for example, the bit to be replaced in the image data is changed for each pixel attribute.

It should be noted that each of the above-described embodiments merely shows specific examples for carrying out the present invention, and the technical scope of the present invention should not be construed to be limited by these. . That is, the present invention can be embodied in various forms without departing from its technical concept or main features.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100: CPU. 101: ROM, 102: RAM, 103: Recording medium, 104: Reconfigurable circuit

Claims (8)

A first conversion for converting data in which a value indicating a pixel attribute is expressed in a first form into data in which a value indicating the pixel attribute is expressed in a second form different from the first form. means and
a second conversion means for converting data in which a value indicating a pixel attribute is expressed in the second form into data in which a value indicating the attribute of the pixel is expressed in the first form;
a first transmission means for transmitting data expressed in the first form to a first transmission line;
a second transmission means for transmitting the data expressed in the second form to a second transmission line different from the first transmission line;
has
The first form is a form in which a value corresponding to the attribute of the pixel is assigned as the pixel value of the pixel in the pixel data,
The second form is characterized in that a value indicating the attribute of the pixel is added to the data of the pixel, or part of the data of the pixel is replaced with a value corresponding to the attribute of the pixel. image processing device. The data expressed in the second form is not transmitted through the first transmission line, and the data expressed in the first form is not transmitted through the second transmission line. The image processing apparatus according to claim 1. wherein said second conversion means converts the data expressed in said second form converted by said first conversion means into data expressed in said first form; Item 3. The image processing device according to Item 1 or 2. wherein said first conversion means converts the data expressed in said first form converted by said second conversion means into data expressed in said second form; 4. The image processing apparatus according to any one of items 1 to 3. The value indicating the attribute of the pixel includes a value indicating a defective pixel, a value indicating a saturated pixel, and a signal used for purposes other than composing a captured image for recording or display. 5. The image processing apparatus according to any one of claims 1 to 4, wherein the pixel is at least one of values indicating that the pixel to be output is a functional pixel. further comprising a plurality of processing means as a data path through which the data is transmitted;
6. The image processing apparatus according to claim 1 , wherein at least one of said plurality of processing means is a processing means whose processing function is updated. The data expressed in the first form is transmitted between the processing means whose processing function is updated and the other processing means,
7. An image processing apparatus according to claim 6 , wherein the data expressed in said second form are transmitted between a plurality of circuits constituting processing means for updating said processing function. A first conversion for converting data in which a value indicating a pixel attribute is expressed in a first form into data in which a value indicating the pixel attribute is expressed in a second form different from the first form. process and
a second conversion step of converting the data in which the value indicating the attribute of the pixel is expressed in the second form into the data in which the value indicating the attribute of the pixel is expressed in the first form;
a first transmission step of transmitting the data expressed in the first form to a first transmission line;
a second transmission step of transmitting the data expressed in the second form to a second transmission line different from the first transmission line;
has
The first form is a form in which a value corresponding to the attribute of the pixel is assigned as the pixel value of the pixel in the pixel data,
The second form is a form in which a value indicating the attribute of the pixel is added to the data of the pixel, or a form in which a part of the data of the pixel is replaced with a value corresponding to the attribute of the pixel. An image processing method characterized by:

Images