JP7464801B2 - IMAGE PROCESSING APPARATUS AND IMAGE DATA TRANSMISSION METHOD - Google Patents

Description

The present invention relates to an image processing device and an image data transmission method that suppress quality degradation caused by external noise when image data output by an imaging element is transmitted to an image processing circuit.

In recent years, in order to realize more advanced driving assistance systems and autonomous driving systems, there has been a demand for more sophisticated external recognition functions for in-vehicle cameras, and one way of achieving this is to increase the resolution of image sensors (e.g., CMOS image sensors) that capture the external world. As a result, the communication frequency (data rate) required to transmit image data output by the image sensor to the image processing circuit is increasing, making it easier for communication reliability to decrease due to the effects of external noise.

Conventional methods for dealing with external noise have mainly been hardware-based, such as making the housing of an in-vehicle camera out of metal to shield the housing from external noise. However, when using communication frequencies higher than conventional methods, hardware-based measures alone may not be able to adequately ensure communication quality in environments where various types of external noise are present.

Therefore, Patent Document 1 proposes a method of reducing the deterioration of communication quality due to the influence of external noise by software. For example, the abstract of the document states that the problem is to "provide an imaging device that can reduce noise due to the influence of actuators, etc. without affecting the frame rate," and as a solution, it states that the device has a pixel section in which a plurality of pixels that perform photoelectric conversion are arranged in a matrix, a pixel signal readout section that includes a function of reading out the reset level and signal level of the pixels from the pixel section to a signal line, and a timing control section that controls the readout timing of the pixel signal readout section, and the pixel signal readout section includes a solid-state imaging element having a column-parallel ADC (Analog Digital Converter) section that converts the analog signal read out in accordance with the pixel column arrangement into a digital signal, a timing change section that changes the readout timing of the pixel signal readout section, and a storage section that stores frequency information of noise generated during the operation of the pixel signal readout section, and determines the readout timing of the pixel signal readout section and the resolution (number of bits) of the AD conversion according to the noise frequency information.

JP 2014-236252 A

However, as in Patent Document 1, when the resolution (number of bits) of the AD conversion is changed according to the noise frequency, if the bit depth of the image data is reduced (for example, changing from RAW12 format to RAW10 format image data), it becomes difficult to recognize color differences in the captured outside world, and there is a problem that the recognition accuracy of the outside world deteriorates. On the other hand, if the bit depth of the image data is increased (for example, changing from RAW12 format to RAW14 format image data), there are problems such as insufficient processing capacity of the image processing circuit for recognizing the outside world and increased heat generation due to increased power consumption of the image processing circuit.

In addition, when the bit depth of image data is changed, the image processing algorithm of the image processing circuit needs to be switched. However, since image processing algorithms cannot be switched dynamically, each time the bit depth of the image data is changed, the image processing circuit needs to be temporarily stopped and a period of time needs to be set aside for switching the image processing algorithm. During this period, the outside environment cannot be recognized, which is a fatal problem in terms of continuing vehicle control by driving assistance systems or autonomous driving systems.

Therefore, the present invention aims to provide an image processing device and an image data transmission method that suppress deterioration of communication quality caused by external noise while maintaining the bit depth and frame rate of image data.

In order to solve the above problem, the present invention provides an image processing device that includes an imaging element that acquires image data at a predetermined frame rate, an image processing unit that processes the image data, a transmission unit that transmits the image data from the imaging element to the image processing unit at a predetermined communication frequency, and a communication control unit that instructs the transmission unit to change the communication frequency while maintaining the predetermined frame rate in accordance with the frequency of external noise.

The image processing device and image data transmission method of the present invention can suppress deterioration of communication quality due to external noise while maintaining the bit depth and frame rate of the image data.

FIG. 1 is a front view of a vehicle equipped with an image processing device according to a first embodiment. FIG. 1 is a functional block diagram showing a configuration of an image processing apparatus according to a first embodiment. 4 is a process flowchart of the image processing apparatus according to the first embodiment. An example of the noise level of external noise An example of the relationship between the standard frequency for communication and the changed frequency An example of a communication frequency change method according to the first embodiment Another example of the communication frequency change method according to the first embodiment FIG. 13 is an explanatory diagram of an error correction method using two identical image data. FIG. 11 is a functional block diagram showing the configuration of an image processing apparatus according to a second embodiment.

Below, embodiments of the image processing device and image data communication method of the present invention are described with reference to the drawings.

Figure 1 is a front view of a vehicle V equipped with an image processing device 100 according to a first embodiment of the present invention. As shown here, the image processing device 100 is attached to the upper part of the inner surface of the windshield of the vehicle V, which is an environment that is susceptible to the influence of external noise, so as to be able to capture an image of the area in front of the vehicle V. Note that while Figure 1 illustrates an image processing device 100 that captures an image of the area in front of the vehicle V, an image processing device 100 for capturing an image of the area behind or to the side of the vehicle V may also be provided.

Figure 2 is a functional block diagram of the image processing device 100 of Figure 1. The image processing device 100 is a device for recognizing the external environment of the vehicle V when performing vehicle control such as emergency automatic braking and adaptive cruise control (ACC), and has an imaging unit 1, an image processing unit 2, an antenna 3, and a communication control unit 4. In this embodiment, the imaging unit 1 and the image processing unit 2 are installed on the same board at a certain distance apart, and are connected to each other via a wiring pattern on the board or a communication cable. Details of each part will be explained in sequence below.

<Imaging unit 1>
The imaging unit 1 is a unit for capturing images of the external environment of the vehicle V at a predetermined frame rate and then outputting image data D of a predetermined bit depth, and includes an imaging section 11 and a transmission section 12 .

The imaging unit 11 is a combination of pixels (photodiodes, etc.) of an imaging element (CMOS image sensor, etc.), logic circuits, etc., and captures the external environment of the vehicle V at a predetermined frame rate and then outputs image data D of a predetermined bit depth. The imaging unit 1 may be a monocular camera or a compound eye camera such as a stereo camera.

The transmission unit 12 transmits the image data D captured by the imaging unit 11 to the image processing unit 2 using a predetermined communication protocol. The communication protocol used here is, for example, MIPI CSI-2 (Mobile Industry Processor Interface, registered trademark), which is an interface standard for cameras in mobile devices.

<Image Processing Unit 2>
The image processing unit 2 is a unit for recognizing the external environment of the vehicle V, and has a receiving section 21 and an image recognition section 22.

The receiving unit 21 decodes the image data D of a specified communication protocol received from the transmitting unit 12 into a format that can be processed by the next image recognition unit 22.

The image recognition unit 22 processes the decoded image data D to recognize the external environment of the vehicle V (e.g., preceding vehicles, pedestrians, traffic signs, traffic signals, white lines on the road, etc.), and outputs the image recognition results. The image recognition results output by the image recognition unit 22 are transmitted via a controller area network (CAN) to an electronic control unit (ECU) that controls the drive system, braking system, steering system, etc. of the vehicle V, and are used for vehicle control, etc.

<Antenna 3>
The antenna 3 is a device for receiving external noise that may affect the transmission of image data D from the imaging unit 1 to the image processing unit 2. It is preferable to place the antenna 3 inside the image processing device 100 so that external noise that has entered the device can be observed, but the antenna 3 may be placed outside the image processing device 100 when it is desired to increase the reception sensitivity of external noise.

<Communication control unit 4>
The communication control unit 4 is a unit for setting the communication method from the imaging unit 1 to the image processing unit 2 to a communication method that is less susceptible to the influence of external noise, and includes a noise influence prediction section 41, a database 42, and a communication control section 43. Specifically, the communication control unit 4 is a computer unit including a calculation device such as a CPU, a storage device such as a semiconductor memory, and hardware such as a communication device. The calculation device executes a program stored in the storage device to realize the functions of the noise influence prediction section 41 and the communication control section 43, but each section will be described below while appropriately omitting such well-known techniques in the computer field.

Here, using the processing flowchart of Figure 3, we will explain the details of the processing by the image processing device 100 of this embodiment, particularly the communication control unit 4.

First, in step S1, the noise influence prediction unit 41 analyzes noise information (noise level, frequency) of the exogenous noise received by the antenna 3. Fig. 4 shows an example of noise information analyzed by the noise influence prediction unit 41, in which the Fourier transform result of the exogenous noise is reflected in a two-dimensional space with the horizontal axis representing frequency and the vertical axis representing noise level. In this example, the noise level of the exogenous noise is maximum near the reference frequency _f0 used for communication.

Next, in step S2, the noise influence prediction unit 41 refers to the noise resistance information 42a registered in the database 42. Here, the contents of the noise resistance information 42a will be specifically described with reference to FIG. 5. The dotted line in the figure illustrates an example of noise resistance information when the reference frequency f ₀ is used for communication, and the solid line in the figure illustrates an example of noise resistance information when the reference frequency f _X is used for communication. As is clear from the figure, each of the noise resistance information has low noise resistance against external noise in three frequency bands including each communication frequency, and the frequency bands with low noise resistance of each noise resistance information do not overlap. Note that only two types of noise resistance information are illustrated in FIG. 5, but it is assumed that noise resistance information for many other communication frequencies is also registered in advance in the database 42.

In step S3, the noise influence prediction unit 41 determines whether the image data D will be affected by external noise when transmitted from the imaging unit 1 to the image processing unit 2 based on the noise tolerance information 42a obtained from the database 42. If the determination is No, the process proceeds to step S4, and if the determination is Yes, the process proceeds to step S5.

For example, if the reference frequency _f0 is used to transmit image data D and the environment has an external noise level as shown in Fig. 5, the external noise level exceeds the noise tolerance of the reference frequency _f0 (the noise tolerance of the dotted line in Fig. 5), and it can be determined that the communication is affected by noise. On the other hand, if the reference frequency _fX is used to transmit image data D and the environment has an external noise level as shown in Fig. 5, the external noise level does not exceed the noise tolerance of the reference frequency _fX (the noise tolerance of the solid line in Fig. 5), and it can be determined that the communication is not affected by noise. As can be seen from this explanation, by switching the communication frequency depending on the type of external noise, it is possible to avoid the effect of external noise on communication.

If the communication is not affected by external noise, in step S4, the communication control unit 43 instructs the transmission unit 12 of the imaging unit 1 to continue communication at the current frequency (e.g., reference frequency f ₀ ). Note that in this step, since there is no need to change the communication frequency, the communication control unit 43 does not need to send a special instruction to the transmission unit 12.

On the other hand, if the communication is affected by the external noise, in step S5, the communication control unit 43 commands the transmission unit 12 of the imaging unit 1 to change the communication frequency to another frequency. The frequency change here is performed by referring to the noise information (noise level, frequency) of the external noise obtained in step S1 and the noise resistance information 42a obtained in step S3. In the example of Fig. 5, it can be determined from the relationship between the external noise and the noise resistance shown in the figure that the influence of the external noise can be avoided by changing the communication frequency from the reference frequency _f0 to the frequency _fX , so the communication control unit 43 commands the transmission unit 12 to set a setting value for changing the communication frequency to the frequency _fX .

Next, in step S6, the communication control unit 43 determines whether the communication frequency changed in step S5 is equal to or greater than twice the reference frequency f _0. If the determination is No, the process proceeds to step S7, and if the determination is Yes, the process proceeds to step S8.

If the communication frequency is not twice the reference frequency _f0 or more, in step S7, the communication control unit 43 calculates a blank time that can offset the increase or decrease in the data transmission time and maintain the desired frame rate period T when the data transmission time increases or decreases due to the change in the communication frequency, and outputs a corresponding blank time adjustment command to the transmission unit 12. FIG. 6A is a diagram for explaining an example of the change in the communication frequency in this step. In FIG. 6A, the first half illustrates the lengths of the data transmission time _t1 and the blank time _t2 when the communication frequency is the reference frequency _f0 , and the second half illustrates the lengths of the data transmission time _t3 and the blank time _t4 when the communication frequency is frequency _f1 (where _f1 < _2f0 ). In this way, even if the data transmission time increases or decreases due to the change in the communication frequency, the desired frame rate period T can be maintained by adjusting the length of the blank time. In FIG. 6A, frequency _f1 is higher than reference frequency _f0 (data transmission time _t3 is shorter than data transmission time _t1 ), but frequency _f1 may be lower than reference frequency _f0 (data transmission time _t3 is longer than data transmission time _t1 ).

Here, the reason for fixing the length of the frame rate period T in this embodiment will be explained. The frame rate period T shown in FIG. 6A is the time secured for image processing of one image data D by the image processing unit 2. If the frame rate period T is too long (if the image data D is received too infrequently), the image processing unit 2 may not be able to properly detect changes in the external environment due to insufficient information. Conversely, if the frame rate period T is too short (if the image data D is received too frequently), the image processing unit 2 may not be able to output the image recognition result at the appropriate timing due to insufficient computing power. Therefore, in this embodiment, a constant frame rate period T that can properly detect changes in the external environment and does not exceed the computing power of the image processing unit 2 is pre-defined as a specification, and is maintained even if the communication frequency increases or decreases. In the above, the length of the blank time is adjusted so that the frame rate period T is constant, but it is not necessary for the frame rate period T to be strictly constant, and the length of the frame rate period T may be variable within the range in which the image recognition/processing performance and power consumption are within the product specifications.

On the other hand, when the communication frequency is twice the reference frequency _f0 or more, in step S8, the communication control unit 43 sets the transmission unit 12 to continuously transmit the same image data D multiple times within one frame rate period T. Fig. 6B is a diagram for explaining an example of the change of the communication frequency in this step. In Fig. 6B, the first half illustrates an example in which the image data D is transmitted only once within one frame rate period T when the communication frequency is the reference frequency _f0 , and the second half illustrates an example in which the same image data D is transmitted twice within one frame rate period T when the communication frequency is frequency _f2 (where _2f0 < _f2 ).
As shown in the latter half of Fig. 6B, when the communication frequency becomes twice the reference frequency _f0 or more, the data transmission time required to transmit the image data D is shortened, so that the same image data D can be transmitted multiple times in succession within the frame rate period T. In this case, too, a constant frame rate period T is maintained by appropriately adjusting the length of the blank time _t6 . Note that, although Fig. 6B illustrates an example of a situation in which the same image data D is transmitted twice in succession within one frame rate period T, if the communication frequency is higher, the same image data may be transmitted three or more times in succession.

Figure 7 is an explanatory diagram of an error correction method that can be executed by the image processing unit 2 when the same image data D is received twice within one frame rate period T. When image data D is transmitted from the imaging unit 1 to the image processing unit 2, there is a possibility that errors will occur in different parts of the first and second image data D due to the influence of external noise. Therefore, by aggregating the error-free parts of each image data D, it is possible to synthesize image data D that is error-free (or has few errors) as a whole. For example, when MIPI CSI-2 is used as the communication protocol, an ECC (Error-Correcting Code) and checksum are assigned to each line into which the image data D is divided, so the presence or absence of errors can be determined for each line by referring to the ECC, etc.

More specifically, the presence or absence of an error in the first image data is determined for each horizontal line.
The same judgment is made for the second and subsequent images, and lines with no errors in each image data are selected. This is repeated from line 1 to line n, and the images are combined into one image data. This makes it possible to obtain image data without errors. Only the combined image data is sent to the image processing unit, so the image recognition and processing performance of the image processing unit does not change before or after the frequency change.

<Effects of this embodiment>
As described above, according to the image processing device of this embodiment, it is possible to suppress the deterioration of communication quality due to external noise while maintaining the bit depth and frame rate of image data. As a result, the amount of information of image data sent to the image processing unit does not change before and after the change of communication frequency to avoid the influence of external noise, so there is no need to change the image processing algorithm or provide a processing stop period for the image processing unit as in the conventional technology. As a result, since the external environment recognition process can be continued even when the communication frequency is changed, a driving assistance system or an automatic driving system that uses the image recognition result of the image processing device of the present invention can continue vehicle control according to the external environment without a pause period.

Next, an image processing device 100 according to a second embodiment of the present invention will be described with reference to FIG. 8. Note that a duplicated description of the points common to the first embodiment will be omitted.

The image processing device 100 of the first embodiment includes an antenna 3 and a noise influence prediction unit 41, and determines a communication frequency for avoiding external noise by using the analysis result (noise information) of the output of the antenna 3 and the noise tolerance information 42a in the database 42. In contrast, the image processing device 100 of the present embodiment uses the setting value table 42b in the database 42 to obtain the same effect as in the first embodiment, even if the image processing device 100 does not include an antenna 3.

In the image processing device 100 of this embodiment shown in Figure 8, the receiving section 21 in the image processing unit 2 checks for the presence or absence of errors in the image data D and determines whether it is affected by external noise. As described above, if MIPI CSI-2 is used as the communication protocol, the presence or absence of errors can be determined for each line that makes up the image data D by referring to the ECC and checksum. Also, as shown in the second half of Figure 6B as an example, if the same image data D is transmitted multiple times within one frame rate period T, the multiple image data D can be compared and the presence or absence of errors can be determined from the presence or absence of differences.

If the receiving unit 21 determines that an error has occurred by any method, the communication control unit 43, which has received the notification, refers to the communication frequency setting value table 42b stored in the database 42, and transmits setting value information (such as the changed communication frequency) selected from multiple setting value information previously registered in the table to the transmitting unit 12. The transmitting unit 12 then changes the communication frequency based on the received information. By repeating this process until the communication error of the image data is eliminated, it is possible to obtain the same effect as in Example 1, even in a device that does not have an antenna 3.

100...image processing device, 1...imaging unit, 11...imaging section, 12...transmission section, 2...image processing unit, 21...receiving section, 22...image recognition section, 3...antenna, 4...communication control unit, 41...noise impact prediction section, 42...database, 42a...noise resistance information, 42b...setting value table, 43...communication control section, V...vehicle, D...image data

Claims (13)

An image sensor for acquiring image data at a predetermined frame rate;
an image processing unit for processing the image data;
a transmission unit that transmits the image data from the imaging element to the image processing unit at a predetermined communication frequency;
a communication control unit that instructs the transmission unit to change the communication frequency while maintaining the predetermined frame rate in accordance with a frequency of an external noise;
An image processing device comprising: 2. The image processing device according to claim 1,
The image processing device is characterized in that, after transmitting the image data at the changed communication frequency, the communication control unit provides a blank time adjusted to change the frame rate within the range in which image recognition/processing performance and power consumption fall within the product specifications. 2. The image processing device according to claim 1,
The image processing device according to claim 1, wherein the transmitting section transmits the image data a plurality of times within one period of the frame rate. 4. The image processing device according to claim 3,
The image processing device is characterized in that the image processing unit has a receiving section that synthesizes multiple image data into one image. 5. The image processing device according to claim 4,
The image processing device according to the present invention, wherein the receiving unit compares a plurality of pieces of image data and determines whether there is a difference between the pieces of image data. 6. The image processing device according to claim 5,
The image processing apparatus according to claim 1, wherein the communication control unit selects a setting value table and changes a communication frequency based on a result of the determination. 2. The image processing device according to claim 1,
The image processing device according to claim 1, wherein the communication control unit compares noise information with noise resistance information in a database based on the noise information and determines whether the noise information will affect transmission. 2. The image processing device according to claim 1,
The image processing device according to claim 1, wherein the communication control unit changes the communication frequency again when a transmission error is detected by the image processing unit. 4. The image processing device according to claim 3,
The image processing device according to claim 1, wherein the communication control unit changes the communication frequency again when a comparison result of the plurality of image data in the image processing unit indicates a mismatch. An image data transmission method for controlling a communication frequency used when transmitting image data from an imaging unit to an image processing unit in an image processing device, comprising:
an imaging element of the imaging unit capturing an image of an external environment at a predetermined frame rate and outputting the image data;
a transmitting section of the imaging unit transmitting the image data to the image processing unit at a predetermined communication frequency;
When the transmitted image data is affected by external noise, the transmitting unit changes the communication frequency to a communication frequency that is not affected by the external noise;
13. An image data transmission method comprising: The image data transmission method according to claim 10,
2. An image data transmission method comprising the step of: when the communication frequency is changed, offsetting an increase or decrease in data transmission time to maintain the predetermined frame rate. The image data transmission method according to claim 10,
A step of analyzing external noise received by an antenna and generating noise information;
obtaining noise resistance information of the communication frequency from a database;
determining whether the transmitted image data is affected by external noise based on the noise information and the noise tolerance information;
13. An image data transmission method comprising: The image data transmission method according to claim 10,
A step of determining whether an error exists from the image data received by the image processing unit; and a step of determining that the image data has been affected by external noise if an error exists in the received image data.
13. An image data transmission method comprising:

Images