JP2024065631A - Image processing device and image processing method - Google Patents

Abstract

[Problem] To efficiently detect image sticking. [Solution] An image processing device according to an embodiment includes a first controller that generates an image, and a second controller that outputs an image input from the first controller or a device other than the first controller. The second controller determines whether or not to perform processing to detect sticking of the input image, depending on whether image input from the first controller has started. [Selected Figure] Figure 1

Description

The present invention relates to a technology for performing image processing.

Conventionally, a technology is known in which one of two processors (e.g., a microcomputer or a central processing unit (CPU)) provided in a device that processes image output responds to an abnormality in the other processor.

For example, Patent Document 1 describes a vehicle data processing device that includes a first CPU and a second CPU. The first CPU outputs an image captured by a camera with guide lines and HMI (Human Machine Interface) display added. If the first CPU has not completed booting or if an abnormality occurs in the operation of the board including the first CPU, the second CPU outputs the image captured by the camera as is, instead of the first CPU.

For example, Patent Document 2 describes a vehicle display system equipped with a main microcomputer and an additional microcomputer. The main microcomputer sequentially outputs multiple frames of a display image based on an image outside the vehicle. The additional microcomputer detects freezing of the display image based on changes in the multiple frames output from the main microcomputer.

JP 2014-197370 A JP 2018-79839 A

However, conventional technology has the problem that it may not be possible to efficiently detect stuck images when they are output.

For example, Patent Document 1 does not describe the period during which abnormality detection of the board including the first CPU is performed. In the technology described in Patent Document 1, the second CPU outputs images while the first CPU is not outputting images. Also, since the second CPU outputs the image captured by the camera as is, it is considered that abnormalities such as sticking are less likely to occur. Therefore, performing abnormality detection including sticking detection during a period when the first CPU is not outputting images may be wasteful, especially in terms of the purpose of detecting image abnormalities.

Furthermore, for example, Patent Document 2 does not state that the additional microcontroller detects sticking depending on the operating status of the main microcontroller, in particular the type of image that the main microcontroller is outputting. Depending on the image that the main microcontroller outputs, the possibility of sticking occurring may be extremely low, so if sticking detection is performed without considering the operating status of the main microcontroller, it is conceivable that waste will occur.

As such, conventional technology can result in unnecessary processing and missed detections, making it difficult to efficiently detect image sticking.

In order to solve the above-mentioned problems and achieve the object, the image processing device according to the present invention includes a first controller that generates an image, and a second controller that outputs an image input from the first controller or a device other than the first controller. The second controller determines whether or not to perform processing to detect adhesion of the input image depending on whether or not image input from the first controller has started.

According to the present invention, image sticking can be detected efficiently.

FIG. 1 is a diagram illustrating an example of the configuration of an image processing apparatus. FIG. 2 is a time chart of the processing executed by the image processing device. FIG. 3 is a flowchart showing the flow of processing by the power supply MCU. FIG. 4 is a flowchart showing the flow of processing of the SoC. FIG. 5 is a flowchart showing the flow of processing in the video IC.

Below, embodiments of the image processing device and image processing method disclosed in the present application will be described in detail with reference to the attached drawings. Note that the present invention is not limited to the embodiments described below.

The configuration of the image processing device will be described using Figure 1. Figure 1 is a diagram showing an example of the configuration of an image processing device.

The image processing device 10 shown in FIG. 1 is mounted on a vehicle. The image processing device 10 is connected to a camera 20 and a vehicle control device 30. The image processing device 10 transmits and receives data between the camera 20 and the vehicle control device 30, for example, by CAN (Controller Area Network) communication.

The camera 20 is provided in the vehicle. In this embodiment, the camera 20 is provided in the rear of the vehicle. For example, the camera 20 captures an image of the rear of the vehicle as the vehicle moves backward.

The camera 20 may capture images of the front of the vehicle and the interior of the vehicle. The camera 20 may also be a so-called all-around camera that captures images of the surroundings of the vehicle.

The vehicle control device 30 controls the vehicle. The vehicle control device 30 is, for example, an ECU (Electronic Control Unit). The vehicle control device 30 transmits signals to other devices according to the operation of the vehicle. For example, the vehicle control device 30 transmits a REV signal indicating that the gear of the vehicle has been shifted to reverse.

The image processing device 10 displays an image on a display. The image processing device 10 can display an input image as is without processing it. The image processing device 10 can also process an input image. The image processing device 10 can also display a processed image. Note that the image referred to here includes both still images and moving images. In the following description, moving images may be referred to as video.

The image processing device 10 receives input of captured images of the exterior or interior of the vehicle, a car navigation system screen, a screen for presenting information about the vehicle to the driver, video content for entertainment, etc. The image processing device 10 can output not only images but also audio. The image processing device 10 is sometimes called a display audio.

Next, the configuration and operation of the image processing device 10 will be described. In the following, we will first explain the configuration of the image processing device 10, and then explain the operation of each component of the image processing device 10.

[Configuration of the image processing device]
As shown in FIG. 1, the image processing device 10 includes a display unit 11, a video IC (Integrated Circuit) 12, a SoC (System on a Chip) 13, a memory unit 14, a power supply MCU (Micro Controller Unit) 15, and a signal detection circuit 16.

The display unit 11 displays an image based on the input signal. The display unit 11 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display.

The video IC 12 is an IC that performs image processing. The video IC 12 includes an A/D conversion circuit 121, a video MCU 122, a MUX (multiplexer) 123, and a register 124.

The A/D conversion circuit 121 converts the input analog signal into a digital signal. The A/D conversion circuit 121 outputs the digital signal obtained by the conversion.

The video MCU 122 monitors the signals being sent and received, and the data stored in the storage device. The video MCU 122 also performs processing to detect image freeze.

For example, the video MCU 122 detects that the video is stuck when one or more specific pixels (or areas) in the video, called viewpoints, do not change for a certain period of time. Also, for example, the video MCU 122 performs sticking detection on the video output from the SoC 13. Note that the method of sticking detection by the video MCU 122 and the video that is the subject of sticking detection are not limited to those described here.

The MUX (multiplexer) 123 combines multiple signals. The MUX 123 also outputs the combined signal.

Register 124 stores information about the state of the MCU.

SoC13 processes the input image. SoC13 outputs the processed image. The process by which SoC13 outputs the processed image is called generation. In other words, SoC13 generates an image.

SoC13 adds HMI, draws objects (guide lines, caution messages, etc.) on the camera image, performs I (Interlace)/P (Progressive) conversion, etc.

For example, when the camera 20 captures an image of the rear of the vehicle, the SoC 13 superimposes guide lines that predict the vehicle's course on the captured image.

For example, when the camera 20 captures images of the vehicle's surroundings, the SoC 13 stitches together the captured images to generate a bird's-eye view image of the vehicle as seen from above.

The storage unit 14 stores the video data. The storage unit 14 is, for example, a flash memory.

The power supply MCU 15 outputs signals to multiple devices according to the input signals.

The signal detection circuit 16 outputs a signal corresponding to a change in the voltage of the input signal. For example, the signal detection circuit 16 outputs a binary signal indicating whether or not the voltage of the input signal exceeds a threshold value.

Here, the first controller and the second controller (hereafter referred to as controllers without distinction) are realized by the computer's CPU functioning as a controller and reading and executing a program stored in a ROM (Read Only Memory).

SoC13 is an example of a first controller. Video MCU122 is an example of a second controller.

The controller may also be realized by a CPU, a DSP (Digital Signal Processor), an ECU, an FPGA (Field Programmable Gate Array), a GPU (Graphics Processing Unit), etc.

The controller may be a single processor or a multi-processor configuration. The controller may also be a multi-core configuration with multiple cores in a single chip connected to a single socket.

[Operation of the image processing device]
The operation of the image processing device 10 will be described with reference to Fig. 2. Fig. 2 is a time chart of the processing executed by the image processing device. In the following description, the time from time _ta to time _tb (where _tb ≧ _ta ) will be expressed as _tb - _ta .

In addition, in the initial state, the video MCU 122 does not perform video fixation detection.

The camera 20 inputs the captured images to the image processing device 10. The camera 20 does not input images to the image processing device 10 while not capturing images.

The camera 20 takes pictures when certain conditions are met. The camera 20 takes pictures when the vehicle's gear is in reverse. The camera 20 obtains information about the vehicle's gear from the vehicle control device 30.

When the vehicle gear is shifted to reverse, the vehicle control device 30 sends a REV signal to the image processing device 10.

When the voltage of the REV signal exceeds a threshold value (time t ₁ ), the signal detection circuit 16 transmits a REV-DET signal to the power supply MCU 15 (time t ₂ ).

Upon receiving the REV-DET signal, the power supply MCU 15 transmits a REV-OUT signal to the video IC 12 and the SoC 13 (time t ₃ ).

Upon receiving the REV-OUT signal, SoC 13 starts processing the image input from camera 20 (time t ₃ ).

At this time, the A/D conversion circuit 121 is assumed to be outputting the image output from the camera 20 to the SoC 13. The A/D conversion circuit 121 may output the image to the SoC 13 via the video MCU 122, or may output the image directly to the SoC 13.

At time _t3 , SoC 13 starts a process to establish I2C (Inter-Integrated Circuit) communication with video IC 12 (UBOOT1). Then, at time _t4 , SoC 13 starts a process to load programs and data to main memory for image processing (UBOOT2). Furthermore, at time _t6 , SoC 13 starts a process to actually process the image (Kernel). After that, at time _t8 , SoC 13 starts outputting the processed image (SoC camera image).

When the I2C communication is established (time t ₄ ), the video IC 12 starts a process of switching the video output by the display unit 11 to the bypass camera image. Note that the process of the video IC 12 described here may actually be performed by the video MCU 122.

At this point, the display unit 11 is assumed to be outputting a black image as the video. In addition, the video IC 12 may cause the display unit 11 to output a video based on data acquired from the memory unit 14 until switching to the bypass camera image is performed.

When the image switching process is completed (time _t5 ), the image IC 12 outputs the bypass camera image to the display unit 11. The bypass camera image is an image input from the camera 20, and is an image that has not been processed by the SoC 13.

At time _t8 , the SoC 13 starts outputting the SoC camera image. Also at time _t8 , the video IC 12 and the display unit 11 start outputting the SoC camera image.

Furthermore, at time _t8 or substantially simultaneously with time _t8 , the video MCU 122 enables sticking detection. Note that substantially simultaneously with time _t8 refers to the period from time _t8 until a monitoring period, which will be described later, has elapsed.

In this way, the video MCU 122 disables sticking detection while the bypass camera image is being output, and enables sticking detection while the SoC camera image, which is more likely to stick than the bypass camera image, is being output. This makes it possible to efficiently detect sticking of the SoC camera image.

[Procedure for enabling sticking detection]
An image is input to the video IC 12 from the SoC 13 or a device other than the SoC 13 (for example, the camera 20). Then, the video IC 12 outputs the image input from the SoC 13 or the device other than the SoC 13.

Here, a value that changes depending on the image input source is stored at a specific address in register 124. This value changes when input of a SoC camera image starts from SoC 13. The value of the address may be a binary value having two types of values, for example, a value indicating that a SoC camera image is being input and a value indicating that an image from camera 20 is being input.

The video MCU 122 references the value of register 124, which changes depending on the image input source, and enables sticking detection when the value indicates that an image is being input from SoC 13. Enabling sticking detection includes deciding to perform processing to detect sticking of the input image, and actually starting sticking detection.

In this way, the video MCU 122 can easily determine the timing to enable sticking detection by simply referencing a specific value in the register 124.

In this way, the video MCU 122 determines whether or not to perform processing to detect freezing of the input image depending on whether image input from SoC 13 has started.

For example, the video MCU 122 can enable sticking detection without causing a time lag by monitoring the register 124 at a specific monitoring cycle (e.g., about 1 ms to 5 ms). As a result, the image processing device 10 can efficiently detect sticking of the output image.

In addition, the video MCU 122 disables the sticking detection when the value of the register 124 indicates that an image is being input from the camera 20. In other words, when image input from the SoC 13 has not started and image input from the camera 20 has started, the video MCU 122 decides not to perform processing to detect sticking of the input image.

This allows the video MCU 122 to not only enable sticking detection, but also disable sticking detection without any time lag. As a result, the image processing device 10 can more efficiently detect sticking of the output image.

The value of register 124 may preferentially indicate whether or not a SoC camera image is being input. In that case, when images are being input to video IC 12 from both SoC 13 and camera 20, the value of register 124 indicates that a SoC camera image is being input. When no image is being input from SoC 13 to video IC 12, the value of register 124 indicates that an image from camera 20 is being input.

In order to eliminate the waste that occurs in the techniques described in the above-mentioned Patent Document 1 or Patent Document 2, it is conceivable to start image sticking detection after a certain time has elapsed from the point in time when the REV signal is generated (time _t1 ). In this case, a time lag occurs between the timing when the image output actually starts and the timing when sticking detection starts, which may result in missed detection.

For example, consider a case where the stuck detection is enabled 2.5 seconds after the REV signal is generated. The time t _{- t} from time _t to a point (time _t ) when the SoC ₁₃ is executing the kernel is about 2 seconds. The time t after _t is _2.5 seconds.

Therefore, a time lag of _t9 - _t8 occurs. Also, there is a possibility that a detection miss occurs during the time lag. Note that if 2.5 seconds have passed since time _t1 before the output of the SoC camera image starts, a period will occur during which the sticking detection is needlessly enabled.

This embodiment not only reduces the period during which the sticking detection is unnecessarily enabled, but also prevents missed detections due to time lags.

In the example of FIG. 2, when SoC13 receives a signal (REV-OUT signal) indicating that the vehicle gear has been changed into reverse, it starts a process of processing the image input from the camera 20 mounted on the vehicle and outputting it to the video MCU 122. Also, when the video MCU 122 receives the REV-OUT signal, it outputs the image input from the camera 20 and checks whether image input from SoC13 has started. Then, when the video MCU 122 checks that image input from SoC13 has started, it decides to detect the fixation of the input image.

Note that the trigger for SoC13 and video MCU122 to start processing is not limited to the REV-OUT signal described above, but may be any signal that indicates a change in vehicle gear.

This allows the video MCU 122 to enable sticking detection in response to the vehicle's movements. As a result, the image processing device 10 can more efficiently detect sticking of the output image in accordance with the vehicle's movements.

[Processing flow of image processing device]
The process flow of the power supply MCU 15, the video IC 12, and the SoC 13 will be described using a flowchart.

Figure 3 is a flowchart showing the processing flow of the power supply MCU. As shown in Figure 3, the power supply MCU 15 waits until a specific signal is generated (step S101, No). In other words, the power supply MCU 15 waits until it receives a specific signal (e.g., a REV-DET signal) from the vehicle control device 30 via the signal detection circuit 16.

When a specific signal occurs (step S101, Yes), the power supply MCU 15 notifies the video IC 12 and SoC 13 that a signal has occurred (step S102). That is, when the power supply MCU 15 receives a specific signal (e.g., REV-DET), it notifies the video IC 12 and SoC 13 of the occurrence of the signal in a format that can be interpreted by the video IC 12 and SoC 13 (e.g., a REV-OUT signal).

Figure 4 is a flowchart showing the flow of SoC processing. As shown in Figure 4, SoC13 waits until it is notified that a specific signal has occurred (step S201, No). In other words, SoC13 waits until it receives a signal equivalent to the notification (e.g., a REV-OUT signal) from power supply MCU15.

When notified that a specific signal has been generated (step S201, Yes), SoC13 generates a SoC camera image based on the camera image (image captured by camera 20) acquired from video IC12 (step S202). Then, SoC13 outputs the generated image to video IC12 (step S203).

In other words, from step S203 onwards, the camera image processed by the SoC13 is input to the video IC12.

Figure 5 is a flowchart showing the flow of processing in the video IC. Each process shown in Figure 5 is executed by the video MCU 122. As shown in Figure 5, the video MCU 122 waits until it is notified that a specific signal has occurred (step S301, No). In other words, the video MCU 122 waits until it receives a signal equivalent to the notification (e.g., a REV-OUT signal) from the power supply MCU 15.

When notified that a specific signal has occurred (step S301, Yes), the video MCU 122 checks whether a SoC image (an image input from the SoC) can be acquired from the SoC 13 (step S302).

If the video MCU 122 is able to acquire a SoC image from the SoC 13 (step S302, Yes), it outputs the SoC image to the display unit 11 (step S303).

On the other hand, if the video MCU 122 is unable to acquire the SoC image from the SoC 13 (step S302, No), it outputs the bypass camera image (the camera image itself) to the display unit 11 (step S304).

In other words, after the output of an image by the SoC 13 begins, the video MCU 122 outputs the SoC image.

The video IC 12 may output the bypass camera image directly to the display unit 11 via the A/D conversion circuit 121 and MUX 123 without going through the video MCU 122.

Here, the video MCU 122 checks the value of the register 124 (step S305). The value of the register 124 indicates whether or not an image is being input from the SoC 13 to the video IC 12.

If the value of register 124 indicates that no image is being input from SoC 13 to video IC 12 (step S305, bypass camera side), video MCU 122 disables sticking detection (step S306). In other words, video MCU 122 does not perform sticking detection on bypass camera images.

When no image is input from SoC 13 to video IC 12, the source of the image input is not limited to the bypass camera, but may be, for example, an image stored in advance in memory unit 14, etc.

On the other hand, if the value of register 124 indicates that an image is being input from SoC 13 to video IC 12 (step S305, SoC side), video MCU 122 enables sticking detection (step S307). That is, video MCU 122 performs sticking detection for the SoC image.

After step S306 or step S307, the video MCU 122 returns to step S302.

Further advantages and modifications may readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and equivalents thereof.

10 Image processing device 11 Display unit 12 Image IC
13 SoC
14 Memory unit 15 Power supply MCU
16 Signal detection circuit 20 Camera 30 Vehicle control device 121 A/D conversion circuit 122 Video MCU
123 MUX
124 Register

Claims (6)

A first controller that generates an image, and a second controller that outputs an image input from the first controller or a device other than the first controller,
The second controller includes:
an image processing apparatus that determines whether or not to perform a process for detecting sticking of an input image depending on whether or not image input from the first controller has started. The second controller includes:
Refer to the register value, which changes depending on the image input source.
The image processing apparatus according to claim 1 , further comprising: a step of: determining, when the value indicates that an image is being input from the first controller, to perform a process for detecting sticking of the input image. The first controller includes:
Processing an image input from a camera and outputting the processed image to the second controller;
The second controller includes:
Outputting an image input from the first controller or the camera;
determining, when image input from the first controller is started, to perform a process for detecting sticking of the input image;
The image processing device according to claim 1 , wherein when image input from the first controller has not started and image input from the camera has started, it is determined not to perform processing for detecting sticking of the input image. The first controller includes:
when a signal indicating a gear change of the vehicle is received, an image input from a camera provided in the vehicle is processed and output to the second controller;
The second controller includes:
When the signal is received, outputting the image input from the camera and checking whether image input from the first controller has started;
The image processing apparatus according to claim 1 , further comprising: a determining unit configured to determine whether or not a part of the image being inputted is stuck when the start of image input from the first controller is confirmed. The first controller includes:
when receiving a signal indicating that the gear of the vehicle has been changed to reverse, starting a process of processing an image input from a camera provided in the vehicle and outputting the processed image to the second controller;
The second controller includes:
When the signal is received, outputting the image input from the camera and checking whether image input from the first controller has started;
The image processing apparatus according to claim 1 , further comprising: a step of determining whether or not the input image is stuck when it is determined that the image input from the first controller has started. 1. An image processing method executed by a computer including a first controller that generates an image, and a second controller that outputs an image input from the first controller or a device other than the first controller, comprising:
The second controller includes:
an image processing method comprising: determining whether or not to perform processing for detecting sticking of an input image depending on whether or not image input from the first controller has started;

Images