JP7544012B2 - Control device, control method, and control program - Google Patents

Description

This disclosure relates to control technology for sensor systems and cleaning systems in vehicles.

In a sensor system mounted on a vehicle, a sensing area is set through a sensor surface exposed to the outside world of the vehicle. Therefore, as disclosed in Patent Document 1, vehicles are increasingly being equipped with a cleaning system that cleans the sensor surface by spraying a cleaning fluid from a cleaning nozzle.

JP 2001-171491 A

In the technology disclosed in Patent Document 1, a cleaning fluid is sprayed from a cleaning nozzle as dirt adheres to the front glass surface, which serves as the sensor surface, to clean the sensor surface. However, even on a sensor surface that has been cleaned in this way, differences in cleanliness occur depending on the position and the passage of time. As a result, image data acquired through the sensor surface is uniformly used as normal data, even though this situation involves a decrease in reliability due to differences in cleanliness. In recent years in particular, a decrease in the reliability of image data is undesirable, as there is concern that it may also lead to a decrease in reliability in autonomous driving control.

The object of the present disclosure is to provide a control device that ensures the reliability of image data. Another object of the present disclosure is to provide a control method that ensures the reliability of image data. Yet another object of the present disclosure is to provide a control program that ensures the reliability of image data.

The technical means of the present disclosure for solving the problems will be explained below. Note that the claims and the reference characters in parentheses in this section indicate the corresponding relationship with the specific means described in the embodiments described in detail later, and do not limit the technical scope of the present disclosure.

A first aspect of the present disclosure is
A control device having a processor (12) and controlling a sensor system (4) in which a sensing area (As) is set through a sensor surface (33) exposed to the outside of a vehicle (2), and a cleaning system (5, 3005) that cleans the sensor surface by spraying a cleaning fluid from a cleaning nozzle (50, 3050),
The processor
Controlling the ejection of a cleaning fluid from a cleaning nozzle in a control period (ωc) for controlling the sensor system and the cleaning system;
The system is configured to assign a priority (Pi) to each pixel region (Dp) in image data (Di) acquired through the sensor surface for each frame (Fi) within a control period, based on the elapsed time (Tc) in the control period and the relative position (Xp) with respect to the cleaning nozzle on the sensor surface.

A second aspect of the present disclosure is
A control method executed by a processor (12) for controlling a sensor system (4) in which a sensing area (As) is set through a sensor surface (33) exposed to the outside of a vehicle (2), and a cleaning system (5, 3005) that cleans the sensor surface by spraying a cleaning fluid from a cleaning nozzle (50, 3050), comprising:
Controlling the ejection of a cleaning fluid from a cleaning nozzle in a control period (ωc) for controlling the sensor system and the cleaning system;
This includes assigning a priority (Pi) to each pixel area (Dp) in image data (Di) acquired through the sensor surface for each frame (Fi) within the control cycle, based on the elapsed time (Tc) in the control cycle and the relative position (Xp) with respect to the cleaning nozzle on the sensor surface.

A third aspect of the present disclosure is
A control program stored in a storage medium (10) for controlling a sensor system (4) in which a sensing area (As) is set through a sensor surface (33) exposed to the outside of a vehicle (2) and a cleaning system (5, 3005) that cleans the sensor surface by spraying a cleaning fluid from a cleaning nozzle (50, 3050), the control program including instructions to be executed by a processor (12),
The command is,
In a control period (ωc) for controlling the sensor system and the cleaning system, controlling the ejection of the cleaning fluid from the cleaning nozzle;
This includes assigning a priority (Pi) to each pixel area (Dp) in image data (Di) acquired through the sensor surface for each frame (Fi) within the control cycle, based on the elapsed time (Tc) in the control cycle and the relative position (Xp) with respect to the cleaning nozzle on the sensor surface.

In this way, in the first to third aspects, the spray of cleaning fluid from the cleaning nozzle is controlled in a control cycle that controls the sensor system and cleaning system. According to the first to third aspects, image data acquired through the sensor surface for each frame in the control cycle is assigned a priority for each pixel region based on the elapsed time in the control cycle and the relative position on the sensor surface with respect to the cleaning nozzle. This allows the difference in cleanliness that occurs on the sensor surface depending on the elapsed time and relative position to be reflected in the image priority for each frame and pixel region, making it possible to ensure the reliability of the image data.

1 is a cross-sectional view showing an automatic driving unit equipped with a control device according to a first embodiment, mounted on a vehicle. FIG. 2 is a schematic diagram for explaining image data according to the first embodiment. FIG. 2 is a schematic diagram for explaining a sensor system according to the first embodiment. FIG. 2 is a schematic diagram showing a configuration of a cleaning nozzle according to the first embodiment. 2 is a block diagram showing the functional configuration of an automatic driving unit equipped with a control device according to the first embodiment. FIG. 6 is a block diagram showing a functional configuration, different from that shown in FIG. 5, of an automatic driving unit including a control device according to the first embodiment. FIG. 5 is a schematic diagram showing a configuration of the cleaning nozzle according to the first embodiment, which is different from that shown in FIG. 4 . FIG. 7 is a block diagram showing a functional configuration of an automatic driving unit equipped with a control device according to the first embodiment, which is different from those shown in FIGS. 5 and 6. 9 is a block diagram showing a functional configuration of an automatic driving unit equipped with a control device according to the first embodiment, which is different from those shown in FIGS. 4 is a flowchart showing a control flow according to the first embodiment. 4 is a graph for explaining a priority assignment process according to the first embodiment. FIG. 4 is a schematic diagram for explaining a priority assignment process according to the first embodiment. FIG. 4 is a schematic diagram for explaining a priority assignment process according to the first embodiment. 10 is a flowchart showing a control flow according to a second embodiment. A block diagram showing the functional configuration of an automatic driving unit equipped with a control device according to a third embodiment. A block diagram showing a functional configuration, different from that shown in FIG. 15, of an automatic driving unit equipped with a control device according to a third embodiment. FIG. 11 is a schematic diagram showing the configuration of a cleaning nozzle according to a third embodiment. 13 is a flowchart showing a control flow according to a third embodiment. FIG. 13 is a schematic diagram for explaining a priority assignment process according to a third embodiment. FIG. 13 is a schematic diagram for explaining a priority assignment process according to a third embodiment. 13 is a flowchart showing a control flow according to the fourth embodiment. 13 is a graph for explaining a priority assignment process according to the fourth embodiment.

Below, several embodiments are described based on the drawings. Note that in each embodiment, corresponding components are given the same reference numerals, and duplicated description may be omitted. Furthermore, when only a part of the configuration is described in each embodiment, the configuration of the other embodiment described above may be applied to the other parts of the configuration. Furthermore, in addition to the combinations of configurations explicitly stated in the description of each embodiment, configurations of several embodiments may be partially combined together even if not explicitly stated, as long as there is no particular problem with the combination.

First Embodiment
1, an automatic driving unit ADU including a control device 1 according to the first embodiment is mounted on a vehicle 2. The vehicle 2 is a moving body such as an automobile that can travel on a road with an occupant on board.

In the autonomous driving mode, the vehicle 2 is capable of steady or temporary autonomous driving. Here, the autonomous driving mode may be realized by autonomous driving control, such as conditional driving automation, highly automated driving, or fully automated driving, in which the system performs all driving tasks when activated. The autonomous driving mode may also be realized in advanced driving assistance control, such as driving assistance or partial driving automation, in which the occupant performs some or all driving tasks. The autonomous driving mode may be realized by either one of the autonomous driving control and the advanced driving assistance control, or by a combination of them, or by switching between them.

The autonomous driving unit ADU includes a housing 3, a sensor system 4, and a cleaning system 5 as components mounted on the vehicle 2 together with the control device 1. In the following, the description of the direction of the autonomous driving unit ADU will be given based on the vehicle 2 on a horizontal plane.

The housing 3 is formed of resin, metal, or a combination thereof, and is shaped, for example, as a hollow, flat rectangular box. The housing 3 is installed on the roof 20 of the vehicle 2. The openings that penetrate the outer peripheral wall portion 30 of the housing 3 at multiple points are covered by sensor covers 32, for example, made of transparent glass. Each sensor cover 32 forms a sensor surface 33 that is exposed to the outside of the vehicle 2 and into which light enters from the outside.

The sensor system 4 is mainly composed of multiple external sensors 40. Each external sensor 40 is housed inside the housing 3 at a position corresponding to an individual sensor surface 33. Therefore, hereinafter, the sensor surface 33 corresponding to an external sensor 40 will simply be referred to as the sensor surface 33 of the external sensor 40.

As shown in FIG. 2, each external sensor 40 acquires image data Di for each frame Fi at a predetermined interval as sensing data representing external information that can be used in the autonomous driving mode of the vehicle 2. Each external sensor 40 is composed of an individual type of device, such as LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), a camera, or an imaging radar. As shown in FIG. 3, each external sensor 40 has a sensing area As that senses through its corresponding sensor surface 33 and is set to a range in the external world of the vehicle 2 according to its placement position.

As shown in FIG. 1, the cleaning system 5 includes multiple cleaning nozzles 50. As shown in FIGS. 4 and 7, each cleaning nozzle 50 is held outside the housing 3 at a position corresponding to an individual external sensor 40 and sensor surface 33. Each cleaning nozzle 50 individually cleans the corresponding sensor surface 33 by spraying cleaning fluid.

Each cleaning nozzle 50 in the first embodiment sprays cleaning fluid in a range that spreads from its respective placement position toward the sensor surface 33, centered on a virtual spray axis Li. In particular, each cleaning nozzle 50 in the first embodiment has a fixed spray direction Ni, which is the virtual direction of the spray axis Li from its respective placement position toward the sensor surface 33. Furthermore, each cleaning nozzle 50 in the first embodiment sprays a cleaning gas, such as air, as the cleaning fluid toward the sensor surface 33.

As shown in Figures 1 and 4 to 6, one cleaning nozzle 50 may be provided corresponding to each external sensor 40. Alternatively, as shown in Figures 7 to 9, multiple cleaning nozzles 50 may be provided corresponding to each external sensor 40. In particular, when multiple cleaning nozzles 50 are provided corresponding to each external sensor 40, the cleaning nozzles 50 may be arranged in parallel with the injection axes Li spaced apart from each other as shown in Figure 7, so that the cleaning ranges on each sensor surface 33 are shifted from each other. However, in this case, the cleaning ranges of the cleaning nozzles 50 arranged in parallel may be shifted with the edges overlapping each other.

The control device 1 shown in Figs. 1 and 2 is connected to the sensor system 4 and the cleaning system 5 as shown in Figs. 5, 6, 8, and 9 via at least one of a LAN (Local Area Network), a wire harness, and an internal bus. Here, as shown in Figs. 5 and 8, the control device 1 may be provided individually for each pair of an external sensor 40 and a cleaning nozzle 50. Alternatively, as shown in Figs. 6 and 9, the control device 1 may be provided commonly for multiple pairs of an external sensor 40 and a cleaning nozzle 50.

As shown in FIG. 1, the control device 1 includes at least one dedicated computer. The dedicated computer constituting the control device 1 may be a driving control ECU that controls the autonomous driving mode in cooperation with an ECU (Electronic Control Unit) in the vehicle 2. The dedicated computer constituting the control device 1 may be a locator ECU that estimates state quantities of the vehicle 2, including its own position. The dedicated computer constituting the control device 1 may be a navigation ECU that navigates the driving route of the vehicle 2.

The control device 1 is configured to include such a dedicated computer, and has at least one memory 10 and one processor 12. The memory 10 is at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium, that non-temporarily stores computer-readable programs and data. The processor 12 includes at least one type of core, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a RISC (Reduced Instruction Set Computer)-CPU, a DFP (Data Flow Processor), or a GSP (Graph Streaming Processor).

In the control device 1, the processor 12 executes a number of instructions contained in a control program stored in the memory 10. This causes the control device 1 to construct a number of functional blocks for controlling the sensor system 4 and the cleaning system 5. The multiple functional blocks constructed by the control device 1 include an injection control block 100, a priority assignment block 120, and an operation control block 140, as shown in Figures 5, 6, 8, and 9.

The control method in which the control device 1 controls the sensor system 4 and the cleaning system 5 by cooperation of these blocks 100, 120, and 140 is executed according to the control flow shown in Figure 10. This control flow is repeatedly executed for each control period ωc that controls the sensor system 4 and the cleaning system 5 while the vehicle 2 is running. Note that each "S" in the control flow represents multiple steps that are executed by multiple commands included in the control program. Also, in order to make the explanation of the control flow easier to understand, the configuration examples shown in Figures 1, 4, and 5 are explained as representative examples, and explanations of other configuration examples are omitted.

In S10 of the control flow shown in FIG. 10, the injection control block 100 in FIG. 5 determines whether or not the cleaning conditions for performing cleaning of the sensor surface 33 of the external sensor 40 are met. In this case, the cleaning conditions are defined as conditions that require cleaning of the sensor surface 33 with respect to at least one of the following conditions of interest: the weather conditions in the driving environment around the vehicle 2, the dirt state of the sensor surface 33, and the driving state of the vehicle 2. Here, the state of interest may be recognized based on at least one of the following: detection data from the sensor system 4 of the vehicle 2 including the external sensor 40, and received data from outside the vehicle 2. If the cleaning conditions are met in S10, the control flow proceeds to S11.

In S20, the injection control block 100 controls the injection of the cleaning fluid from the cleaning nozzle 50 whose injection direction Ni is fixed in the current control cycle ωc, particularly the injection of the cleaning gas in the first embodiment. At this time, the injection start timing ts from the cleaning nozzle 50 shown in Figures 5 and 11 is set to a predetermined timing after a set time including the execution time of S10 has elapsed from the start timing (0) of the control cycle ωc. In addition, the injection end timing te from the cleaning nozzle 50 shown in Figures 5 and 11 is set to a timing after the elapse of the cleaning period ΔT during which the cleaning is actually performed from the injection start timing ts. Here, the cleaning period ΔT may be set to a fixed value or may be set to a variable value based on the cleaning conditions, etc., that are determined to be satisfied in S10. Furthermore, the injection amount Qi from the cleaning nozzle 50 shown in Figure 5 may be set to a fixed value or may be set to a variable value based on the cleaning conditions, etc., that are determined to be satisfied in S10.

In S30 of the control flow shown in FIG. 10, the priority assignment block 120 in FIG. 5 executes a priority assignment process. Specifically, in the priority assignment process of S30, the priority assignment block 120 assigns a priority Pi to each pixel region Dp shown in FIG. 12 in the image data Di acquired by the external sensor 40 through the sensor surface 33 for each frame Fi within the current control cycle ωc. Here, a pixel region Dp is composed of individual pixels or groups of pixels that make up the image data Di. Note that in FIG. 12, only some of the pixel regions Dp are labeled with a symbol.

In the priority assignment process of S30, the priority Pi is assigned based on the elapsed time Tc in the control period ωc as shown in FIG. 11 and the relative position Xp on the sensor surface 33 with respect to the cleaning nozzle 50 as shown in FIG. 12. In particular, the assignment of the priority Pi in the first embodiment is based on a first reliability distribution C1 that shows the reliability R1 of cleaning for each elapsed time Tc corresponding to each frame Fi as shown in FIG. 11, and a second reliability distribution C2 that shows the reliability R2 of cleaning for each relative position Xp corresponding to each pixel region Dp as shown in FIG. 12. Here, the reliability R1 and R2 of cleaning can be said to be the cleanliness of the sensor surface 33 by cleaning in the first embodiment. Also, the reliability R1 and R2 can be said to be the goodness of visibility of the external sensor 40 through the sensor surface 33 in the first embodiment. The reliability of cleaning is hereinafter referred to as cleaning reliability.

As shown in FIG. 11, the first reliability distribution C1 is defined by a reliability function so as to give the time progression of the cleaning reliability R1 according to the change in the elapsed time Tc corresponding to the change in frame Fi. ωc in the first reliability distribution C1 is a fixed variable parameter that defines the current control cycle in the reliability function. te in the first reliability distribution C1 is a fixed or variable variable parameter that defines the jetting end timing set for the current control cycle ωc in the reliability function.

In the first reliability distribution C1, cu is a coefficient parameter of the reliability function that defines the graph shape in FIG. 11 from the injection start timing ts to the injection end timing te during the current control cycle ωc. In particular, the coefficient parameter cu in the first embodiment is specified to give a higher cleaning reliability R1 as the elapsed time Tc from the injection start timing ts increases. This specification is based on the fact that the cleanliness of the sensor surface 33 gradually increases as the time Tc elapses during the injection of cleaning gas.

In the first reliability distribution C1, cd is a coefficient parameter of a reliability function that defines the graph shape of FIG. 11 from the injection end timing te to the completion timing of the current control cycle ωc. In particular, the coefficient parameter cd in the first embodiment is specified so as to give a lower cleaning reliability R1 as the elapsed time Tc from the injection end timing te increases. This specification is based on the fact that, during startup of the vehicle 2, the more time Tc has passed since the completion of the cleaning gas injection, the more likely it is that the cleanliness of the sensor surface 33 will decrease due to, for example, the adhesion of rain, snow, or dust.

One or both of these coefficient parameters cu and cd may be variably set based on at least one of the weather conditions in the driving environment around the vehicle 2, the driving state of the vehicle 2, and the spray characteristics from the cleaning nozzle 50. Here, the weather conditions include at least one of, for example, the presence or absence of rainfall, the amount of rainfall during rainfall, the presence or absence of snowfall, the amount of snowfall during snowfall, the presence or absence of wind, and the amount of wind during wind. The driving state includes at least one of, for example, speed, acceleration, and yaw rate. The spray characteristics include at least one of, for example, the above-mentioned timings ts, te, the cleaning period ΔT, and the spray amount Qi. As a specific example, the coefficient parameter cd from the injection end timing te to the completion timing of the control cycle ωc may be set so that the greater the amount of rainfall during rainfall, the greater the decay rate per hour of the cleaning reliability R1.

In the first reliability distribution C1, rmax and rmin are variable parameters that respectively define the upper and lower limit values of the cleaning reliability R1 in the reliability function. One or both of these upper limit values rmax and lower limit values rmin may be variably set according to at least one of the weather conditions in the driving environment around the vehicle 2, the driving conditions of the vehicle 2, and the spray characteristics from the cleaning nozzle 50. As a specific example, the upper limit value rmax of the cleaning reliability R1 may be set to be smaller the more the amount of rainfall during rainfall. On the other hand, the lower limit value rmin of the cleaning reliability R1 may be set to be larger the more the amount of rainfall during rainfall.

12, the second reliability distribution C2 gives a cleaning reliability R2 for each relative position Xp with respect to the cleaning nozzle 50 corresponding to each pixel region Dp on the sensor surface 33. In particular, the second reliability distribution C2 of the first embodiment is defined so that the further the relative position Xp is from the projection line Lp obtained by projecting the injection axis Li of the cleaning nozzle 50 onto the sensor surface 33, the lower the cleaning reliability R2 is given. This definition is based on the fact that the more the relative position Xp is away from the projection line Lp, the less the amount of cleaning gas that is the cleaning fluid reaches, and the lower the cleaning performance becomes. Here, the projection line Lp is defined as the intersection line with the sensor surface 33 on a virtual plane that includes the injection axis Li of the cleaning nozzle 50 and intersects the sensor surface 33 at a right angle (i.e., a plane perpendicular to the illustrated surface of FIG. 12). Note that FIG. 12 shows the pixel region Dp corresponding to the relative position Xp on the sensor surface 33 as a rectangular mesh that is typically superimposed on the sensor surface 33. Therefore, FIG. 12 is drawn so that the lighter the shading of the relative position Xp for each pixel region Dp, the lower the cleaning reliability R2.

The priority assignment block 120 in the priority assignment process of S30 assigns a priority Pi to the image data Di for each frame Fi and each pixel region Dp by combining the first reliability distribution C1 and the second reliability distribution C2 thus provided as shown in Fig. 13. At this time, the combination of the distributions C1 and C2 is performed by multiplying the cleaning reliability R1 provided by the first reliability distribution C1 for each frame Fi by the cleaning reliability R2 provided by the second reliability distribution C2 for each pixel region Dp, thereby acquiring a priority Pi that satisfies the formula 1. As a result, a priority Pi is assigned that decreases as the elapsed time Tc from the ejection end timing te increases, and that decreases as the relative position Xp is separated from the projection line Lp of the ejection axis Li on the sensor surface 33.

The priority assignment block 120 in the priority assignment process of S30 may store the image data Di to which the priority Pi has been assigned in the memory 10. The priority assignment block 120 in the priority assignment process of S30 may transmit the image data Di to which the priority Pi has been assigned to an external center capable of communicating with the vehicle 2, and store the image data Di in the external center.

In S40 of the control flow shown in FIG. 10, the driving control block 140 in FIG. 5 controls the autonomous driving mode of the vehicle 2 based on the image data Di to which the priority Pi has been assigned in S30. In this case, in the autonomous driving mode, the priority Pi may be used to recognize the driving environment around the vehicle 2. In the autonomous driving mode, the priority Pi may be used to plan a driving path including the route and trajectory of the vehicle 2. In the autonomous driving mode, the priority Pi may be used to monitor and determine the driving risk of the vehicle 2. In the autonomous driving mode, the priority Pi may be used to set the control parameters of the vehicle 2. In the autonomous driving mode, the priority Pi may be used to control the driving actuators of the vehicle 2 based on the control parameters. Note that the driving control block 140 in S40 may execute the control of the autonomous driving mode in cooperation with the ECU in the vehicle 2, or may execute the control of the autonomous driving mode independently.

When execution of S40 is completed, the current execution of the control flow ends as shown in FIG. 10. The current execution of the control flow also ends if the cleaning conditions are not met in S10 described above. If the cleaning conditions are not met in S10, the elapsed time from the previous cleaning may be monitored before the current execution of the control flow ends, and if the elapsed time exceeds or is equal to or greater than a set time, the fluid spray from the cleaning nozzle 50 may be controlled. In this case, S20, S30, and S40 may be executed, or may not be executed as in FIG. 10.

(Action and Effect)
The effects of the first embodiment described above will be described below.

In this way, in the first embodiment, the spray of cleaning fluid from the cleaning nozzle 50 is controlled in the control cycle ωc that controls the sensor system 4 and the cleaning system 5. Therefore, according to the first embodiment, the image data Di acquired through the sensor surface 33 for each frame Fi within the control cycle ωc is assigned a priority Pi for each pixel region Dp based on the elapsed time Tc in the control cycle ωc and the relative position Xp on the sensor surface 33 with respect to the cleaning nozzle 50. As a result, the difference in cleanliness that occurs on the sensor surface 33 depending on the elapsed time Tc and the relative position Xp can be reflected in the priority Pi for each frame Fi and each pixel region Dp, making it possible to ensure the reliability of the image data Di.

In the first embodiment, spraying from a cleaning nozzle 50 with a fixed spraying direction Ni is controlled. Thus, in the first embodiment, a priority Pi is assigned based on a first reliability distribution C1 representing cleaning reliability R1 for each elapsed time Tc corresponding to a frame Fi, and a second reliability distribution C2 representing cleaning reliability R2 for each relative position Xp corresponding to a pixel region Dp. This allows the cleaning reliability R1 for each elapsed time Tc and the cleaning reliability R2 for each relative position Xp to be accurately reflected in the priority Pi for each frame Fi corresponding to the elapsed time Tc and for each pixel region Dp corresponding to the relative position Xp. This makes it possible to increase the reliability of the image data Di.

According to the first embodiment, the first reliability distribution C1 for each elapsed time Tc is variably set based on at least one of the weather conditions in the driving environment around the vehicle 2, the driving conditions of the vehicle 2, and the spray characteristics from the cleaning nozzle 50. As a result, the cleaning reliability R1, which increases or decreases depending on factors that change the cleanliness of the sensor surface 33 over time, can be reflected in the priority Pi for each frame Fi corresponding to the elapsed time Tc. This makes it possible to increase the reliability of the image data Di.

In the first embodiment, the injection end timing te at which the injection of the cleaning gas, which is the cleaning fluid, ends is controlled for each control period ωc. Thus, the image data Di according to the first embodiment is assigned a priority Pi that decreases as the elapsed time Tc from the injection end timing te increases, and decreases as the relative position Xp moves away from the projection line Lp of the injection axis Li of the cleaning nozzle 50 projected onto the sensor surface 33. This allows the priority Pi to be accurately assigned for each frame Fi corresponding to the elapsed time Tc and for each pixel region Dp corresponding to the relative position Xp in accordance with the cleaning characteristics of the cleaning gas. Therefore, it is possible to increase the reliability of the image data Di.

According to the first embodiment, image data Di to which a priority level Pi has been assigned is stored in memory 10 as a storage medium. As a result, image data Di read from memory 10 can be used as data whose reliability has been ensured by the assignment of priority level Pi.

According to the first embodiment, the autonomous driving mode of the vehicle 2 is controlled based on the image data Di to which a priority Pi has been assigned. This makes it possible to realize control of the autonomous driving mode according to the priority Pi for each frame Fi and each pixel region Dp, thereby increasing the reliability of the control as well as the reliability of the image data Di.

Second Embodiment
The second embodiment is a modification of the first embodiment.

As shown in FIG. 14, in the control flow of the second embodiment, S250 is executed between S30 and S40. In this S250, the priority assignment block 120 selects a pixel region Dp in which the priority Pi assigned by S30 is within the allowable range as a valid pixel region in the image data Di. This selection may be realized by invalidating a pixel region Dp in which the priority Pi is outside the allowable range in the image data Di, for example, by masking. In addition, at this time, among the pixel regions Dp that are outside the allowable range, a more accurate invalidation may be realized by limiting the pixel regions Dp that correspond to the adhering portions of raindrops detected on the sensor surface 33 by, for example, image processing of the image data Di. In S40 of the second embodiment following such S250, it is possible to improve the control reliability of the automatic driving mode based on the pixel values of the valid pixel region in the image data Di to which the priority Pi is assigned.

Third Embodiment
The third embodiment is a modification of the first embodiment.

In the cleaning system 3005 of the third embodiment shown in Figures 15 to 17, cleaning nozzles 3050 are provided corresponding to each external sensor 40, and each is driven by an individual drive unit 3052. As shown in Figure 17, for each cleaning nozzle 3050, the extension direction of the injection axis Li from the support position by the drive unit 3052 toward the sensor surface 33 is defined as the injection direction Ni. Each drive unit 3052 swing-drives each individual cleaning nozzle 3050 within a drive angle range θ (see Figure 19 described below), thereby rotating and changing the injection direction Ni of the individual cleaning nozzle 3050 over time. Note that the cleaning system 3005 is configured similarly to the cleaning system 5 of the first embodiment in all respects except for the injection direction Ni.

In the third embodiment having such a configuration, the control device 1 may be provided individually for each set of the external sensor 40, the cleaning nozzle 3050, and the drive unit 3052, as shown in FIG. 15. Alternatively, the control device 1 of the third embodiment may be provided commonly for multiple sets of the external sensor 40, the cleaning nozzle 3050, and the drive unit 3052, as shown in FIG. 16.

As shown in FIG. 18, in the control flow of the third embodiment, S320 and S330 are executed in place of S20 and S30 in the first embodiment. In addition, in order to make the explanation of the control flow easier to understand, the configuration examples shown in FIGS. 15 and 17 are explained as representative examples, and explanations of other configuration examples are omitted.

In S320 of the control flow shown in FIG. 18, the injection control block 100 in FIG. 15 gradually rotates the injection direction Ni of the cleaning nozzle 3050 to one side of the drive angle range θ over the cleaning period ΔT in the current control cycle ωc. At this time, the angular velocity of the rotational change may be controlled to be constant over time, or may be variably controlled depending on, for example, the cleanliness of the sensor surface 33. S320 is executed in accordance with S20 in all respects other than the injection direction Ni. However, the cleaning period ΔT may be set to, for example, a period longer than that of the first embodiment.

In S330 of the control flow shown in FIG. 18, the priority assignment block 120 in FIG. 15 executes a priority assignment process different from S30 in the first embodiment, depending on the difference in the injection direction Ni. In the priority process of S330, the priority assignment block 120 assigns a priority Pi based on a reliability distribution C representing the cleaning reliability R, as shown in FIG. 19. Here, the cleaning reliability R represented by the reliability distribution C is the cleanliness of the sensor surface 33 due to cleaning, and can be said to be the degree of visibility of the external sensor 40 through the sensor surface 33.

The reliability distribution C of the third embodiment gives a cleaning reliability R for each relative position Xp on the sensor surface 33 with respect to the cleaning nozzle 3050, which changes as shown in Figures 19 and 20 according to the injection direction Ni for each elapsed time Tc in the current control cycle ωc. In this case, the cleaning reliability R is defined so that the more the relative position Xp moves away from the projection line Lp in the opposite direction to the direction of change each time the injection direction Ni changes by a predetermined swing angle ψ in the drive angle range θ, the lower the cleaning reliability R becomes. In this definition, similar to the first embodiment, the projection line Lp is defined as the line of intersection with the sensor surface 33 on a virtual plane that includes the injection axis Li of the cleaning nozzle 3050 and intersects the sensor surface 33 at a right angle (i.e., a plane perpendicular to the illustrated plane in Figures 19 and 20).

In this third embodiment, the reliability distribution C also provides a cleaning reliability R that is lower on the opposite side of the change direction as the elapsed time Tc increases from each injection end timing, which is defined as the timing when the injection direction Ni changes by the swing angle ψ. Furthermore, on the side of the change direction before cleaning, the reliability distribution C is specified so that the lower the cleaning reliability R is provided as the elapsed time Tc increases from each injection end timing for each swing angle ψ in the previous cleaning. Note that Figures 19 and 20 show pixel areas Dp corresponding to relative positions Xp on the sensor surface 33 as a rectangular mesh that is superimposed on the sensor surface 33. Thus, Figures 19 and 20 are drawn so that the lighter the relative positions Xp are for each pixel area Dp, the lower the cleaning reliability R is.

The priority assignment block 120 in the priority assignment process of S330 converts the cleaning reliability R given by the reliability distribution C into a priority Pi and assigns it to the image data Di. At this time, the cleaning reliability R represented by the reliability distribution C for each elapsed time Tc and each relative position Xp is converted into a priority Pi for each frame Fi and each pixel area Dp corresponding to each of the elapsed times Tc and the relative positions Xp. As a result, the priority Pi is assigned so that it becomes lower as the elapsed time Tc increases from the injection end timing each time the injection direction Ni changes by the swing angle ψ, and in particular, the priority Pi becomes lower as the relative position Xp moves away from the projection line Lp of the injection axis Li on the opposite side of the change direction on the sensor surface 33. Note that, like S30 in the first embodiment, the priority assignment block 120 in the priority assignment process of S330 may store the image data Di to which the priority Pi has been assigned in the memory 10, or may transmit it to an external center capable of communicating with the vehicle 2 for storage.

In this way, in the third embodiment, spraying from the cleaning nozzle 3050, whose spray direction Ni changes, is controlled. Therefore, according to the third embodiment, a priority Pi is assigned based on a reliability distribution C that represents the cleaning reliability R at each relative position Xp according to the spray direction Ni for each elapsed time Tc. As a result, the cleaning reliability R for each elapsed time Tc and each relative position Xp can be accurately reflected in the priority Pi for each frame Fi corresponding to the elapsed time Tc and each pixel region Dp corresponding to the relative position Xp. Therefore, it is possible to increase the reliability of the image data Di.

Furthermore, in the third embodiment, the injection end timing at which the injection of the cleaning gas, which is the cleaning fluid, ends for each predetermined change in the injection direction Ni is controlled for each control period ωc. Therefore, the image data Di according to the third embodiment is assigned a priority Pi that decreases as the elapsed time Tc increases from the injection end timing for each predetermined change in the injection direction Ni, and decreases as the relative position Xp moves away from the projection line Lp of the injection axis Li of the cleaning nozzle 3050 projected onto the sensor surface 33. This makes it possible to accurately assign a priority Pi for each frame Fi corresponding to the elapsed time Tc and for each pixel region Dp corresponding to the relative position Xp in accordance with the cleaning characteristics of the cleaning gas. Therefore, it is possible to increase the reliability of the image data Di.

(Fourth embodiment)
The fourth embodiment is a modification of the first embodiment.

As shown in FIG. 21, in the control flow of the fourth embodiment, S420 is executed in place of S20 in the first embodiment. In this S420, the injection control block 100 controls the injection of a cleaning fluid, such as water or a surfactant, from the cleaning nozzle 50 in the current control cycle ωc, into a spray shape centered on the injection axis Li. S420 is executed in accordance with S20 except for the type of cleaning fluid to be injected.

Furthermore, in the control flow of the fourth embodiment, a priority assignment process is executed by S430 instead of S30 of the first embodiment. In this priority assignment process of S430, the priority assignment block 120 specifies the coefficient parameter cu shown in FIG. 22 of the reliability function that gives the first reliability distribution C1 so that the lower the cleaning reliability R1 is given as the elapsed time Tc from the injection start timing ts increases. At the same time, the priority assignment block 120 specifies the coefficient parameter cd shown in FIG. 22 of the reliability function that gives the first reliability distribution C1 so that the higher the cleaning reliability R1 is given as the elapsed time Tc from the injection end timing te increases. These specifications are based on the fact that during and immediately after the injection of the cleaning liquid, the visibility of the external sensor 40 through the sensor surface 33 is deteriorated by the cleaning liquid adhering to the sensor surface 33. That is, in the fourth embodiment, the cleaning reliability R1 can be said to be the degree of visibility through the sensor surface 33.

S430 is executed in accordance with S30 except for the specification of the coefficient parameters cu and cd that give the first reliability distribution C1. Here, in the priority assignment process of S430, the priority assignment block 120 also specifies the second reliability distribution C2 so that the lower the cleaning reliability R2, the further the relative position Xp is from the projection line Lp of the injection axis Li of the cleaning nozzle 50 projected onto the sensor surface 33. This specification is based on the fact that even if the cleaning fluid is a cleaning liquid, the further it is from the projection line Lp, the smaller the amount of the cleaning fluid that reaches the sensor surface 33 and the lower the cleaning performance. That is, in the fourth embodiment, the cleaning reliability R2 is the cleanliness of the sensor surface 33 due to cleaning, and can be said to be the degree of visibility of the external sensor 40 through the sensor surface 33.

The result of the priority assignment process in S430 of the fourth embodiment is to assign a priority Pi to the image data Di that increases as the elapsed time Tc from the injection end timing te increases and decreases as the relative position Xp moves away from the projection line Lp. Note that the priority assignment block 120 in the priority assignment process in S430 may store the image data Di to which the priority Pi has been assigned in the memory 10, as in S30 of the first embodiment, or may transmit the image data Di to an external center capable of communicating with the vehicle 2 for storage.

In this fourth embodiment, the injection end timing te at which the injection of the cleaning fluid, which is the cleaning fluid, ends is controlled for each control period ωc. Thus, the image data Di according to the fourth embodiment is assigned a priority Pi that increases as the elapsed time Tc from the injection end timing te increases, and decreases as the relative position Xp moves away from the projection line Lp of the injection axis Li projected onto the sensor surface 33. This allows the priority Pi to be accurately assigned for each frame Fi corresponding to the elapsed time Tc and for each pixel region Dp corresponding to the relative position Xp in accordance with the cleaning characteristics of the cleaning fluid. This makes it possible to increase the reliability of the image data Di.

Other Embodiments
Although several embodiments have been described above, the present disclosure should not be construed as being limited to those embodiments, and can be applied to various embodiments and combinations within the scope not departing from the gist of the present disclosure.

In a modified example, the dedicated computer constituting the control device 1 may be a computer other than the vehicle 2 that constitutes an external center or a mobile terminal capable of communicating with the vehicle 2. In a modified example, the dedicated computer constituting the control device 1 may have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is at least one of the following types: ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SOC (System on a Chip), PGA (Programmable Gate Array), and CPLD (Complex Programmable Logic Device). In addition, such a digital circuit may have a memory that stores a program.

In a modified example, the sensor surfaces 33 of adjacent external sensors 40 may be formed by a common sensor cover 32. In a modified example, the sensor cover 32 that forms the sensor surface 33 may be provided on the external sensor 40 itself. In a modified example, the sensor surface 33 may be formed by an optical member, such as a lens, of the external sensor 40 itself.

In a modified example, the features described in the second embodiment may be applied in combination with each of the third and fourth embodiments. In a modified example, S10 by the operation control block 140 may be omitted. In a modified example, S40 by the operation control block 140 may be omitted.

In the modified example, the vehicle to which the control device 1 is applied may be, for example, a drone whose running on a road can be remotely controlled from an external center. In addition to the forms described so far, the above-mentioned embodiments and modified examples may be implemented as a semiconductor device (e.g., a semiconductor chip, etc.) having at least one processor 12 and at least one memory 10.

1: Control device, 2: Vehicle, 4: Sensor system, 5,3005: Cleaning system, 33: Sensor surface, 50,3050: Cleaning nozzle, 10: Memory, 12: Processor, As: Sensing area, C: Reliability distribution, C1: First reliability distribution, C2: Second reliability distribution, Di: Image data, Dp: Pixel area, Fi: Frame, Li: Injection axis, Lp: Projection line, Ni: Injection direction, Pi: Priority, R, R1, R2: Reliability, Tc: Elapsed time, Xp: Relative position, ωc: Control period

Claims (11)

A control device having a processor (12) and controlling a sensor system (4) in which a sensing area (As) is set through a sensor surface (33) exposed to the outside of a vehicle (2), and a cleaning system (5, 3005) that cleans the sensor surface by spraying a cleaning fluid from a cleaning nozzle (50, 3050),
The processor,
controlling the ejection of the cleaning fluid from the cleaning nozzle in a control period (ωc) for controlling the sensor system and the cleaning system;
and assigning a priority (Pi) to each pixel region (Dp) in image data (Di) acquired through the sensor surface for each frame (Fi) within the control cycle based on an elapsed time (Tc) in the control cycle and a relative position (Xp) with respect to the cleaning nozzle on the sensor surface. Controlling the spray from the cleaning nozzle includes:
controlling the injection from said cleaning nozzle (50) whose injection direction (Ni) is fixed;
The assigning of priority includes:
The control device according to claim 1, further comprising: assigning the priority based on a first reliability distribution (C1) representing a reliability (R1) for cleaning for each elapsed time corresponding to the frame, and a second reliability distribution (C2) representing a reliability (R2) for cleaning for each relative position corresponding to the pixel area. The assigning of priority includes:
The control device according to claim 2, further comprising: variably setting the first reliability distribution for each elapsed time based on at least one of weather conditions in the driving environment around the vehicle, a driving condition of the vehicle, and spray characteristics from the cleaning nozzle. Controlling the spray from the cleaning nozzle includes:
Controlling the jet from the cleaning nozzle (3050) whose jet direction (Ni) is changed;
The assigning of priority includes:
The control device according to claim 1 , further comprising: assigning the priority based on a reliability distribution (C) representing a reliability (R) of cleaning for each of the relative positions according to the spray direction for each of the elapsed times. Controlling the spray from the cleaning nozzle includes:
controlling a jetting end timing at which jetting of the cleaning gas as the cleaning fluid ends for each of the control cycles;
The assigning of priority includes:
The control device according to any one of claims 1 to 4, further comprising: assigning a priority that becomes lower as the elapsed time from the end of spray increases, and that becomes lower as the relative position becomes farther away from a projection line (Lp) of the spray axis (Li) of the cleaning nozzle onto the sensor surface. Controlling the spray from the cleaning nozzle includes:
controlling a jetting end timing at which jetting of the cleaning fluid as the cleaning fluid ends for each of the control cycles;
The assigning of priority includes:
The control device according to any one of claims 1 to 3, further comprising: assigning a priority that increases as the elapsed time from the end of spray increases, and that decreases as the relative position becomes farther away from a projection line (Lp) of the spray axis (Li) of the cleaning nozzle onto the sensor surface. The processor,
The control device according to any one of claims 1 to 6, further configured to select, in the image data, the pixel regions in which the assigned priority falls within an acceptable range. A storage medium (10),
The processor,
The control device according to any one of claims 1 to 7, further configured to store the image data to which the priority has been assigned in the storage medium. The processor,
The control device according to any one of claims 1 to 8, further configured to control an autonomous driving mode of the vehicle based on the image data to which the priority has been assigned. A control method executed by a processor (12) for controlling a sensor system (4) in which a sensing area (As) is set through a sensor surface (33) of a vehicle (2) exposed to the outside world, and a cleaning system (5, 3005) that cleans the sensor surface by spraying a cleaning fluid from a cleaning nozzle (50, 3050), comprising:
controlling the ejection of the cleaning fluid from the cleaning nozzle in a control period (ωc) for controlling the sensor system and the cleaning system;
and assigning a priority (Pi) to each pixel region (Dp) in image data (Di) acquired through the sensor surface for each frame (Fi) within the control period, based on an elapsed time (Tc) in the control period and a relative position (Xp) with respect to the cleaning nozzle on the sensor surface. A control program stored in a storage medium (10) for controlling a sensor system (4) in which a sensing area (As) is set through a sensor surface (33) exposed to the outside of a vehicle (2) and a cleaning system (5, 3005) that cleans the sensor surface by spraying a cleaning fluid from a cleaning nozzle (50, 3050), the control program including instructions to be executed by a processor (12),
The instruction:
controlling the ejection of the cleaning fluid from the cleaning nozzle in a control period (ωc) for controlling the sensor system and the cleaning system;
and assigning a priority (Pi) to each pixel area (Dp) in image data (Di) acquired through the sensor surface for each frame (Fi) within the control cycle based on an elapsed time (Tc) in the control cycle and a relative position (Xp) with respect to the cleaning nozzle on the sensor surface.

Images