JP7109528B2 - VEHICLE IMAGE INFORMATION PROCESSING DEVICE, VEHICLE IMAGE INFORMATION TRANSMITTER AND IMAGE INFORMATION PROCESSING METHOD - Google Patents

Description

The present application relates to an in-vehicle image information processing device, an in-vehicle image information transmission device, and an image information processing method.

A device has been proposed that captures an image around the vehicle with a camera mounted on the vehicle and recognizes objects around the vehicle. In order to recognize what the object photographed by the camera is, a high-speed in-vehicle computer with large capacity memory and high processing power is required to process a large amount of information for image processing and image analysis. . In order to perform image processing and image analysis on an in-vehicle computer, an expensive in-vehicle computer with a large memory capacity and high processing capability is required, which is a factor driving up costs.

On the other hand, there has been proposed an in-vehicle device that captures an image around the vehicle and wirelessly transmits the captured image information to an external information processing device (image information processing center) (for example, Patent Document 1). The external information processing device performs image processing using a high-performance computer. In this way, since it is not necessary to use a high-performance and expensive computer for the on-vehicle device, it is possible to suppress an increase in the cost of the on-vehicle device.

JP 2014-81831 A

In the technique proposed in Patent Document 1, information about an image around the vehicle captured by a camera mounted on the vehicle is constantly transmitted to an external information processing device. For this reason, a large amount of information is transmitted by wireless communication, and communication is congested. It has been proposed to remove noise and compress image information according to the degree of communication congestion, and to adjust the frequency of transmission. However, since the image information is periodically transmitted regardless of the content of the image, the amount of information to be transmitted increases. When image information is transmitted from many vehicles to the external information processing device, communication may be congested. If the communication is congested, the communication may be interrupted, and the reliability of the transmission of image information will be impaired.

The present application has been disclosed in order to solve the above-described problems. An object of the present invention is to obtain an in-vehicle image information processing apparatus that generates an image.

A further object of the present application is to provide an in-vehicle image information transmission device that transmits appropriate image information around the vehicle to an external information processing device while reducing the amount of information to be transmitted.

Another object of the present application is to provide an image information processing method for generating transmission information for transmitting suitable image information of the surroundings of a vehicle to an external information processing device while reducing the amount of information to be transmitted.

The in-vehicle image information processing device according to the present application is
an image input unit that receives image information from a camera that is provided in the vehicle and captures an image of the surroundings of the vehicle;
an abnormal area detection unit that detects an abnormal area from image information input to the image input unit;
an importance calculation unit that calculates the importance of the abnormal area based on the ratio of the abnormal area detected by the abnormal area detection unit to the image information ;
a storage unit for storing important image information including image information in which the degree of importance and the abnormal region for which the degree of importance is calculated is detected;
a selection unit that selects the most important image information having the highest degree of importance from a plurality of pieces of important image information stored in the storage unit;
A transmission information generation section is provided for generating transmission information based on the most important image information selected by the selection section.

Furthermore, the in-vehicle image information transmission device according to the present application is
camera and
a transmitter that transmits transmission information to the outside of the vehicle;
It is equipped with an in-vehicle image information processing device.

Further, the image information processing method according to the present application includes:
a step of inputting image information of an image captured by a camera provided in the vehicle for capturing the surroundings of the vehicle into an image input unit;
a step in which an abnormal area detection unit detects an abnormal area from image information input to the image input unit;
A step of calculating the importance of the abnormal region by the importance calculation unit based on the ratio of the abnormal region detected by the abnormal region detection unit to the image information ;
a step of storing, by a storage unit, important image information including the importance and the image information of the detected abnormal region for which the importance has been calculated;
a step of selecting the most important image information having the highest degree of importance from a plurality of pieces of important image information stored in the storage unit;
and a step of generating transmission information based on the most important image information selected by the selection unit by the transmission information generation unit.

According to the in-vehicle image information processing device, the in-vehicle image information processing device, and the image information processing method according to the present application, it is possible to reduce the amount of information to be transmitted and generate transmission information of image information around the vehicle suitable for an external information processing device. Therefore, it is possible to suppress communication congestion and transmit image information with high reliability.

1 is a configuration diagram of an in-vehicle image information processing apparatus according to Embodiment 1; FIG. 1 is a block diagram of an in-vehicle image information processing device according to Embodiment 1; FIG. 1 is a diagram showing a hardware configuration of an in-vehicle image information processing apparatus according to Embodiment 1; FIG. 4 is a diagram showing captured image information of the in-vehicle image information processing device according to Embodiment 1. FIG. 4 is a first flowchart showing image information processing of the in-vehicle image information processing device according to Embodiment 1; 7 is a second flow chart showing image information processing of the in-vehicle image information processing device according to Embodiment 1. FIG. 2 is a block diagram of an in-vehicle image information processing device according to Embodiment 2. FIG. FIG. 10 is a diagram showing captured image information of the vehicle-mounted image information processing device according to Embodiment 2; 9 is a first flow chart showing image information processing of the vehicle-mounted image information processing device according to Embodiment 2; 9 is a second flow chart showing image information processing of the in-vehicle image information processing device according to Embodiment 2;

Embodiments of an on-vehicle image information processing device, an on-vehicle image information transmission device, and an image information processing method according to the present application will be described below with reference to the drawings.

1. Embodiment 1
<Configuration>
FIG. 1 is a configuration diagram of an in-vehicle image information processing device 220 according to Embodiment 1. As shown in FIG. The vehicle 101 includes a camera 102 for capturing images around the vehicle, a positioning sensor 105 for measuring the position of the vehicle 101, a transmitter 104 for transmitting transmission information based on image information to the external information processing device 300, and an external information processing device. A receiver 107 for receiving an image analysis result from 300 and an in-vehicle image information processing device 220 are mounted. An in-vehicle image information transmitting device 200 is composed of an in-vehicle image information processing device 220 , a camera 102 , a positioning sensor 105 , a transmitter 104 and a receiver 107 . Here, receiver 107 may be omitted.

The transmission information generated by the in-vehicle image information processing device 220 is transmitted to the external information processing device 300 by the transmitter 104 of the in-vehicle image information transmission device 200 . When a road abnormality 106 or the like is found while the vehicle 101 is traveling, image information and position information of the road abnormality 106 or the like are transmitted from the transmitter 104 to the external information processing device 300 . If the road abnormality 106 or the like is not found, no transmission is performed from the transmitter 104, so an increase in communication traffic of the wireless network can be suppressed.

The external information processing device 300 receives the transmission information transmitted from the transmitter 104 of the in-vehicle image information transmission device 200 by the reception unit 302 , and identifies the photographed object by the image information analysis unit 303 . The display device 301 displays an image of an identified object or an image of an unidentified object. Confirmation of identified objects or re-identification of objects may be performed by surveillance personnel as needed.

A shooting range 109 indicates a range shot by the camera 102 . It is sufficient if the road environment around the vehicle 101 is captured in a part of the imaging range 109, and it does not have to be in front of the vehicle 101. FIG. For example, the sides and rear of the vehicle 101 may be photographed.

A road abnormality 106 indicates a state in which the road is depressed and dented. From the image information captured by the camera, not only road depressions, but also road cracks, bumps, falling objects, parallel running vehicles, stopped vehicles, pedestrians, damaged road signs, etc. may be identified. The identification result of the object identified by the image information analysis unit 303 and the received image information are stored in the external storage unit 304 .

Depending on the object identified by the image information analysis unit 303 of the external information processing device 300 or by the monitoring personnel, the road administrator can go to the site and treat the object. By repairing roads for depressions, cracks, and bumps, road conditions can be restored to normal and stable traffic volumes can be maintained. In addition, by taking appropriate measures for fallen objects, stopped vehicles, etc., the road conditions can be restored to normal by removing the obstacles, and a stable traffic volume can be maintained. Furthermore, by repairing damaged road signs, the road signs can be restored to normal, leading to the maintenance of stable traffic volume.

The object identification result may be transmitted from the transmission unit 305 to the vehicle. The object identification result can be used for driving assistance of the vehicle. It is possible to select an operation of the vehicle such as an object avoidance operation, a stopping operation, or a passing operation by the vehicle according to the object identification result.

The positioning sensor 105 can utilize a GNSS (Global Navigation Satellite System) receiver. This makes it possible to grasp the position of the vehicle. As the positioning sensor 105, instead of GNSS, a DMI (Distance Measuring Instrument), an IMU (Inertial Measurement Unit), an acceleration sensor, a geomagnetic measuring instrument, etc. are used to independently grasp the position of the vehicle. You may Alternatively, the position may be grasped by reading electromagnetic markers or signposts installed on the road.

<Functional block>
FIG. 2 shows a functional block diagram of the in-vehicle image information processing device 220 according to the first embodiment. Functional blocks of the in-vehicle image information transmission device 200 are also shown at the same time. Image information of an image captured by the camera 102 installed on the vehicle 101 is transmitted to the abnormal area detection unit 201 and the storage unit 203 via the image input unit 208 .

The abnormal area detection unit 201 detects the presence or absence of an abnormal area from the image information, and if there is an abnormal area, calculates the degree of importance. When an object different from the normal road is detected in the image information, it is judged as an abnormal area. The abnormal area detection unit 201 only needs to detect that there is some kind of object in the abnormal area. What the object is is identified by the image information analysis unit 303 of the external information processing device 300 or by the monitoring personnel. The storage unit 203 stores the importance of the abnormal region calculated by the importance calculation unit 202 and the image information including the abnormal region for which the importance is calculated. At this time, the position information detected by the positioning sensor 105 is transmitted to the storage unit 203 via the position input unit 211, and the importance level, the image information, and the position information where the image was taken are stored in the storage unit 203 as important image information. remembered.

The selection unit 204 reads a predetermined plurality of pieces of important image information stored in the storage unit 203, selects the important image information with the highest degree of importance, and sets it as the most important image information. Also, the estimation unit 209 estimates the importance of important image information that can be acquired in the future from the most recent continuous important image information.

If the estimated importance is higher than the importance of the most important image information, the processing of the most important image information by the processing unit 206 is stopped, and the transmission information generation unit 207 does not generate transmission information. After waiting for acquisition of important image information with a higher degree of importance in the future, when new most important image information is stored, the most important image information is processed to generate transmission information.

In order to reduce transmission information, the processing unit 206 can extract only the abnormal region from the image information of the most important image information and use only the image of the abnormal region as transmission information. Alternatively, the transmission information can be data in which the image outside the abnormal region is filled with a single color (for example, white).

If the importance estimated by the estimation unit 209 is not higher than the importance of the most important image information, the transmission information generation unit 207 generates transmission information by processing the most important image information. The transmission information generated by the transmission information generation unit 207 is transmitted to the external information processing device 300 by the transmitter 104 . The receiver 107 receives the identification result of the object identified by the external information processing device 300 .

<Hardware configuration of in-vehicle information processing device>
FIG. 3 shows the hardware configuration of the vehicle-mounted image information processing apparatuses 220 and 220a according to the first embodiment. The in-vehicle image information processing devices 220 and 220 a are processing devices that process image information obtained from the camera 102 provided in the vehicle 101 . Each function of the in-vehicle image information processing devices 220 and 220a is realized by a processing circuit provided in the in-vehicle image information processing devices 220 and 220a. Specifically, the in-vehicle image information processing devices 220 and 220a include, as processing circuits, an arithmetic processing unit 90 (computer) such as a CPU (Central Processing Unit), a storage device 91 for exchanging data with the arithmetic processing unit 90, an arithmetic An input circuit 92 for inputting an external signal to the processing device 90 and an output circuit 93 for outputting a signal from the arithmetic processing device 90 to the outside are provided.

As the arithmetic processing unit 90, an ASIC (Application Specific Integrated Circuit), an IC (Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), various logic circuits, various signal processing circuits, and the like are provided. may Further, as the arithmetic processing unit 90, a plurality of units of the same type or different types may be provided, and each process may be shared and executed. As the storage device 91, a RAM (random access memory) configured so that data can be read and written from the arithmetic processing unit 90, a ROM (read only memory) configured so that data can be read from the arithmetic processing unit 90, and the like are provided. It is The input circuit 92 is connected to various sensors including the camera 102 , the positioning sensor 105 and the receiver 107 , switches, and communication lines. It is equipped with a D converter, a communication circuit, and the like. The output circuit 93 includes a communication line for outputting control signals from the arithmetic processing unit 90 to the transmitter 104 and various driving devices, a driving circuit, and the like.

Each function provided in the in-vehicle image information processing devices 220 and 220a is performed by the arithmetic processing device 90 executing software (program) stored in a storage device 91 such as a ROM, and executing the storage device 91, the input circuit 92, and the output circuit 93. It is realized by cooperating with other hardware of the in-vehicle image information processing devices 220 and 220a such as. Setting data such as threshold values and determination values used by the vehicle-mounted image information processing devices 220 and 220a are stored in a storage device 91 such as a ROM as part of software (program). Each function of the in-vehicle image information processing devices 220 and 220a may be composed of software modules, or may be composed of a combination of software and hardware.

<Calculation of importance>
FIG. 4 is a diagram showing image information 401 of the in-vehicle image information processing device 220 according to the first embodiment. An effective range 403 for abnormal area detection is shown for image information 401 captured by the camera 102 . The presence of objects such as road anomalies 106 on the road is limited to the effective range 403 . Objects (eg, birds, airplanes, buildings, etc.) that are outside the coverage area 403 are ignored as not road objects.

When the abnormal area detection unit 201 detects an object inside the effective range 403 , a rectangular area surrounding the object is selected as an abnormal area 402 . It is calculated that the greater the size of the abnormal region 402 is, the greater the importance of the abnormal region 402 is. When the area of the abnormal region 402 is Sm and the area of the effective range 403 is Se, the abnormal region ratio A is given by A=Sm/Se.

The abnormal region 402 has a center position P (X, Y) and a rectangular size (width: XL, height: YL), P1 (X-XL/2, Y-YL/2), P2 It consists of a rectangle surrounded by four points (X-XL/2, Y+YL/2), P3 (X+XL/2, Y+YL/2), and P4 (X+XL/2, Y-YL/2). When all of the four points P1, P2, P3, and P4 are inside the effective range 403, it indicates that the entire abnormal region 402 is easily visible, and the effective range flag FP is set to 1. When the four points P1, P2, P3, and P4 partially or entirely protrude from the effective range 403, it is assumed that it is difficult to identify the object in the abnormal area 402, and the effective range flag FP is set to 0. . (P(X, Y), XL, YL are not shown)

From the abnormal area ratio A and the valid range flag FP, the importance I of the abnormal area 402 can be calculated by Equation (1).

The importance level I may be obtained from only one index of the abnormal area ratio A and the valid range flag FP. Moreover, you may define the index different from these. If the abnormal area detection unit 201 does not detect an object, the abnormal area 402 is not selected.

When the abnormal area detection unit 201 detects an object, a rectangular abnormal area 402 surrounding the object is selected and the importance calculation unit 202 calculates the importance. The image information 401, the vehicle position information at the time of photographing, and the importance of the abnormal region 402 are stored in the storage unit 203 as important image information.

<Selection of the most important image information>
In FIG. 2, the storage unit 203 sequentially stores important image information for a predetermined time TM (for example, one minute in the past), and when the storage capacity of the storage unit 203 is exceeded, older important image information is deleted in order. If the predetermined time TM has passed since the transmission information was generated by the transmission information generation unit 207 and transmitted from the transmitter 104 last time, the communication will not be congested even if the transmission information is transmitted. Therefore, the selection unit 204 selects the most important image information having the highest importance among the important image information stored in the storage unit 203 . Although an example in which the predetermined time TM is 1 minute has been described here, the predetermined time TM may take a value such as 10 seconds, 1 second, or 100 ms.

The most important image information selected by the selection unit 204 is processed by the processing unit 206 to reduce the amount of information, and then transmission information is generated by the transmission information generation unit 207 and transmitted from the transmitter 104 . In the processing unit 206, only the image of the abnormal area 402 is extracted from the image information 401, the center position P (X, Y) of the abnormal area 402, the size of the abnormal area 402 (width: XL, height: YL), the abnormal area The image of the area, the position of the vehicle at the time of image capturing, and the information of importance are transmitted to the transmission information generation unit 207 . A transmission information generation unit 207 generates transmission information from the transmitted information and causes the transmitter 104 to transmit the information. The processing unit may process the image information with the same color (for example, white) except for the image of the abnormal region 402 to reduce the amount of information and transmit it to the transmission information generation unit 207 .

Here, the selection unit 204 selects the most important image information having the highest degree of importance among the important image information stored in the storage unit 203 every time a predetermined time TM elapses, and processes the most important image information before transmitting information. was to be generated. However, the most important image information may be selected each time the vehicle travels a predetermined distance DM instead of each time the predetermined time TM elapses. Also, the most important image information may be selected each time a predetermined number NM of important image information is stored in the storage unit 203 . Furthermore, the most important image information is selected each time a predetermined time TM elapses, each time a predetermined distance DM is traveled, and each time a predetermined number NM of important image information is stored. The distance DM and the predetermined number NM may be changed according to the state of the vehicle 101 (eg, vehicle speed, distance between vehicles traveling in parallel) and road conditions (eg, road type, congestion). (DM and NM are not shown)

When no important image information is stored in the storage unit 203 , the selection unit 204 does not select the most important image information, so the transmission information is not transmitted from the transmitter 104 . In this case, the communication network becomes idle (waiting state), and congestion of communication can be prevented.

<Continuous detection of abnormal areas>
Consider a case where the abnormal area detection unit 201 continuously detects similar objects in time, continuously detects abnormal areas 402, and calculates the degree of importance. For example, the selection unit 204 may select the important image information with the highest importance only when the same object is detected three times or more consecutively. By doing so, it is possible to prevent a windshield wiper, a bird, or the like crossing the effective range 403 of the photographing range 109 from being misidentified as an object on the road. In addition, with respect to continuous important image information, as the object on the road approaches, the abnormal area 402 expands and the importance increases, and an image that can be clearly identified is selected as the most important image information.

Assuming that the selection unit 204 selects the most important image information only when similar objects are detected continuously in time, the abnormal regions 402 are continuously detected, and the degree of importance is calculated, the abnormal region Optimal image information can be selected while preventing erroneous detection. It is possible to limit the transmission information to be transmitted from the transmitter 104 to prevent communication congestion, and to generate transmission information for appropriate image information.

In addition, when similar objects are detected continuously in time and abnormal areas 402 are detected continuously, the approach speed of the object or abnormal area is calculated based on the degree of increase in the size of the object or abnormal area 402. can be estimated. From this, it can be determined whether the object is stationary on the road or a moving object running side by side (or approaching).

<Importance that can be acquired in the future>
Here, a procedure for estimating the importance of important image information that can be obtained in the future from the important image information that is obtained continuously will be described. It can be assumed that the vehicle 101 is in uniform linear motion in a minute section along the road. Assuming that the image information 401 is acquired at regular time intervals, the road abnormality 106 fixed on the road approaches the vehicle 101 at regular distance intervals in each photographing in the coordinate system fixed to the vehicle 101. It can be regarded as That is, the square root of the abnormal area ratio A of the abnormal area 402 and the amount of movement of the center position P (X, Y) of the abnormal area 402, which is the position of the road abnormality 106 in the image information 401 defined in FIG. , and increases in inverse proportion to the distance of the road anomaly 106 .

For the n-th captured image information, the abnormal area ratio An, the center position Pn (Xn, Yn) of the abnormal area, the size of the abnormal area 402 (width: XLn, height: YLn), the boundary of the abnormal area 4 Points P1n (Xn-XLn/2, Yn-YLn/2), P2n (Xn-XLn/2, Yn+YLn/2), P3n (Xn+XLn/2, Yn+YLn/2), P4n (Xn+XLn/2, Yn-YLn /2), the valid range flag FPn, and the current number of shots is n, the next (n+1th image) is calculated using the current (nth image) and previous (n-1th image) image information. can be estimated by the equation (2).

At this time, if the selection unit 204 selects the most important image information from among the important image information stored in the storage unit 203 up to the n-th captured image information, the maximum importance Imax of the most important image information is estimated. If the estimated importance In+1 of the next important image information is higher, processing of the image information by the most important image information and generation of transmission information are suspended, and the n+1th photographed image information is waited for. By doing so, transmission information is transmitted from the transmitter 104 after obtaining important image information of an abnormal region with a higher degree of importance that can be obtained in the future. This can prevent the communication network from being congested.

Note that the method for estimating the degree of importance of an abnormal region in image information that can be acquired in the future is not limited to Equation 2. Further past shooting information may be used. Also, the formula for estimation may be changed according to the vehicle 101 and road conditions.

<Detection reliability of abnormal area>
A single-shot multibox detector whose software code has been published by Wei Liu et al. 2016, Vol.9905, p.21-37, <URL: https://doi.org/10.1007/978-3-319-46448-0_2>) may be used. By using a single-shot multi-box detector, it is possible to select a rectangular area surrounding an object included in image information by machine learning, and to calculate the precision of that area as a real number from 0 to 1. Here, assuming that an area surrounded by an object is an abnormal area 402 and the matching rate of this area is the detection reliability R of the abnormal area 402, the importance I of the abnormal area 402 can be calculated from the detection reliability R. In addition, when inferring by machine learning, the learning data of objects such as road depressions, cracks, bumps, falling objects, parallel running vehicles, stopped vehicles, pedestrians, and damaged road signs shall be learned in the learning model in advance. .

The abnormal area detection unit 201 and the importance calculation unit 202 use a single-shot multi-box detector to perform high-speed calculation with a relatively light load, determine the abnormal area, and calculate the relevance ratio as the detection reliability R. can do. The importance calculation unit 202 determines the importance I from the calculated detection reliability R. The degree of importance I can be obtained from the detection reliability R, the abnormal area ratio A, and the valid range flag FP by Equation (3).

重要度Ｉは、検出信頼度Ｒ、異常領域比率Ａ、有効範囲内フラグＦＰのいずれか一つまたはいずれか二つの指標から求めることとしてもよい。また、ここではシングルショットマルチボックス検出器を利用して異常領域４０２の適合率を異常領域４０２の検出信頼度Ｒとして計算したが、シングルショットマルチボックス検出器を利用せずに検出信頼度Ｒを算出してもよい。例えば、パターンマッチングによって、異常領域４０２を検出し画像と異常モデルのマッチング度合いを異常領域４０２の検出信頼度Ｒとして計算してもよい。

The importance level I may be obtained from any one or any two of the detection reliability R, the abnormal area ratio A, and the in-range flag FP. Also, here, the precision of the abnormal region 402 was calculated as the detection reliability R of the abnormal region 402 using the single-shot multi-box detector, but the detection reliability R was calculated without using the single-shot multi-box detector. can be calculated. For example, the abnormal region 402 may be detected by pattern matching, and the degree of matching between the image and the abnormal model may be calculated as the detection reliability R of the abnormal region 402 .

The in-vehicle image information processing device 220 generates transmission information from the most important image information having the highest importance I among the plurality of pieces of important image information stored in the storage unit 203 . Then, the transmission information is transmitted from the transmitter 104 of the in-vehicle image information transmission device 200 . The identification of the object caught in the abnormal region 402 may be performed by performing detailed image analysis with a large-scale, high-speed, and large-capacity computer of the external information processing device 300 .

Further, regarding the detection reliability R, the n+1-th detection reliability Rn+1 is calculated based on the detection reliability Rn-1, Rn of the important image information of the n-1-th and n-th images taken consecutively in time. can guess. Therefore, if the estimated importance In+1 of the estimated next important image information is higher, processing of the image information by the most important image information and generation of transmission information are suspended, and the important image information of the n+1th shot image is suspended. You can wait for

<Flowchart>
FIG. 5 shows a first flow chart showing image information processing of the in-vehicle image information processing device 220 according to the first embodiment. FIG. 6 shows a continuation of the flow chart of FIG. The flowchart of FIG. 5 is executed at predetermined time intervals (for example, 10 ms intervals). The processing of the flowchart may be performed not at every predetermined time but at every occurrence of another event, such as every time image information is transmitted from the camera 102 mounted on the vehicle 101 to the image input unit 208 . It is assumed that the timer added in step S507 is cleared at the time of initial setting.

The process starts from step S500, and the current position information transmitted from the positioning sensor 105 to the position input unit 211 is acquired in step S501. In step S502, image information input from the camera 102 mounted on the vehicle 101 to the image input unit 208 is acquired. In step S503, the abnormal area detection unit 201 detects the presence or absence of an abnormal area from the image information.

In step S504, it is determined whether an abnormal area has been detected. If the abnormal region can be detected (determination is YES), the importance calculation unit 202 calculates the importance of the abnormal region in step S505. Calculation of importance can be performed by, for example, equation (1) or equation (3). Thereafter, the process advances to step S506 to store the image information, the position information, and the degree of importance as important image information in the storage unit 203, and the process advances to step S507. If the abnormal region cannot be detected in step S504 (determination is NO), the process proceeds to step S507.

The timer is incremented in step S507. Here, as an example, a timer that is incremented every 10 ms is assumed. In step S508, it is determined whether or not the value of the timer is equal to or greater than a predetermined time TM (for example, 1 minute). It is determined whether or not the predetermined time TM has passed since the previous transmission information was generated. If the predetermined time TM has passed (determination is YES), the process proceeds to step S509. If the predetermined time TM has not elapsed (determination is NO), the process proceeds to step S519 and ends.

In step S509, it is determined whether or not there is an abnormal area detected three times or more consecutively. If there is an abnormal area detected three times or more in succession (determination is YES), the process proceeds to step S510. If there is no abnormal area detected three times or more in succession (determination is NO), the process proceeds to step S517, the timer is cleared in step S517, and the process ends in step S519. Assuming that there is no reliable abnormal area, the detection of the abnormal area within the next predetermined time TM is started without generating the transmission information.

In step S510, the selection unit 204 checks the degree of importance of the abnormal regions detected three or more times in succession, and selects the abnormal region with the highest degree of importance. The important image information having the maximum importance Imax in this case is the most important image information.

In step S511, the estimated importance In+1 is calculated. The estimated importance In+1 is a value obtained by estimating the importance of an abnormal region of image information that can be acquired in the future. Calculation of the inferred importance In+1 can be performed, for example, by Equation (2).

Next, in step S512, it is determined whether or not the estimated importance In+1 is greater than the maximum importance Imax. If the estimated importance In+1 is greater than the maximum importance Imax (determination is YES), the process proceeds to step S519 and ends. In this case, the most important image information is selected again after waiting for the acquisition of the next image information. Since the transmission information is not transmitted from the transmitter 104 and the acquisition of the next image information is awaited, communication congestion can be suppressed.

In step S512, if the estimated importance In+1 is not greater than the maximum importance Imax (determination is NO), the process proceeds to step S513 to determine whether the maximum importance Imax is greater than the importance threshold Ith. If the maximum importance Imax is greater than the importance threshold Ith (determination is YES), the process proceeds to step S514. If the maximum importance Imax is not greater than the importance threshold Ith (determination is NO), the process advances to step S517 to clear the timer, and the process ends in step S519. Since there is no abnormal area having the importance I exceeding the importance threshold Ith, the image information is not transmitted, and the detection of the abnormal area is started for the next predetermined time TM.

In step S514, the processing unit 206 extracts only the image of the abnormal region from the image information of the most important image information to reduce the amount of information. Then, in step S516, transmission information is generated and transmitted to the transmitter 104. FIG. The transmission information includes the image of the extracted abnormal area, the center position P (X, Y) of the abnormal area, the size of the abnormal area (width: XL, height: YL), vehicle position information, and importance. .

At step S517, the timer is cleared. This is because the predetermined time TM is newly counted, new most important image information is confirmed, and transmission information is generated. Since the transmission information is only generated based on a maximum of one piece of most important information in the predetermined time TM, congestion of communication can be prevented.

After that, the process ends in step S519. If an abnormal area is not detected, if an abnormal area is detected but not detected three or more times in a row, even if an abnormal area is detected three or more times in a row, the importance level I does not exceed the importance level threshold Ith. If it does not exceed the threshold, no transmission information is generated, and therefore transmission to the external information processing apparatus 300 is not performed. This makes it possible to prevent communication congestion.

That is, the in-vehicle image information processing device 220 implements the image information processing method through the following steps. (1) A step of inputting the image information 401 of the image captured by the camera 102 provided in the vehicle 101 for capturing the surroundings of the vehicle to the image input unit 208 from the camera 102 . (2) A step in which the abnormal region 402 is detected from the image information 401 input to the image input unit 208 by the abnormal region detection unit 201 . (3) A step in which the degree of importance of the abnormal region 402 detected by the abnormal region detection unit 201 is calculated by the degree of importance calculation unit 202 . (4) A step in which the storage unit 203 stores important image information including the detected image information 401 of the abnormal region 402 whose importance and importance have been calculated. (5) A step of selecting the most important image information having the highest degree of importance from the plurality of pieces of important image information stored in the storage unit 203 by the selection unit 204 . (6) A step of generating transmission information based on the most important image information selected by the selection unit 204 by the transmission information generation unit 207 .

By executing the image processing in such steps, the in-vehicle image information processing device 220 can reduce the amount of information to be transmitted, and perform transmission for transmitting appropriate image information of the surroundings of the vehicle to the external information processing device 300. Since the image information processing method for generating information can be implemented, it is possible to suppress communication congestion and enable highly reliable transmission of image information.

2. Embodiment 2
<Functional block>
FIG. 7 shows a functional block diagram of an in-vehicle image information processing device 220a according to the second embodiment. The vehicle 101 includes a camera 102 for capturing images around the vehicle, a positioning sensor 105 for measuring the position of the vehicle 101, a transmitter 104 for transmitting transmission information based on image information to the external information processing device 300, and an external information processing device. A receiver 107 for receiving an image analysis result from 300 and an in-vehicle image information processing device 220a are mounted. An in-vehicle image information transmitting device 200 a is composed of the in-vehicle image information processing device 220 a, the camera 102 , the positioning sensor 105 , the transmitter 104 and the receiver 107 . Receiver 107 may be omitted.

The block diagram of FIG. 7 differs from the block diagram of FIG. 2 according to the first embodiment in that a compression unit 210 is added to the in-vehicle image information processing device 220a, the estimation unit 209 is omitted, and an abnormal The difference is that the area detection unit 201 is changed to 201a, the importance calculation unit 202 is changed to 202a, the processing unit 206 is changed to 206a, and the transmission information generation unit 207 is changed to 207a. The hardware configuration of the vehicle-mounted image information processing device 220a according to the second embodiment is the same as the configuration shown in FIG. 3 according to the first embodiment. The following description will focus on the parts that are different from the first embodiment.

Embodiment 2 describes an example in which the abnormal area detection unit 201a detects an abnormal area by object detection by pattern matching. The geometric features of objects such as road depressions, cracks, bumps, fallen objects, parallel running vehicles, stopped vehicles, pedestrians, and damaged road signs are stored in advance as template features (abnormal models), and the features of captured images are used. Objects are detected by pattern matching between the and template features. Let the area where the object exists be the abnormal area.

The abnormal area detection unit 201a of the in-vehicle image information processing device 220a detects an abnormal area if there is a possibility that an object can be detected by pattern matching, and the importance calculation unit 202a calculates the importance. The external information processing apparatus 300 may perform image analysis, object identification, and identification of what the object is. The external information processing device 300 can perform image analysis using a high-speed, large-scale, and large-capacity computing device based on the image information received from the vehicle-mounted image information processing device 220a, or can identify objects by monitoring personnel. .

<Calculation of importance>
FIG. 8 is a diagram showing image information 401a of the vehicle-mounted image information processing device 220a according to the second embodiment. An effective range 403a for abnormal area detection is shown for image information 401a captured by the camera 102 . The presence of objects such as road anomalies 106a on the road is limited to the effective range 403a. Objects (eg, birds, airplanes, buildings, etc.) that exist outside the range of coverage 403a are ignored as non-road objects.

When the abnormal area detection unit 201a detects an object inside the effective range 403a, it detects a polygonal area surrounding the object as an abnormal area 402a. FIG. 8 shows an illuminated portion 404 of the headlights of the vehicle 101 in the image information 401a taken at night.

The importance of the abnormal region 402a is calculated by the importance calculator 202a. The following three evaluation indicators can be cited as factors that affect the degree of importance.

First, the abnormal area ratio Aa of the detected abnormal area 402a is obtained. When the area of the abnormal region 402a is Sma and the area of the effective range 403a is Se, the abnormal region ratio Aa is expressed by Aa=Sma/Se. The higher the abnormal area ratio Aa, the higher the importance.

Second, the variance (contrast) of luminance values within the detected abnormal region is calculated. The variance of luminance values is used as a measure of the clarity of the captured image. When the road anomaly is strongly illuminated by headlights, streetlights, sunlight, etc., the variance of luminance values increases. From this, it is determined that the greater the dispersion of the luminance values, the better the light source environment at the time of photographing, and the higher the importance.

Third, calculate the maximum intensity of the edges within the detected anomalous region. The intensity of the edge in the abnormal region can be calculated, for example, by convoluting a Sobel filter, which is a spatial filter used for image contour detection, and the captured image. The edge strength is the degree of change in the gradation of the boundary between the object and the background, and the higher the edge strength, the higher the degree of success in object identification, and the higher the importance.

The importance I of the abnormal region 402a is calculated by the equation (4) based on the abnormal region ratio Aa of the abnormal region 402a detected for the image information 401a, the variance V of the luminance value of the abnormal region 402a, and the maximum intensity E of the edge of the abnormal region 402a. be able to.

In FIG. 8, the road abnormality 106a is located in the illuminated portion 404 of the headlight, so that the contrast is high, and the shape of the road abnormality 106a is large and clearly captured. According to Equation (4), the larger the area of the abnormal region 402a, the higher the importance I is given to the image information 401a. A higher degree of importance I is given to the image information 401a as the variance of the luminance values increases. The higher the intensity of the edge, the higher the importance I is given to the image information 401a.

The formula for calculating the degree of importance is not limited to formula (4). The importance level I may be determined based on one or two indices of the detected abnormal area ratio Aa, the variance V of the brightness values, and the maximum intensity E of the edge. Further, the importance level I may be determined using other indicators including the detection reliability R and the valid range flag FP, which are indicators for determining the importance level I according to the first embodiment.

When the abnormal area detection unit 201a in the block diagram of FIG. 7 detects an abnormal area 402a in the image information 401a, the image information 401a, the vehicle position information at the time of photographing, and the importance I of the abnormal area 402a are the important images. It is stored in the storage unit 203 as information.

<Selection of the most important image information>
The storage unit 203 sequentially stores the important image information for a predetermined time TM (for example, one minute in the past), and when the storage capacity of the storage unit 203 is exceeded, the old important image information is deleted in order. If the predetermined time TM has passed since the transmission information was generated by the transmission information generation unit 207a and transmitted from the transmitter 104 last time, the communication will not be congested even if the transmission information is transmitted. Therefore, the selection unit 204 selects the most important image information having the highest importance among the important image information stored in the storage unit 203 .

The most important image information selected by the selection unit 204 is processed by the processing unit 206a to reduce the amount of information. In the processing unit 206a of the second embodiment, high-frequency components are cut by a low-pass filter for image information of portions other than the abnormal region 402a with respect to the image information 401a. Then, the compression unit 210 compresses the entire image information according to the JPEG (Joint Photographic Experts Group) method. Through this processing and compression, it is possible to reduce the amount of information to be transmitted by utilizing the compression characteristics of the JPEG system while ensuring the clarity of the image of the abnormal region 402a. Information about the compressed image information, the position of the abnormal region in the entire image information, the position of the vehicle at the time of image capturing, and the degree of importance are transmitted to the transmission information generation unit 207 . Based on these pieces of information, transmission information is generated by transmission information generating section 207 and transmitted from transmitter 104 .

If information about areas other than the abnormal area 402a is unnecessary, the amount of information may be reduced by filling the area other than the abnormal area 402a with a single color (for example, white). In addition to the JPEG method, other compression methods such as the PNG (Portable Network Graphics) method and the GIF (Graphics Interchange Format) method may be used as the image compression method.

When the abnormal area detection unit 201a does not detect an object in the effective range 403a of the image information 401a, the important image information is not stored in the storage unit 203, and the selection unit 204 does not select the most important image. Therefore, since the transmission information is not generated by the transmission information generating section 207 and the transmission is not performed from the transmitter 104, it is possible to prevent the communication from being congested.

<Flowchart>
FIG. 9 shows a first flow chart showing image information processing of the vehicle-mounted image information processing device 220a according to the second embodiment. FIG. 10 shows a continuation of the flow chart of FIG. The flowchart of FIG. 9 is assumed to be executed at predetermined time intervals (for example, at intervals of 10 ms). The processing of the flowchart may be performed not at every predetermined time but at every occurrence of another event, such as every time image information is transmitted from the camera 102 mounted on the vehicle 101 to the image input unit 208 . Note that the timer added in step S907 is cleared at the time of initial setting.

Processing starts from step S900. Since steps S900 to S906 in FIG. 9 are the same in outline as steps S501 to S506 in FIG. 5, the description thereof will be omitted. Although the method of abnormal area detection in step S903 is different, the functions of the abnormal area detection unit 201a have already been described. Although the method of calculating the degree of importance in step S905 is different, the function of the degree-of-importance calculation unit 202a has already been described.

Steps S907 to S919 in FIG. 10 are different from steps S507 to S519 in FIG. 6 in that steps S511 and S512 are deleted, step S915 is newly added, and the content of step S914 is changed from step S514. Other than this, the outline of the processing is the same. Description will be made from step S909 in FIG.

In step S909, it is determined whether or not there is important image information in which abnormal regions are detected three times or more consecutively. If there is an abnormal area detected three times or more in succession (determination is YES), the process proceeds to step S910. If there is no abnormal area detected three times or more in succession (determination is NO), the process proceeds to step S917, the timer is cleared in step S917, and the process ends in step S919. Assuming that there is no reliable abnormal area, the detection of the abnormal area within the next predetermined time TM is started without generating the transmission information.

In step S910, the selection unit 204 checks the degree of importance of the abnormal regions detected three or more times in succession, and selects the abnormal region with the highest degree of importance. The important image information having the maximum importance Imax in this case is the most important image information.

Proceeding to step S913, it is determined whether or not the maximum importance Imax is greater than the importance threshold Ith. If the maximum importance Imax is greater than the importance threshold Ith (determination is YES), the process proceeds to step S914. If the maximum importance Imax is not greater than the importance threshold Ith (determination is NO), the process advances to step S917 to clear the timer, and the process ends in step S919. Since there is no abnormal area having the importance I exceeding the importance threshold Ith, the image information is not transmitted, and the detection of the abnormal area is started for the next predetermined time TM.

In step S914, the processing unit 206a applies a low-pass filter to image information other than the image of the abnormal region from the image information of the most important image information to reduce the amount of information. In step S915, the compression unit 210 compresses the image whose information amount has been reduced by the JPEG method. Then, in step S916, transmission information is generated and transmitted to the transmitter 104. FIG. The transmission information includes the compressed image, the coordinates of the points surrounding the abnormal area, the position information of the vehicle at the time of photographing, and the degree of importance.

At step S917, the timer is cleared. This is because the predetermined time TM is newly counted, new most important image information is confirmed, and transmission information is generated. Since the transmission information is only generated based on one piece of most important information in the predetermined time TM, congestion of communication can be prevented.

After that, the process ends in step S919. If an abnormal area is not detected, if an abnormal area is detected but not detected three or more times in a row, even if an abnormal area is detected three or more times in a row, the importance level I does not exceed the importance level threshold Ith. If it does not exceed the threshold, no transmission information is generated, and therefore transmission to the external information processing apparatus 300 is not performed. This makes it possible to prevent communication congestion.

While this application describes various exemplary embodiments and examples, various features, aspects, and functions described in one or more embodiments may not apply to particular embodiments. can be applied to the embodiments singly or in various combinations. Accordingly, numerous variations not illustrated are envisioned within the scope of the technology disclosed herein. For example, modification, addition or omission of at least one component, extraction of at least one component, and combination with components of other embodiments shall be included.

101 vehicle, 102 camera, 104 transmitter, 105 positioning sensor, 106, 106a road abnormality, 200, 200a in-vehicle image information transmission device, 201, 201a abnormal area detection unit, 202, 202a importance calculation unit, 203 storage unit, 204 selection unit, 206, 206a processing unit, 207, 207a transmission information generation unit, 208 image input unit, 209 estimation unit, 210 compression unit, 211 position input unit, 220, 220a in-vehicle image information processing device, 401, 401a image information, 402, 402a abnormal region, 403, 403a effective range, A, Aa abnormal region ratio, E maximum edge intensity, FP effective range flag, I importance, Imax maximum importance, R detection reliability, V brightness value variance

Claims (19)

an image input unit that receives image information from a camera that is provided in a vehicle and captures an image of the surroundings of the vehicle;
an abnormal area detection unit that detects an abnormal area from the image information input to the image input unit;
an importance calculation unit that calculates the importance of the abnormal area based on the ratio of the abnormal area detected by the abnormal area detection unit to the image information ;
a storage unit that stores important image information including the image information in which the degree of importance and the abnormal region for which the degree of importance is calculated is detected;
a selection unit that selects the most important image information having the highest degree of importance from the plurality of important image information stored in the storage unit;
An in-vehicle image information processing apparatus comprising a transmission information generation unit that generates transmission information based on the most important image information selected by the selection unit. 2. The in-vehicle image information processing apparatus according to claim 1, wherein said importance calculating section further calculates the importance based on the position of said abnormal region in said image information. an image input unit that receives image information from a camera that is provided in a vehicle and captures an image of the surroundings of the vehicle;
an abnormal area detection unit that detects an abnormal area from the image information input to the image input unit;
an importance calculation unit that calculates the importance of the abnormal area detected by the abnormal area detection unit;
a storage unit that stores important image information including the image information in which the degree of importance and the abnormal region for which the degree of importance is calculated is detected;
a selection unit that selects the most important image information having the highest degree of importance from the plurality of important image information stored in the storage unit;
an estimating unit that estimates the importance of the abnormal region of the image information that can be obtained in the future and calculates the estimated importance;
generating transmission information based on the most important image information selected by the selection unit, wherein the estimated importance calculated by the estimation unit is the importance of the most important image information selected by the selection unit; a transmission information generation unit that suspends generation of transmission information if higher than
In - vehicle image information processing device. an image input unit that receives image information from a camera that is provided in a vehicle and captures an image of the surroundings of the vehicle;
an abnormal area detection unit that detects an abnormal area from the image information input to the image input unit;
an importance calculation unit that calculates the importance of the abnormal area detected by the abnormal area detection unit;
a storage unit that stores important image information including the image information in which the degree of importance and the abnormal region for which the degree of importance is calculated is detected;
a selection unit that selects important image information related to the abnormal area continuously detected by the abnormal area detection unit as the most important image information having the highest degree of importance from the plurality of important image information stored in the storage unit ;
An in-vehicle image information processing apparatus comprising a transmission information generation unit that generates transmission information based on the most important image information selected by the selection unit. an image input unit that receives image information from a camera that is provided in a vehicle and captures an image of the surroundings of the vehicle;
an abnormal area detection unit that detects an abnormal area from the image information input to the image input unit and calculates the detection reliability of the detected abnormal area;
an importance calculation unit that calculates the importance of the abnormal area detected by the abnormal area detection unit based on the detection reliability;
a storage unit that stores important image information including the image information in which the degree of importance and the abnormal region for which the degree of importance is calculated is detected;
a selection unit that selects the most important image information having the highest degree of importance from the plurality of important image information stored in the storage unit;
An in-vehicle image information processing apparatus comprising a transmission information generation unit that generates transmission information based on the most important image information selected by the selection unit. an image input unit that receives image information from a camera that is provided in a vehicle and captures an image of the surroundings of the vehicle;
an abnormal area detection unit that detects an abnormal area from the image information input to the image input unit;
an importance calculation unit that calculates the importance of the abnormal region based on the variance of the luminance values of the abnormal region detected by the abnormal region detection unit ;
a storage unit that stores important image information including the image information in which the degree of importance and the abnormal region for which the degree of importance is calculated is detected;
a selection unit that selects the most important image information having the highest degree of importance from the plurality of important image information stored in the storage unit;
An in-vehicle image information processing apparatus comprising a transmission information generation unit that generates transmission information based on the most important image information selected by the selection unit. an image input unit that receives image information from a camera that is provided in a vehicle and captures an image of the surroundings of the vehicle;
an abnormal area detection unit that detects an abnormal area from the image information input to the image input unit;
an importance calculation unit that calculates the importance of the abnormal region based on the edge strength included in the abnormal region detected by the abnormal region detection unit ;
a storage unit that stores important image information including the image information in which the degree of importance and the abnormal region for which the degree of importance is calculated is detected;
a selection unit that selects the most important image information having the highest degree of importance from the plurality of important image information stored in the storage unit;
An in-vehicle image information processing apparatus comprising a transmission information generation unit that generates transmission information based on the most important image information selected by the selection unit. a processing unit that processes the image associated with the most important image information selected by the selection unit to generate processed information with a reduced amount of information;
The in-vehicle image information processing apparatus according to any one of claims 1 to 7 , wherein the transmission information generating section generates the transmission information based on the processed information generated by the processing section. 9. The in-vehicle image information processing apparatus according to claim 8 , wherein the processing unit extracts the abnormal region from the image related to the most important image information to generate the processing information. 9. The in-vehicle image information processing apparatus according to claim 8 , wherein the processing unit generates the processing information by reducing the information amount of an image outside the abnormal region of the image related to the most important image information. A compression unit that compresses the processed information generated by the processing unit to generate compressed information,
The in-vehicle image information processing apparatus according to any one of claims 8 to 10 , wherein the transmission information generation section generates the transmission information based on the compression information generated by the compression section. A position input unit for inputting the position information of the vehicle,
The storage unit stores, as important image information, information including the degree of importance, the image information in which the abnormal region for which the degree of importance is detected, and the position information when the image related to the image information was captured. The in-vehicle image information processing device according to any one of claims 1 to 11 . the camera;
a transmitter that transmits the transmission information to the outside of the vehicle;
An in-vehicle image information transmission device comprising the in-vehicle image information processing device according to any one of claims 1 to 12 . a step of inputting image information of an image captured by a camera provided in a vehicle for capturing the surroundings of the vehicle into an image input unit;
a step in which an abnormal area detection unit detects an abnormal area from the image information input to the image input unit;
A step of calculating the importance of the abnormal region by an importance calculation unit based on the ratio of the abnormal region detected by the abnormal region detection unit to the image information ;
a step of storing, by a storage unit, important image information including the degree of importance and the image information of the detected abnormal region for which the degree of importance is calculated;
a step of selecting, by a selection unit, the most important image information having the highest degree of importance from the plurality of important image information stored in the storage unit;
and a step of generating transmission information based on the most important image information selected by the selection section by a transmission information generation section. a step of inputting image information of an image captured by a camera provided in a vehicle for capturing the surroundings of the vehicle into an image input unit;
a step in which an abnormal area detection unit detects an abnormal area from the image information input to the image input unit;
A step of calculating the importance of the abnormal area detected by the abnormal area detection unit by an importance calculation unit;
a step of storing, by a storage unit, important image information including the degree of importance and the image information of the detected abnormal region for which the degree of importance is calculated;
a step of selecting, by a selection unit, the most important image information having the highest degree of importance from the plurality of important image information stored in the storage unit;
a step of estimating an estimated importance, which is the importance of an abnormal region of image information that can be obtained in the future, by an estimating unit;
Transmission information based on the most important image information selected by the selection unit is generated by a transmission information generation unit, and the estimated importance calculated by the estimation unit is the most important image selected by the selection unit. if the importance of information is higher than said importance, then the generation of transmission information is paused;
An image information processing method comprising: a step of inputting image information of an image captured by a camera provided in a vehicle for capturing the surroundings of the vehicle into an image input unit;
a step in which an abnormal area detection unit detects an abnormal area from the image information input to the image input unit;
A step of calculating the importance of the abnormal area detected by the abnormal area detection unit by an importance calculation unit;
a step of storing, by a storage unit, important image information including the degree of importance and the image information of the detected abnormal region for which the degree of importance is calculated;
A selecting unit selects important image information related to the abnormal areas continuously detected by the abnormal area detecting unit as the most important image information having the highest degree of importance from the plurality of important image information stored in the storage unit. a step;
and a step of generating transmission information based on the most important image information selected by the selection section by a transmission information generation section. a step of inputting image information of an image captured by a camera provided in a vehicle for capturing the surroundings of the vehicle into an image input unit;
an abnormal area is detected by an abnormal area detection unit from the image information input to the image input unit, and a detection reliability of the detected abnormal area is calculated;
A step of calculating the importance of the abnormal area detected by the abnormal area detection unit by an importance calculation unit based on the detection reliability;
a step of storing, by a storage unit, important image information including the degree of importance and the image information of the detected abnormal region for which the degree of importance is calculated;
a step of selecting, by a selection unit, the most important image information having the highest degree of importance from the plurality of important image information stored in the storage unit;
and a step of generating transmission information based on the most important image information selected by the selection section by a transmission information generation section. a step of inputting image information of an image captured by a camera provided in a vehicle for capturing the surroundings of the vehicle into an image input unit;
a step in which an abnormal area detection unit detects an abnormal area from the image information input to the image input unit;
A step of calculating the importance of the abnormal area detected by the abnormal area detection unit by an importance calculation unit based on the variance of the luminance value of the abnormal area;
a step of storing, by a storage unit, important image information including the degree of importance and the image information of the detected abnormal region for which the degree of importance is calculated;
a step of selecting, by a selection unit, the most important image information having the highest degree of importance from the plurality of important image information stored in the storage unit;
and a step of generating transmission information based on the most important image information selected by the selection section by a transmission information generation section. a step of inputting image information of an image captured by a camera provided in a vehicle for capturing the surroundings of the vehicle into an image input unit;
a step in which an abnormal area detection unit detects an abnormal area from the image information input to the image input unit;
A step of calculating the importance of the abnormal area detected by the abnormal area detection unit by an importance calculation unit based on the edge strength included in the abnormal area;
a step of storing, by a storage unit, important image information including the degree of importance and the image information of the detected abnormal region for which the degree of importance is calculated;
a step of selecting, by a selection unit, the most important image information having the highest degree of importance from the plurality of important image information stored in the storage unit;
and a step of generating transmission information based on the most important image information selected by the selection section by a transmission information generation section.

Images