JP2023055204A - Vehicle recording device - Google Patents

Abstract

To provide a vehicle recording device which can suppress the storage data size in such a configuration that periodically stores image data captured by a camera.SOLUTION: A processor in a camera ECU acquires image frames captured by a camera at a prescribed sampling interval, and cuts out as a partial image PI a portion that is set as a cut-out region CR in each image frame. The partial image is, for example, cut out in a landscape format so as to include a prescribed line range in the image frame. The cut-out region CR is set in a portion in which an arbitrary target object such as a road surface, the sky, and a preceding vehicle is likely to be captured in the image frame. The processor generates a connected image in which the plurality of partial images with different imaging time points (acquired time points) are connected in a direction orthogonal to a cut-out direction and stores the connected image in a storage memory.SELECTED DRAWING: Figure 5

Description

TECHNICAL FIELD The present disclosure relates to a vehicle recording device that stores image data representing conditions outside/inside a vehicle.

Patent Literature 1 discloses a recording device that continuously records camera images in a buffer memory or the like and stores video data for a predetermined time before and after a predetermined incident is detected in an external memory. This type of recording device is also called an event recording type drive recorder. In addition to the event recording type, there is also a continuous recording type (constant recording type) drive recorder that continuously records camera images on a storage medium as a device for recording the situation outside the vehicle.

JP 2011-96063 A

In event-recording drive recorders, things of interest that the viewer of the video would like to see may be partially/totally not recorded. A constant recording type drive recorder can reduce the possibility that an object of interest is not recorded, but may increase the size of stored data.

The present disclosure has been made based on the above studies, and one of its purposes is to provide a vehicle recording apparatus capable of suppressing the size of stored data in a configuration for periodically storing image data captured by a camera. is to provide

The vehicle recording device disclosed herein includes a video acquisition unit (F1) that acquires a video signal from a camera installed in a vehicle, and an image frame that is generated based on the video signal acquired by the video acquisition unit. An image acquisition unit (F41) that sequentially acquires at a predetermined sampling interval, and a clipping unit that clips a row range or a column range set as a clipping region from the image frame acquired by the image capturing unit as a partial image (PI). (F42); and a connection processing unit (F43) that connects the plurality of partial images cut out by the cutout unit in chronological order in a direction orthogonal to the cutout direction, which is the direction in which the partial images are cut out from the image frame; A storage processing unit (F44) for storing a connected image (CI), which is an image in which a plurality of partial images are connected and generated by the processing unit, in a predetermined storage medium (15).

According to the above configuration, image frames sequentially acquired at sampling intervals are not stored in their original size, but only a portion corresponding to the clipped region is extracted and stored as a connected image arranged in chronological order. Therefore, the data size can be suppressed.

It should be noted that the symbols in parentheses described in the claims indicate the corresponding relationship with specific means described in the embodiments described later as one aspect, and limit the technical scope of the present disclosure. isn't it.

1 is a block diagram showing an overview of a camera image processing system; FIG. It is a figure which shows an example of the mounting position and mounting attitude|position of a camera. 4 is a functional block diagram of a camera ECU; FIG. FIG. 10 is a diagram for explaining the operation of the region-of-interest clipping section; 4 is a diagram for explaining the operation of the data processing unit; FIG. FIG. 4 is a diagram showing the structure of a connected image; 4 is a flow chart for explaining the operation of a data processing unit relating to generation of a connected image; FIG. 10 is a diagram showing a setting example of a cutout region; FIG. 9 is a diagram showing an example of a connected image generated by setting the cutout regions shown in FIG. 8; FIG. 10 is a diagram showing another setting example of a cutout region; FIG. 11 is a diagram showing a connected image generated by setting the cutout regions shown in FIG. 10; FIG. 10 is a diagram showing another setting example of a cutout region; FIG. 13 is a diagram showing a connected image generated by setting the cutout regions shown in FIG. 12; FIG. 5 is a block diagram showing a modification of the configuration of the data processing unit; FIG. 10 is a diagram schematically showing a sequence for generating a connected image based on image frames containing markers indicating recognition results; FIG. 10 is a diagram showing a group of image frames containing markers indicating recognition results; 17 is a diagram showing an example of a connected image generated based on the image frame group shown in FIG. 16; FIG. FIG. 10 is a diagram for explaining a configuration for dynamically adjusting the clipping width according to the recognition status of the target; 1 is a block diagram showing a configuration with multiple cameras; FIG. FIG. 3 is a diagram for explaining a configuration for collectively saving images from a plurality of cameras; FIG. FIG. 10 is a diagram for explaining another example of a configuration for collectively saving images from a plurality of cameras; FIG. 4 is a diagram for explaining a configuration for extracting a partial image based on a bird's-eye view image; FIG. 10 is a diagram for explaining a mode of changing a sampling interval or the like according to a scene; FIG. FIG. 11 is a block diagram showing another configuration example of the camera image processing system; It is a functional block diagram in case a data processing part is provided with an inspection process part. FIG. 10 is a diagram showing an example of setting a cutout region when an upper sign is set as an inspection object; FIG. 4 is a diagram showing an example of items contained in shooting situation data; It is a figure for demonstrating the operation|movement of an inspection process part. 4 is a flowchart for explaining the operation of a camera ECU relating to image diagnosis processing; 9 is a flow chart showing another operation example of the camera ECU relating to image diagnosis processing.

Embodiments of the present disclosure will be described below with reference to the drawings.

<Introduction>
The camera image processing system Sys according to the present disclosure is used while being mounted on a vehicle. The own vehicle in the following description refers to a vehicle equipped with the camera image processing system Sys. In the following description, directions of front and rear, left and right, and up and down are defined with reference to the own vehicle. Specifically, the longitudinal direction corresponds to the longitudinal direction of the vehicle. The left-right direction corresponds to the width direction of the host vehicle. The vertical direction corresponds to the vehicle height direction. From another point of view, the vertical direction corresponds to a direction perpendicular to a plane parallel to the front-rear direction and the left-right direction. In the present disclosure, the plane perpendicular to the vehicle height direction is also referred to as a vehicle horizontal plane RP. The vehicle horizontal plane RP corresponds to a horizontal plane determined with the vehicle as a reference.

"Parallel" in the present disclosure is not limited to a perfect parallel state. It may be inclined from several degrees to about 20 degrees. That is, it can include a substantially parallel state (a so-called substantially parallel state). The expression “vertical” in the present disclosure is not limited to a completely vertical state, and can include a state of being tilted from several degrees to 20 degrees.

<Explanation of overall configuration>
FIG. 1 is a diagram showing an example of a schematic configuration of a camera image processing system Sys according to the present disclosure. As shown in FIG. 1, the camera image processing system Sys includes a camera ECU 1, a camera 2, an input device 3, an in-vehicle sensor 4, and an in-vehicle ECU 5. Note that the ECU in the member name is an abbreviation for Electronic Control Unit, meaning an electronic control unit.

The camera ECU 1 is connected to each of the camera 2, the input device 3, the vehicle-mounted sensor 4, and the vehicle-mounted ECU 5 so as to be able to communicate with each other. The camera ECU 1 is connected to the various devices/sensors described above via an in-vehicle network, which is a communication network built in the vehicle. Some of the various devices/sensors described above may be individually connected to the camera ECU 1 via dedicated lines. For example, the camera 2 and the camera ECU 1 may be directly connected by a dedicated video signal line.

The camera ECU 1 is an ECU that executes various processes based on video signals input from the camera 2 . The camera ECU 1 functions as a vehicle recording device. That is, the camera ECU 1 executes a process of storing, in a predetermined memory, a specific region of an image frame extracted at a predetermined sampling rate from temporally continuous image frames forming a video signal input from the camera 2. . The output of the camera 2 is not limited to video and may be a still image. The expression "video signal" also includes an image signal.

As a more preferable configuration, the camera ECU 1 can recognize an object set as a detection target based on a video signal input from the camera 2 . In addition, the camera ECU 1 can execute a process of displaying an image captured by the camera 2 as it is or after performing predetermined processing on the display. For example, a bird's-eye view image obtained by converting a camera image from a viewpoint, an image obtained by superimposing a marker image indicating a recognition result on a camera image, or the like can be generated and displayed on an arbitrary display. The details of the functions of the camera ECU 1 will be described separately later.

The camera ECU 1 is implemented using a computer. That is, the camera ECU 1 includes a processor 11, a RAM (Random Access Memory) 12, a storage 13, an I/O 14, a storage memory 15, a bus line connecting these components, and the like. The processor 11 is hardware (in other words, an arithmetic core) for arithmetic processing coupled with the RAM 12 . The processor 11 is, for example, a CPU (Central Processing Unit). The processor 11 may be implemented using a GPU (Graphics Processing Unit). By accessing the RAM 12, the processor 11 executes various processes corresponding to each functional unit, which will be described later. RAM 12 is a volatile storage medium.

The storage 13 is a non-volatile storage medium such as flash memory. The storage 13 corresponds to a so-called internal storage provided on a circuit board forming the camera ECU 1 together with the processor 11, RAM 12, and the like. The storage 13 stores dictionary data/identification models used for image recognition processing, drawing data for generating a synthesized image, and the like. The I/O 14 is a circuit module for communicating with other devices. The I/O 14 corresponds to an input/output circuit. The I/O 14 is implemented using an analog circuit element, an IC, or the like.

The storage memory 15 is a rewritable non-volatile storage medium. The storage memory 15 is a removable storage medium such as an SD memory card (so-called SD card). The storage memory 15 corresponds to a storage destination of connected image data based on camera images. Note that the storage memory 15 may be a memory device incorporated in a circuit board, like the storage 13 .

The camera 2 has at least a lens and an imaging element, and electronically acquires an image showing the surroundings of the own vehicle. The camera 2 is mounted at a predetermined position on the vehicle in a predetermined posture so as to capture an image of a predetermined range outside the vehicle. As an example, the camera 2 of the present embodiment is an in-vehicle camera that captures an image in front of the own vehicle. The camera 2 is arranged at the upper end of the windshield on the interior side of the vehicle, as shown in FIG. Such a camera 2 can be called a front camera. Note that the camera 2 as a front camera may be arranged on the front grille, the front bumper, or the like.

The horizontal angle of view of the camera 2 is set to 60° to 130°, and the vertical angle of view is set to about 45° to 60°. Of course, the camera 2 may be a wide-angle camera using a fisheye lens. Alternatively, the camera 2 may be a narrow-angle/telephoto camera with a horizontal angle of view of less than 30°. The camera 2 is mounted in a posture in which the optical axis is parallel to the horizontal plane RP of the vehicle, or directed downward/upward by several degrees from the horizontal plane RP of the vehicle. The camera 2 may be mounted in a posture that faces downward or upward by about 10°. The camera 2 of this embodiment is mounted in a posture capable of capturing an image of the road surface 10 m or more in front and also capturing an image of the sky ahead. According to the mounting attitude, the image frame captured by the camera 2 can include the road surface and the sky.

θu shown in FIG. 2 indicates an upper limit angle, which is the angle formed by the upper boundary Bu of the range that can be captured by the camera 2 in the vertical direction (vertical direction) with respect to the vehicle horizontal plane RP. θd shown in FIG. 2 indicates a lower limit angle that is the angle formed by the lower boundary Bd of the range that can be captured by the camera 2 in the vertical direction with respect to the vehicle horizontal plane RP. The sum of the upper limit angle θu and the lower limit angle θd corresponds to the vertical angle of view. The bisector of the interior angle formed by the upper boundary Bu and the lower boundary Bd corresponds to the optical axis. The camera 2 is designed to have an upper limit angle θu of 10° or more so that it can capture an image of the sky. In addition, the camera 2 is designed to have a lower limit angle θd of 20° or more so as to be able to image the road surface. Such a mounting posture of the camera 2 corresponds to a posture in which an upward direction of 10° or more from the horizontal plane of the vehicle is included in the imaging range. Moreover, the mounting attitude described above corresponds to an attitude in which the imaging range includes a downward direction of 20° or more from the horizontal plane of the vehicle.

The camera 2 outputs a video signal based on the captured image to the camera ECU 1 . The frame rate of camera 2 is set to, for example, 30 fps (frames per second). Of course, the frame rate of camera 2 may be 60 fps or the like. The frame rate of the camera 2 may be configured to be dynamically changeable based on an instruction signal from the camera ECU 1 . A video signal output by the camera 2 corresponds to a set of temporally continuous image frames. The image frame size is, for example, 1920×1080 pixels. Of course, the size of the image frame may be 2880×1860 pixels, or the like.

The input device 3 is a device that receives a user's instruction to the camera ECU 1 . The input device 3 is a switch for instructing the camera ECU 1 to start/stop recording. The input device 3 is provided on the steering wheel of the host vehicle as one of steering switches, for example. The input device 3 outputs to the camera ECU 1 an operation signal indicating the content of the operation performed by the user. The input device 3 may be a hardware switch provided on the instrument panel, or may be a touch panel laminated on the display. The user here may be an occupant in the driver's seat (that is, a driver) or an occupant in the passenger's seat.

The in-vehicle sensor 4 is a sensor that detects state quantities related to travel control of the own vehicle. The in-vehicle sensor 4 includes a vehicle speed sensor, a steering angle sensor, an acceleration sensor, a yaw rate sensor, and the like. A vehicle speed sensor is a sensor that detects the running speed of the host vehicle. The steering angle sensor is a sensor that detects the rotation angle of the steering wheel (so-called steering angle). The acceleration sensor is a sensor that detects acceleration acting on the host vehicle in at least one of the longitudinal direction, the lateral direction, and the vertical direction. Here, it is assumed that a triaxial acceleration sensor is employed as the acceleration sensor. The detected value of the acceleration sensor can be used as a material for judging the attitude of the vehicle with respect to the horizontal plane. A yaw rate sensor is a sensor that detects a yaw rate acting on the host vehicle. Note that the type of sensor used by the camera image processing system Sys as the in-vehicle sensor 4 may be appropriately designed, and it is not necessary to include all the sensors described above. The vehicle-mounted sensors 4 can also include a direction sensor, a rain sensor, an illuminance sensor, a GNSS (Global Navigation Satellite System) receiver, a collision detection sensor, and the like. Each sensor outputs to the camera ECU 1 data indicating the current value of the physical state quantity to be detected (that is, the detection result).

The azimuth sensor is a sensor that detects an azimuth angle corresponding to the traveling direction of the vehicle. A geomagnetic sensor can be employed as the direction sensor. A rain sensor is a sensor that detects rainfall. The illuminance sensor is a sensor that detects the brightness (illuminance) outside the vehicle. The GNSS receiver is a device that sequentially detects the position coordinates of the GNSS receiver (for example, every 100 milliseconds) by receiving navigation signals transmitted from positioning satellites that constitute the GNSS. As GNSS, GPS (Global Positioning System), GLONASS, Galileo, IRNSS, QZSS, Beidou, etc. can be adopted.

In-vehicle ECU5 is ECU other than camera ECU1. The in-vehicle ECU 5 is, for example, a communication ECU, a driving support ECU, or the like. The communication ECU is an ECU that controls wireless communication between the vehicle and the outside. The communication ECU can include, for example, a communication module for implementing cellular communication such as 4G and 5G, and a communication module for implementing Wi-Fi (registered trademark) communication. The communication ECU can acquire data stored in the storage memory 15 from the camera ECU 1 and transfer the data to a predetermined server or user terminal. The transfer of the saved data by the communication ECU may be executed based on a user's operation, or may be automatically executed based on the satisfaction of a predetermined transfer condition. The transfer condition can be, for example, turning off the running power supply, the fact that the free space of the storage memory 15 has become equal to or less than a predetermined value, or the detection of a collision or the like.

The driving assistance ECU is an ECU that executes processing for assisting the driving operation of the driver. The driving ECU can automatically execute part or all of driving control related to acceleration/deceleration and steering on behalf of the driver based on the driver's instruction to execute the control. Note that the driving support ECU may have a function as a so-called automatic operation device that executes automatic driving. The driving support ECU can notify the driver of a collision with another moving object or stationary object based on the recognition result of the surrounding object by the camera ECU 1 . Other moving bodies refer to pedestrians, other vehicles, cyclists, and the like. For example, when turning right or left at an intersection, leaving a parking lot, or parking at low speed, the driver is notified of the presence of a pedestrian or the like that may come into contact with the driver by outputting sound from a speaker or displaying an image on the display. The driving assistance ECU may have a function as a parking assistance ECU that assists/automatically executes travel control for parking.

<Functions of camera ECU>
The camera ECU 1 includes various functional units shown in FIG. That is, it includes an image acquisition section F1, an operation reception section F2, a vehicle state acquisition section F3, and a data processing section F4. Some or all of these functional units can be realized by the processor 11 executing programs stored in the storage 13 . Of course, some of these functional units may be implemented using hardware such as an IC (Integrated Circuit).

The video acquisition unit F1 acquires video signals from the camera 2 . The image acquisition unit F1 converts the image signal input from the camera 2 into digital image data in a predetermined data format, and outputs the digital image data to the data processing unit F4. Based on an input signal from the input device 3, the operation reception unit F2 receives user operations such as start and stop of image recording. The operation reception unit F2 may be configured to be able to receive an instruction operation for transferring data stored in the storage memory 15 to a predetermined device based on a signal from the input device 3 .

The vehicle state acquisition unit F3 acquires vehicle information, which is information indicating the behavior and state of the vehicle, such as vehicle speed, steering angle, and yaw rate, from the in-vehicle sensor 4 . The vehicle information includes the operating state of the direction indicators, the operating state of the wipers, and the shift position. As a more preferable mode, the vehicle state acquisition unit F3 may detect changes (transitions) in driving scenes, such as turning left or right, stopping, starting, parking, etc., based on the input from the vehicle-mounted sensor 4 and the image recognition result. . A change point in the driving scene can be used as a trigger for creating a new data set to be linked with the partial images PI.

The data processing unit F4 is a module that performs various processes using the camera image input from the video acquisition unit F1. The data processing unit F4 includes, as sub-function units, a base image acquisition unit F41, a region-of-interest extraction unit F42, a connection processing unit F43, a storage processing unit F44, and a recognition processing unit F45. The processor 11/circuit module functioning as the data processing unit F4 corresponds to the vehicle recording device.

The base image acquisition unit F41 extracts image frames at predetermined sampling intervals from the video data acquired by the video acquisition unit F1 as base image data. A base image is an image used in the region-of-interest clipping unit F42 and the like, and refers to data that is the source of a partial image PI and a connected image CI, which will be described later. A base image can be understood as an image for recording or an image for generating a connected image/partial image. In one aspect, the process of extracting image frames from video data at sampling intervals can be understood as downsampling or thinning.

The sampling interval is set to a necessary and sufficient length, such as 1 second, 2 seconds, or 10 seconds, for confirming the state of a predetermined target of interest. Of course, depending on the type/characteristics of interest, the sampling interval can be set to 100 ms, 250 ms, 500 seconds, or less than 1 second. The sampling interval can be set to M times the imaging cycle of the camera 2 . M is a parameter corresponding to the downsampling rate. M is an integer of 2 or more, and is set to 5, 10, or 25, for example. The imaging cycle is an interval at which imaging is performed, and corresponds to the reciprocal of the frame rate. A shooting cycle corresponding to 30 fps is about 33 milliseconds.

The objects of interest include, for example, sky conditions (that is, weather conditions) and road surface conditions. The weather does not change suddenly in a short time such as one second. Therefore, if weather conditions are of interest, the sampling interval may be set to 2 seconds, 4 seconds, 10 seconds, and so on. In other words, the downsampling rate M is set to 100, 200, 500, and so on.

In addition, when the target of interest is the road surface condition, more specifically, an element derived from the weather such as snow cover, puddles, and presence or absence of sand cover, the sampling interval can be set to 1 second, 2 seconds, or the like. Furthermore, if the object of interest is the presence or absence of a preceding vehicle or the inter-vehicle distance to the preceding vehicle, the sampling interval can be set to 500 milliseconds, 1 second, 2 seconds, or the like.

If the object of interest is the recognition state of the lane markings, the sampling interval can be set to 200 milliseconds, 400 milliseconds, 1 second, or the like. Similarly, when the object of interest is the recognition state of an object that exists within a predetermined distance from the own vehicle, such as a preceding vehicle or a pedestrian, the time is set to 200 milliseconds, 400 milliseconds, 1 second, or the like. The sampling interval may be dynamically adjusted according to the running speed of the host vehicle. The sampling interval may be set shorter as the travel speed increases.

The region-of-interest clipping unit F42 clips a region preset as a clipping region CR as a partial image PI in the image frame for recording acquired by the base image acquiring unit F41. Either the vertical direction or the horizontal direction can be selectively adopted as a method of cutting out the partial image PI as shown in FIG. (A) of FIG. 4 shows a mode in which the partial image PI is cut out in the horizontal direction, and (B) in FIG. 4 shows a mode in which the partial image PI is cut out in the vertical direction. The arrows shown in FIG. 4 indicate the extraction direction, and the dashed lines conceptually indicate the regions that are actually extracted. A partial area is extracted so that the length in either the horizontal direction or the vertical direction is the same as that of the original frame.

The cropping direction is selected in consideration of the range in which the object of interest is captured. The cutting direction is basically horizontal. However, the extraction direction can be vertical depending on the purpose/object of interest. The cutout direction and cutout area can be specified by the operator via the recording setting screen. The record setting screen is a screen for changing the specifications related to recording, and is displayed on the in-vehicle display or the like based on the signal from the input device 3 . The extraction direction is constant in one recording process.

Cutting out the partial image PI from the image frame in the horizontal direction corresponds to cutting out the image on a line-by-line basis. One row in the image frame corresponds to a group of pixels arranged in one column in the horizontal direction. It should be noted that cutting out in units of lines also includes cutting out a plurality of lines at once. The horizontal length of the partial image PI when the partial image PI is cut out in the horizontal direction matches the horizontal length of the original image. When the partial image PI is cut out in the horizontal direction, the vertical length of the partial image PI can be made constant. When the partial image PI is cut out in the horizontal direction, the cutout width Wc hereinafter refers to the length of the partial image PI/cutout region CR in the vertical direction.

When the cutout direction is set to the horizontal direction, the cutout region CR is represented by the line range to be cutout. That is, the cutout region CR can be defined by a line number indicating the cutout start position and a line number indicating the cutout end position. For example, the cutout width is set to a desired size that can record the object of interest, such as 150 pixels (px), 200px, or 400px. The cutout width Wc when cutting out the partial image PI in the horizontal direction may be defined based on the frame length, which is the length of the image frame in the vertical direction. For example, the cutout width Wc can be set to 5%, 10%, 15%, or the like of the vertical length. As another aspect, the cut-out width Wc may be dynamically adjusted according to the driving scene and the recognition state of the predetermined object. The cutout width Wc can be expressed in pixels.

Cutting out the partial image PI from the original image frame in the vertical direction corresponds to cutting out the image in units of columns. One column in the image frame corresponds to a group of pixels arranged in one column in the vertical direction. It should be noted that cutting out on a column-by-column basis also includes cutting out a plurality of columns at once. The cutout width Wc when the partial image PI is cut out in the vertical direction corresponds to the length of the partial image PI in the horizontal direction. The vertical length of the partial image PI when the partial image PI is cut out in the vertical direction matches the vertical length of the original image. When the partial image PI is cut out in the vertical direction, the cutout width Wc hereinafter refers to the length of the partial image PI/cutout region CR in the horizontal direction.

The cutout width Wc for cutting out the partial image PI in the vertical direction can also be made constant. When the cutout direction is the vertical direction, the cutout region CR is represented by the column range to be cutout. That is, the cutout region CR can be defined by a column number indicating the cutout start position and a column number indicating the cutout end position. Alternatively, the lateral length may be dynamically adjusted. The cutout width Wc for cutting out the partial image PI in the vertical direction may be dynamically determined based on the frame width, which is the length of the image frame in the horizontal direction. For example, the cutout width Wc can be set to 5%, 10%, 15%, etc. of the frame width.

The cropped region CR is set so as to encompass the range in which the object of interest is expected to appear. For example, if the target of interest is a sky pattern, the cutout region CR is set above the center of the image frame. Also, when the target of interest is the road surface condition, the cutout region CR is set below the center of the image frame. The cropped region CR can be changed according to the mounting orientation of the camera 2 . The cropped region CR corresponds to an image region in which a checker who checks the recorded data is interested, and therefore can be called a region of interest or a region of interest. The cutout width Wc can be set so that the total amount of data recorded in a given period of time does not exceed the capacity of the storage memory 15 .

The connection processing unit F43 generates a connection image CI by connecting the partial images PI cut out by the region of interest cutout unit F42 in a direction orthogonal to the cutout direction. In the present disclosure, the direction orthogonal to the cutting direction is also referred to as the connecting direction. For example, as shown in FIG. 5, when the cropping direction is horizontal, the cropped partial images PI are vertically connected in chronological order. The connection length Lc, which is the length of the connection image CI in the connection direction, is updated each time a new partial image PI is cut out.

When the number of connections, which is the number of partial images PI forming the connected image CI, reaches or exceeds a predetermined value, the connection processing unit F43 creates a new connected image CI starting with the next provided partial image PI. to create The connection processing unit F43 may create a new connection image CI based on the fact that the connection length Lc has become equal to or greater than a predetermined value. Generating a new connected image CI corresponds to saving the partial images PI extracted thereafter as a separate file (data set). A corner of each partial image PI forming the combined image CI may be provided with a time stamp indicating the shooting time as shown in FIG. The position to which the time stamp is assigned in the partial image PI may be configured so that the operator can specify it via the recording setting screen. Such a configuration can reduce the risk of timestamps overlapping objects of interest. Note that the shooting time information for each partial image PI may be attached to the image itself as metadata.

The storage processing unit F44 executes processing for storing the connected image CI generated by the connection processing unit F43 in the storage memory 15 periodically/at a predetermined timing. For example, the connection processing unit F43 stores the connected image CI in the storage memory 15 at the timing when the number of connections or the connection length Lc becomes equal to or greater than a predetermined value. Further, the storage processing unit F44 may store the connected image CI generated during the trip in the storage memory 15 based on the turning off of the traveling power source. The running power source here refers to a power source that turns on in accordance with the operation of the power source/driving source for moving the driving source. For example, if the own vehicle is an engine vehicle, the ignition power supply corresponds to the power supply for running, and if the own vehicle is an electric vehicle, the system main relay corresponds to the power supply for running. The concept of an electric vehicle includes electric vehicles, battery-powered vehicles, hybrid vehicles, plug-in hybrid vehicles, and other vehicles that have both a motor and an engine as drive sources. A trip refers to a series of runs from when the power source for running is turned on until it is turned off.

In this embodiment, as an example, the camera ECU 1 includes the storage memory 15 as a storage destination of the connected image CI, but the present invention is not limited to this. The storage memory 15 may be provided in another ECU. Also, the storage memory 15 may be provided in the server. That is, the storage destination of the connected image CI may be a server/database provided outside the vehicle. In this case, the storage processing unit F44 cooperates with a communication ECU or the like to transmit the connected image CI to an external server and execute processing for storing it in a predetermined database.

The recognition processing unit F45 is configured to detect the position and type of a predetermined detection target by analyzing the image input from the camera 2 . The recognition processing unit F45 has a function as a discriminator that discriminates the type of object based on, for example, the feature amount vector of the image. The recognition processing unit F45 identifies an object using CNN (Convolutional Neural Network) or DNN (Deep Neural Network) technology to which deep learning is applied, for example. Objects to be detected include pedestrians, other vehicles, road markings, road edges, and the like. Road markings include lane markings, stop lines, arrow lines that indicate the direction of travel at intersections, and the like. Further, the detection targets may include traffic signs such as direction signboards, and structures attached to roads such as traffic lights. The recognition processing unit F45 can also specify the inter-vehicle distance from the preceding vehicle when the preceding vehicle is recognized. Note that the detection results of millimeter wave radar, LiDAR, or sonar may be used for the recognition of the preceding vehicle and the estimation of the inter-vehicle distance. The term "preceding vehicle" as used herein refers to a vehicle closest to the own vehicle among other vehicles traveling in front of the own vehicle on the lane in which the own vehicle is traveling. The recognition result of the recognition processing unit F45 is output to, for example, a driving support ECU.

FIG. 7 summarizes the operation of generating a connected image in the data processing unit F4, and can include steps S11 to S15. Step S11 is a step in which the base image acquisition unit F41 sequentially acquires image frames generated based on video signals as image data for generating a connected image at predetermined sampling intervals. Step S12 is a step in which the region-of-interest clipping unit F42 clips a partial image PI from the image frame acquired in step S11. Step S13 is a step in which the connection processing unit F43 executes processing for connecting the partial images PI.

Step S14 is a step of determining whether or not a saving execution condition, which is a condition for saving the connected image CI, is satisfied. The condition for executing saving of the connected image CI is that the number of connections/connection length Lc is equal to or greater than a predetermined value, that the power source for running is turned off, and that an instruction operation to end recording is performed. can be adopted. Step S15 is a step in which the saving processing unit F44 saves the connected image CI generated at that time in the saving memory 15 when the saving execution condition is satisfied.

<Specific example (1)>
Here, an example of recording settings and the operation of the data processing unit F4 for the purpose of confirming temporal changes in weather conditions (sky conditions) will be described with reference to FIGS. 8 and 9. FIG. FIG. 8 shows image frames with different photographing times extracted at predetermined sampling intervals, and FIG. 9 shows an example of a connected image CI generated based on these. FIG. 9 is shown enlarged from the original size for the sake of visibility of the figure. As mentioned above, weather conditions do not change by seconds, so the sampling interval can be set to 10 seconds. "t" shown in FIG. 8, etc., indicates the elapsed time from the start of recording, and the unit is seconds.

The dashed lines superimposed on each image frame in FIG. 8 indicate the cutout region CR, which is not included in the actual image frame. The cropped region CR is set near the upper end so that weather conditions can be extracted. The cutout width Wc can be set to 180px, for example. Although FIG. 8 shows the case where the cutout region CR is set downward from the upper end of the image frame by a predetermined amount, the cutout region CR may of course be set along the upper end of the image frame.

According to such an embodiment, as shown in FIG. 9, image data representing weather conditions at a plurality of points in time can be obtained as a connected image CI. According to the connected image CI, changes in weather conditions before and after occurrence of an important event that is separately detected can be grasped at a glance. Important events include contact with other mobile/stationary objects, as well as sudden steering and sudden braking. Also, the vehicle may be a vehicle equipped with an automatic operation device. An automatic operation device is a device that autonomously drives a vehicle based on the recognition results of peripheral monitoring sensors such as cameras and LiDAR. LiDAR stands for Light Detection and Ranging or Laser Imaging Detection and Ranging. Important events related to automatic driving include termination of the automatic driving mode, execution of notification processing for transferring driving authority from the automatic driving system to the driver, start of MRM (Minimal Risk Maneuver), and the like. A connected image CI generated by the camera ECU 1 of the present disclosure can be employed as a record indicating the situation during automatic driving.

A sharp steering wheel corresponds to a yaw rate/steering speed exceeding a predetermined value. Sudden braking corresponds to deceleration exceeding a predetermined value. MRM is vehicle control that is executed when the driver is unable to drive in a situation where the driver should be driving. The specific contents of the MRM can be, for example, processing to stop the vehicle after autonomously driving the vehicle to a safe place while issuing an alarm to the surroundings. Safe places include road shoulders with a width greater than or equal to a predetermined value, zebra zones, places defined as emergency evacuation areas, and the like. Note that the content of the RM may be a slow deceleration to stop in the current lane.

<Specific example (2)>
Next, an operation setting example of the data processing unit F4 and its operation when the purpose is to check the time change of the road surface condition such as whether the road surface is wet or not will be described with reference to FIGS. 10 and 11. FIG. FIG. 10 shows image frames with different shooting times extracted at intervals of 10 seconds, and FIG. 11 shows an example of a connected image CI generated based on these. As in FIG. 9, FIG. 11 is also shown enlarged from the original size for the sake of visibility of the figure.

The dashed lines superimposed on each image frame in FIG. 10 indicate the cutout region CR, which is not included in the actual image frame. The cropped region CR is set below the center of the image frame so that the road surface condition as an object of interest can be recorded. The cutout width Wc can be set to 360px, for example. The position of the cutout region CR in the vertical direction can be appropriately adjusted according to the imaging direction of the camera 2 . The cut-out region CR is set to correspond to a road surface range from a point a first distance ahead of the front end of the vehicle to a point a second distance ahead. For example, the first distance is set to 8m and the second distance is set to 12m. With this setting, road surface images from 8m to 12m ahead of the vehicle are recorded. It should be noted that if the cutout region CR is too low, the body of the vehicle may be reflected depending on the mounting position of the camera 2 or the like. Therefore, the cutout region CR is preferably arranged at a position above the lower end of the image frame by a predetermined amount. Also, if the cutout region CR is too high, it becomes difficult for the preceding vehicle or the like to record the road surface condition. Therefore, the cutout region CR is preferably arranged at a position lower than the center of the image frame by a predetermined amount.

According to such an embodiment, as shown in FIG. 11, image data collectively showing road surface conditions at a plurality of points in time can be obtained as a connected image CI. According to the connected image CI, it is possible to grasp the temporal change of the road surface condition within a predetermined time before and after the event occurrence time at a glance.

<Specific example (3)>
In the specific examples (1) and (2) above, the partial image PI is cut out in the horizontal direction. can be Here, the operation when the region-of-interest clipping unit F42 clips an image in the vertical direction will be described with reference to FIGS. 12 and 13. FIG. FIG. 12 shows image frames extracted at intervals of 10 seconds in the same manner as in FIGS. 8 and 10 and taken at different times. shows an example. As in FIG. 9, FIG. 13 is also shown enlarged from the original size for the sake of visibility of the figure.

The dashed lines superimposed on each image frame in FIG. 12 indicate the cutout region CR, which is not included in the actual image frame. The cropped region CR is set so as to pass through the center of the image frame. The cutout width Wc is set to, for example, 340px. Of course, the cutout width Wc may be 170px or the like. The cutout region CR is set to divide the image frame into left and right. Note that FIG. 10 shows the case where the cutout region CR is set so as to pass through the center of the image frame, but of course the cutout region CR is set at a position shifted left or right from the center of the image frame by a predetermined amount. Also good.

According to such recording settings, as shown in FIG. 11, image data collectively showing road surface conditions and weather conditions at a plurality of points in time can be obtained as a connected image CI. According to the connected image CI, it is possible to grasp, at a glance, temporal changes in the road surface condition and the weather condition within a predetermined time before and after the event occurrence time.

Moreover, according to the configuration in which the cutout region CR is a predetermined row range corresponding to the front direction of the own vehicle, it is possible to record the presence or absence of the preceding vehicle and the inter-vehicle distance. According to the above setting, it is possible to grasp at a glance the change over time of the situation in the front direction of the own vehicle, such as the presence or absence of a preceding vehicle within a predetermined time before or after the event occurrence time. Based on the above effect, if the preceding vehicle is the target of interest, it is preferable to set the cutout direction to the vertical direction and set the horizontal position of the cutout region CR so as to include the center of the frame. It can be said.

By the way, the presence or absence of a preceding vehicle, the inter-vehicle distance, and the vehicle corresponding to the preceding vehicle can change dynamically compared to weather conditions. Therefore, when the object of interest is the preceding vehicle, the sampling interval is preferably set in units of seconds such as 1 second or 2 seconds. Of course, when the object of interest is the preceding vehicle, the sampling interval may be 1 second or less, such as 250 milliseconds or 500 milliseconds. The cutout width Wc when the object of interest is the preceding vehicle may also be set so as to be able to record desired information as appropriate.

According to the configuration described above, since only a part of the image frames generated by the camera 2 is saved, the data size saved in the saving memory 15 can be reduced. Accordingly, the requirements for the storage memory 15 can be relaxed. In addition, since the number of times data is stored in the storage memory 15 can be suppressed, the usable period (so-called life) of the memory can be extended. Furthermore, since the situation at each time can be viewed with one image, there is no need to spend time checking the recorded video data. In other words, it is possible to improve the efficiency of the work of confirming the time-dependent change in the situation of the object of interest. Further, the camera ECU 1 corresponds to a kind of so-called constant recording type recorder that records images at predetermined sampling intervals instead of the event recording type. Therefore, it is possible to reduce the risk that the target of interest is not recorded compared to an event recording type recorder.

Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and various modifications described below are also included in the technical scope of the present disclosure. Various modifications can be made without departing from the scope of the present invention. For example, the following various supplements and modifications can be implemented in combination as appropriate within a range that does not cause technical contradiction. It should be noted that members having the same functions as those of the members described above are given the same reference numerals, and explanation thereof may be omitted. Also, when only a part of the configuration is mentioned, the above description can be applied to the other parts.

<Modification (1)>
Although the case where the camera 2 is a camera that captures an image in front of the vehicle has been described above, the present invention is not limited to this. The camera 2 may be a camera that captures images of the rear of the vehicle, or may be a camera that captures images of the sides of the vehicle.

Further, it may be a driver monitoring camera positioned to capture an image of the driver's face. A driver here mainly refers to a person sitting in a driver's seat of a vehicle. In addition, a person (so-called operator) sitting in a remote control seat outside the vehicle can also be included in the concept of a driver. Driver monitoring cameras are arranged, for example, on the upper surface of the steering column cover, the upper surface of the instrument panel, the rearview mirror, the upper end of the windshield, and the like.

If the camera 2 is a driver monitoring camera, the driver's eyes may be set as the object of interest. The position of the cutout region CR in the vertical direction is adjusted manually or automatically so as to include the driver's eye position. For example, when the recognition processing unit F45 detects the driver's eyes, the height position of the cutout region CR can be dynamically adjusted so as to correspond to the detected positions of the driver's eyes. According to the configuration for extracting an image region including the driver's eye area as a partial image PI and storing it as a connected image CI, it is possible to record changes over time in the arousal state of the driver. Along with this, it becomes possible to easily verify the transition of the driver's arousal level before and after the occurrence of a serious event such as an accident. The sampling interval when the eye opening degree of the driver is to be recorded (object of interest) can be set to 1 second, 2 seconds, 4 seconds, or the like.

<Modification (2)>
The camera ECU 1 may generate the connected image CI based on an image in which a marker indicating the recognition result of the recognition processing unit F45 is superimposed on the image captured by the camera 2 . The camera ECU 1 may include an image processing unit F46 shown in FIG. 14 so as to correspond to the technical concept. The image processing unit F46 is configured to generate a synthesized image by adding a recognition result, a time stamp, etc. to the image of the camera 2 . Further, the image processing unit F46 may have a function of performing distortion correction, keystone correction, etc. according to the lens characteristics on the image captured by the camera 2 . The base image acquisition unit F41 can acquire the processed image generated by the image processing unit F46 as a base image for generating a connected image. The image processing unit F46 can also be called an image synthesizing unit or a video processing unit.

Such an image processing section F46 may be integrated with the base image acquisition section F41 as one sub-functional section of the base image acquisition section F41. Acquisition here also includes generation/detection by internal computation. Also, the image processing unit F46 may be provided in the video acquisition unit F1. Furthermore, the image processing section F46 may be interposed between the video acquisition section F1 and the base image acquisition section F41.

The image processing unit F46 generates an image in which, for example, an image frame input from the camera 2 is superimposed with another vehicle marker for emphasizing the other vehicle. The other vehicle marker is a rectangular frame line and is arranged so as to surround the other vehicle recognized by the recognition processing unit F45. The other vehicle to which the other vehicle marker is added may be only the preceding vehicle or only the oncoming vehicle. The image processing unit F46 may be configured to add the other vehicle marker only to the other vehicle for which the separately calculated collision risk is equal to or greater than a predetermined value. Another vehicle with a collision risk greater than or equal to a predetermined value refers to a vehicle whose collision remaining time (so-called TTC: Time-To-Collision) or collision margin (MTC: Margin-To-Collision) is less than or equal to a predetermined value. .

The recognition processing unit F45 can recognize not only other vehicles but also pedestrians and stationary objects. The image processing unit F46 may generate an image in which a pedestrian or a predetermined stationary object is marked with a marker. The image processing unit F46 may also generate an image in which the lane recognition line is superimposed on the image frame input from the camera 2 . A lane recognition line is a linear marker indicating the position of a lane division line recognized by the recognition processing unit F45.

FIG. 15 conceptually shows the operation of the camera ECU 1 in this modified example. As shown in FIG. is added at predetermined sampling intervals. Also, the region-of-interest clipping unit F42 in this modified example clips a predetermined region in the composite image frame that may include a marker, as shown in FIG. 15B. Of course, the presence or absence of a marker is affected by the presence or absence of a recognition target object, so it is possible that the image frame from which the partial image PI is extracted does not include the marker. The connection processing unit F43 generates a connection image CI based on a plurality of partial images PI, as shown in FIG. 15C, as in the above-described embodiment. With such a configuration, it is possible to record the recognition status of the recognition processing unit F45 for a predetermined object.

16 and 17 are diagrams for explaining the operation of the camera ECU 1 when the recognition status of lane markings at a point ahead of the vehicle at a predetermined verification distance is set as the target of interest. The expression "point" here includes the concept of a section or area having a given length. FIG. 16 shows image frames at a plurality of time points on which markers Mk as lane recognition lines are superimposed, and FIG. 17 shows an example of a connected image CI formed by connecting these partial images PI in the horizontal direction. there is

The verification distance may be a value corresponding to a relatively short distance, such as 8m, 10m, or 16m, or a value corresponding to a relatively long distance, such as 25m, 30m, or 40m. According to the above configuration, it is possible to grasp, at a glance, changes in recognition results that are of interest to the developer/verifier. In addition, it is possible to easily find a spot where the lane marking has failed to be recognized even though the lane marking is shown. The sampling interval may be 10 seconds or the like, or may be several seconds such as 1 second or 2 seconds. Furthermore, the sampling interval may be 250 milliseconds, 500 milliseconds, or the like. The cutout width Wc can be 180px, 360px, or the like.

<Application example of modification (2)>
The region-of-interest clipping unit F42 may dynamically change the clipping width Wc according to the recognition result of the object set as the target of interest. For example, when the preceding vehicle is set as both the recognition target and the interest target, the region-of-interest extraction unit F42 dynamically sets the extraction width Wc so as to include the preceding vehicle when the preceding vehicle is recognized. Also good. FIG. 18 is a diagram conceptually showing the operation of the data processing section F4 when dynamically adjusting the cutting width Wc so as to include the preceding vehicle.

According to this configuration, it is possible to grasp the time change of the recognition target (for example, preceding vehicle) of interest to the developer at a glance. Also, it is possible to efficiently confirm the accuracy of the recognition logic. For example, it becomes easier to identify scenes where the preceding vehicle is likely to be lost. Note that even in the configuration for dynamically adjusting the cutout width Wc, the basic cutout width Wc and the cutout region CR are set. The region-of-interest clipping unit F42 adjusts the clipping width Wc so as to include a predetermined basic setting range of the clipping region CR. The default range refers to the row range/column range that should be cut out if there is no recognition target. In the configuration in which the partial image PI is cut out in the horizontal direction, adjusting the cutout region CR to include the recognition result means adjusting the cutout start position upward and the cutout end position downward as necessary. equivalent to The cutout start position indicates the uppermost line number of the cutout region CR, and the cutout end position indicates the lowermost line number of the cutout region CR.

<Modification (3)>
The camera image processing system Sys may include a plurality of cameras 2 that capture images of the outside of the vehicle. For example, as shown in FIG. 19, the camera ECU 1 is connected with a front camera 2A, a rear camera 2B, a left camera 2C, and a right camera 2D as cameras 2 . These four cameras 2 are arranged at different positions on the own vehicle and photograph different directions around the own vehicle. Specifically, it is as follows.

The front camera 2A is a camera that captures an image of the front of the vehicle with a predetermined angle of view. The front camera 2A is mounted, for example, at the front end of the vehicle, such as the front grille, with its optical axis facing the front of the vehicle. The rear camera 2B is a camera that captures an image of the rear of the vehicle with a predetermined angle of view. The rear camera 2B is arranged at a predetermined position on the rear surface of the vehicle body, such as near the rear license plate or near the rear window, with the optical axis directed toward the rear of the vehicle. The left camera 2C is a camera that captures an image of the left side of the own vehicle. The left camera 2C is mounted, for example, on the left side mirror or the left pillar, with the optical axis directed toward the left side of the vehicle. The right camera 2D is a camera that captures an image of the right side of the vehicle. The right camera 2D is attached to the right side mirror, the right pillar, or the like, with its optical axis facing the right side of the vehicle. Of course, the mounting positions and mounting postures of the respective cameras 2 are examples and can be changed as appropriate.

A wide-angle lens such as a fish-eye lens is adopted as the lens of these cameras 2, and each camera 2 has an angle of view of 180 degrees or more. Therefore, by using the four cameras 2, it is possible to photograph the entire surroundings of the own vehicle. The camera imaging system Sys may comprise a camera 2 mounted on the roof. The mounting positions of the left camera 2C and the right camera 2D described above are not limited to the side mirrors, and may be a rooftop or the like. Some or all of the plurality of cameras 2 may be cameras retrofitted, for example, on the roof, on the dashboard, near window frames, or the like. A plurality of cameras 2 each output a video signal based on a captured image to the camera ECU 1 . The data processing unit F4 of the camera ECU 1 may generate the above-described connected image CI for each camera 2 individually. Since the imaging range differs for each camera 2 , parameters such as the cutout direction and cutout width Wc may differ for each camera 2 .

<Application example (1) of modification (3)>
The data processing unit F4 may be configured to extract a partial image PI based on an image obtained by integrating images captured by a plurality of cameras 2 and generate a connected image CI. As a premise, the data processing unit F4 includes an image processing unit F46. FIG. 20 is a diagram showing an outline of the operation of the camera ECU 1 in this application example. As shown in FIG. 20A, the image processing unit F46 extracts images captured by each camera 2 at common sampling intervals based on video signals provided from each of the plurality of cameras 2 . The image frames extracted from each camera 2 are captured images at approximately the same time.

Next, as shown in FIG. 20B, the image processing unit F46 combines the image frames captured by the cameras 2 at the same point in time in a predetermined order. FIG. 20 shows a mode of combining image frames in the horizontal direction. As another aspect, the image processing unit F46 may combine image frames in the vertical direction. For convenience, an image frame formed by connecting image frames captured by a plurality of cameras 2 will be referred to as a connected frame. The direction in which the image frames are connected is the same as the extraction direction. The connected frame generated by the image processing unit F46 is provided to the base image acquisition unit F41. That is, the base image acquisition unit F41 acquires the connected frames at predetermined sampling intervals from the image processing unit F46.

Then, as shown in FIG. 20B, the region-of-interest clipping unit F42 clips a portion set as a predetermined clipping region CR in the concatenated frame, thereby obtaining a portion from which the concatenated image CI is based. Generate an image PI. A partial image PI corresponds to a part of the image captured by each camera 2 .

The connection processing unit F43 adds the newly generated partial image PI to the end of the existing connected image CI, for example, each time the region of interest cutout unit F42 generates a partial image PI. Note that the connection processing unit F43 may be configured to collectively connect a plurality of partial images PI in chronological order at a predetermined timing.

As described above, the image acquired by the base image acquisition unit F41 may be a combination of images acquired from a plurality of cameras. According to the above configuration, it is possible to grasp the temporal change of image data captured by a plurality of cameras at a glance. Targets of interest in this modified example/application example are assumed to be the road surface condition, the distance to surrounding vehicles, and the recognition state of pedestrians and the like.

By the way, in FIG. 20, the aspect of extracting the partial image PI after connecting the image frames of each camera 2 has been described, but the processing procedure is not limited to this. As shown in FIG. 21, the region-of-interest extraction unit F42 extracts partial images PI of the same size from each camera image, and the connection processing unit F43 connects them in a predetermined order in the extraction direction to obtain a final portion. A camera-integrated partial image CPI, which is the image PI, may be generated. The connection processing unit F43 generates a connected image CI by connecting the sequentially generated camera-integrated partial images CPI in chronological order in the connecting direction.

Note that the cropped region CR may be different for each camera 2 as indicated by the dashed line in FIG. 21(B). This is because the area in which the target of interest is captured may differ for each camera 2 . However, in order to combine the images, a common value/direction is applied to the clipping width Wc and the clipping direction. According to this modified example/application example, it is possible to store an image region that may include an object of interest more efficiently.

<Application example (2) of modification (3)>
The image processing unit F46 may have a function of generating a bird's-eye view image based on images from a plurality of cameras 2, as shown in FIG. A bird's-eye view image is an image looking down around the vehicle from a virtual viewpoint set in the sky. The partial image PI forming the connected image CI may be a predetermined area of the bird's eye image. That is, the camera ECU 1 may extract predetermined regions of bird's-eye images generated based on a plurality of camera images at predetermined sampling intervals, and sequentially connect them to generate the connected image CI. In such a configuration, the base image acquisition unit F41 acquires the bird's-eye view image generated by the image processing unit F46 at predetermined sampling intervals. A plurality of bird's-eye images sequentially acquired by the base image acquisition unit F41 have different generation times at sampling intervals. The region-of-interest clipping unit F42 clips a clipping region CR preset in the bird's-eye image acquired by the base image acquiring unit F41 as a partial image PI. The connection processing unit F43 sequentially generates and updates the connection image CI by connecting the partial images PI sequentially generated by the region-of-interest extraction unit F42 in chronological order in the connection direction. According to the configuration in which the connected image CI is generated based on the bird's-eye view image in this way, it is possible to improve the recognizability of changes in road surface conditions over time.

In the bird's-eye view image, when the image range corresponding to the cropped region CR is in front of the vehicle and does not include image data derived from the rear camera 2B, the image processing unit F46 does not use the image frame of the rear camera 2B, A bird's-eye view image specialized in front of the vehicle may be generated. Further, in the bird's-eye view image, if the image range corresponding to the cutout region CR can be generated only by the front camera 2A, the image processing unit F46 may generate an image obtained by converting the viewpoint of the image of the front camera 2A. Viewpoint conversion processing is processing for converting a captured image into an image viewed from a virtual viewpoint by mapping onto a predetermined projection plane. The base image acquisition unit F41 may be configured to acquire image data obtained by subjecting an image captured by the camera 2 to viewpoint conversion processing by the image processing unit F46 as base image data from the image processing unit F46.

The technical concept of storing superimposed markers indicating recognition results can also be applied to configurations in which partial images PI are extracted based on image frames that have undergone viewpoint conversion, such as bird's-eye images, and are stored as connected images. . For example, the base image acquisition unit F41 acquires, as a base image frame, an image frame obtained by superimposing a marker indicating a recognition result on a processed image obtained by performing viewpoint conversion, and the region-of-interest extraction unit F42 extracts from the image frame A partial image PI may be extracted. The cutout width Wc may also be dynamically changed according to the recognition result.

A processed image is an image that has undergone predetermined processing such as viewpoint conversion, distortion correction, text/marker/time stamp superimposition, brightness correction, and color tone adjustment on an image captured by a single camera 2. do. It should be noted that in one embodiment, it is possible to restrictively interpret an image that has undergone viewpoint conversion and an image to which supplementary information such as a marker is added as a processed image. Also, an image obtained by combining one image with another image corresponds to a composite image. A composite image is typically a bird's-eye view image. The composite image includes not only an image obtained by combining a plurality of images output from different sources, but also an image obtained by superimposing a previously prepared image element such as a marker on one image. In other words, the synthetic image can be understood as a kind of processed image. Note that the underfloor transmission image and the like also correspond to a kind of composite image. The underfloor transmission image is an image showing the bottom of the vehicle body and the like in a transparent manner by synthesizing the past image of the front camera 2A acquired when the vehicle moves forward with the current image of another camera. The virtual viewpoint of the composite image such as the underfloor transmission image is not limited to the sky above the vehicle, and may be the driver's viewpoint, in other words, the driver's eye box.

<Modification (4)>
The camera ECU 1 may be configured to change the values of setting items related to data recording from default setting values to temporary applied values in a specific scene. The setting items related to data recording refer to the sampling period, the cutout width Wc, and the like. Default setting values refer to setting values that are applied in a normal scene. Also, the temporary application value is a set value that is applied when it is determined that the scene is an intensive recording scene. A normal scene refers to a scene other than the concentrated recording scene. An intensive recording scene corresponds to a scene in which data is stored more densely than a normal scene. A normal scene corresponds to a scene whose data storage density is sparse compared to an intensive recording scene.

Intensive recording scenes in which the recording mode is changed include (i) when passing through an intersection, (ii) when driving at low speed, (iii) when parking, and (iv) when leaving a garage. (i) When passing through an intersection can also be subdivided into three types: (a) turning right, (b) turning left, and (c) going straight. Low-speed running refers to, for example, a state in which the moving speed is less than a predetermined low-speed threshold. The low speed threshold is, for example, 10 km/h or 20 km/h. Low-speed driving can also include autonomous driving for parking. (iii) when parking and (iv) when leaving the garage can also be understood as an example of (ii) when traveling at low speed. In the present disclosure, a state in which the moving speed is equal to or higher than the low speed threshold is also referred to as a normal running state. The normal running state includes high-speed running. Note that moving at high speed refers to a state in which the vehicle exceeds a predetermined high speed threshold such as 60 km/h or 80 km/h, for example.

The temporary applied value may be any value that is different from the default set value. The temporary application value can be set to a value that allows information to be recorded more densely than in a normal scene. For example, if the default setting value for the sampling interval is 10 seconds, the temporary application value can be set to 2 seconds, 1 second, 500 milliseconds, or the like. The temporary application value over the sampling interval can be set to less than half of the default setting. Also, if the default setting value for the clipping width Wc is 180px, the temporary application value may be set to 360px or the like. The temporary application value for the cutting width Wc can be set to 1.5 times or more, more preferably 2 times or more, the default set value.

FIG. 23 is a flow chart for explaining an example of the operation of the camera ECU 1 corresponding to this modified example. The processing flow shown in FIG. 23 is started, for example, when it is detected that the power supply for running the vehicle is turned on and the user or the like performs an operation to instruct the start of data recording. Of course, in the setting of always recording when the driving power supply is turned on, the flow shown in FIG. 23 can be started based on the turning on of the driving power supply. In the following, as an example, a case in which the time of passing through an intersection is set as an intensive recording scene will be described. In parallel/independently of this flow, the processor 11 extracts the partial image PI and updates the connected image CI at set sampling intervals.

First, the processor 11 determines whether or not the scene corresponds to an intensive recording scene, based on information input from the in-vehicle sensor 4 and the in-vehicle ECU 5, such as the vehicle speed and the operation state of the direction indicator. More specifically, the processor 11 determines whether or not the vehicle is passing through an intersection based on an input signal from the in-vehicle sensor 4 or the like. Note that the result of image recognition of a feature such as a road marking can also be used as a material for scene determination. Further, the processor 11 may acquire information about the planned travel route and the travel road from the navigation device and the driving support ECU, and may determine the scene based on the information.

If it is determined in the determination process in step S21 that the vehicle is passing through the intersection, step S23 is executed. On the other hand, if it is determined that the vehicle is not passing through the intersection, step S22 is executed. Step S22 is a step of maintaining default settings. Step S23 is a step of changing various setting values related to data recording from default setting values to temporary applied values.

While the temporary application value is being adopted, the processor 11 sequentially determines in step S24 whether or not the concentrated recording scene continues based on signals from the in-vehicle sensor 4 or the like. More specifically, the processor 11 determines whether or not the vehicle is still passing through the intersection based on the vehicle speed, yaw rate, vehicle position information, and road marking recognition status.

When the intensive recording scene ends, for example, when it is detected that the user has exited the intersection, in step S25, the various set values for data recording are returned from the temporary applied values to the default set values. Step S26 is a step for determining whether or not a recording end operation has been performed, based on a signal from the input device 3 . The recording end operation refers to pressing of a switch for stopping/ending recording of data, or the like. Note that an operation to turn off the driving power source can also be included in the recording end operation. Determination of whether or not the recording end operation has been performed can be performed even during the continuation of the concentrated recording scene.

When detecting that the recording end operation has been performed, the processor 11 determines the connected image CI and transmits it to a predetermined server or the like (step S26). If the vehicle/camera ECU 1 is wired/wirelessly connected to a tablet terminal or a notebook computer, the camera ECU 1 is triggered by the detection of the recording end operation, and transmits the connected image data to the connected terminal. Also good.

According to the above configuration, in a specific scene, information of interest can be recorded more densely than in a normal scene. If the object of interest is the recognition state of another moving object when passing through an intersection, the data while passing through the intersection can be recorded at necessary and sufficient intervals, and the amount of data in scenes other than those when passing through the intersection can be further reduced. As a result, it is possible to increase the analysis efficiency of the target of interest while suppressing the amount of stored data. The above configuration corresponds to a configuration in which the data storage density is changed between specific scenes and other scenes. Moreover, the above configuration corresponds to a configuration that dynamically changes at least one of the sampling interval and the cutout width Wc according to the driving scene/vehicle speed. Configurations for changing the sampling interval can also include configurations for effectively stopping recording by setting the sampling interval to a sufficiently large value corresponding to infinity.

Further, the camera ECU 1 may be configured to generate and update the connected image CI at predetermined sampling intervals only when the scene is a specific scene designated as a scene to be recorded. In other words, the camera ECU 1 may be configured to stop extracting the partial image PI/updating the combined image CI even when the vehicle is running if the scene does not correspond to the scene to be recorded. Scenes such as those exemplified as intensive recording scenes can be used as scenes to be recorded. According to this configuration, since data is not collected except for scenes to be recorded, it is possible to further reduce the amount of data to be saved. Note that setting a scene to be recorded paradoxically corresponds to setting a recording-excluded scene in which data recording is not performed. The above configuration corresponds to a configuration in which the camera ECU 1, which is configured to be able to register recording-excluded scenes, stops data recording when a recording-excluded scene applies.

<Modification (5)>
The camera ECU 1 may have a plurality of operation modes. For example, the camera ECU 1 may be configured to be switchable among a non-recording mode, a first recording mode, and a second recording mode. The non-recording mode is an operation mode in which no image is saved. The first recording mode is an operation mode in which original image frames are saved as they are, not partial images. The first recording mode may be a mode for saving video data. The second recording mode is an operation mode for saving a plurality of partial images as a connected image. The camera ECU 1 that operates in the second recording mode can correspond to various configurations described above as embodiments/modifications.

The input device 3 in the modification may include a switch for switching operation modes. For example, input device 3 may include a toggle switch for switching between operating modes. The switch may be a mechanical switch, or may be an image button realized by cooperation between a touch panel and a display. The operation reception unit F2 receives a user instruction operation for changing the operation mode based on a signal from the input device 3 .

Note that the camera ECU 1 may automatically change the operation mode according to the free space of the storage memory 15 . For example, the camera ECU 1 may automatically change the operation mode from the first recording mode to the second recording mode when the free space of the storage memory 15 becomes less than a predetermined value. When the operation mode is automatically switched, the camera ECU 1 preferably notifies the user that the operation mode has been switched by display or sound, as described below.

The camera ECU 1 may be connected to the display device 6 and the speaker 7 as shown in FIG. 24 as devices for notifying the user of the operation mode. The display device 6 displays an icon or the like indicating the operation mode of the camera ECU 1 based on the input signal from the camera ECU 1 . The display device 6 may be a meter display or a head-up display. The meter display is a display provided in an area located in front of the driver's seat on the instrument panel, and can be realized using a liquid crystal panel, an organic light emitting (Electro Luminescence: EL) panel, or the like. A head-up display is a device that displays a virtual image at a position in front of the vehicle as viewed from the driver by projecting light representing an image (hereinafter referred to as image light) onto a projection member such as a windshield of the vehicle. An icon indicating the operating mode may be displayed at all times while the driving power supply is on. The icon indicating the operation mode may be displayed for a certain period of time after the operation mode changes.

The status of the camera ECU 1 may be notified to the driver not only by icon display but also by text display. Also, the state of the camera ECU 1 may be represented by the lighting of an LED (lighting color and presence/absence of blinking). For example, the camera ECU 1 may turn off the LED in the non-recording mode, turn on the LED in the first recording mode, and blink the LED in the second recording mode. The driver may be notified of the state of the camera ECU 1 by outputting a voice message from the speaker 7 . The camera ECU 1 may cause the speaker 7 to output an announcement/notification sound for notifying the changed operation mode when the operation mode is switched.

<Modification (6)>
The camera ECU 1, as shown in FIG. 25, may include an inspection processing unit F47 that detects an abnormality of a predetermined inspection object based on partial images. The camera ECU 1 as the inspection processing unit F47 detects an abnormality of the inspection object shown in the partial image by comparing the latest image and the past image taken at the same point.

The latest image in the present disclosure refers to a partial image newly generated on this trip. The latest image can be rephrased as the current image. A past image in the present disclosure refers to a partial image generated and saved in a trip before the previous trip. A past image generally refers to a partial image generated one day or one week or more ago.

The inspection object in the present disclosure refers to facilities/features that should be inspected by image comparison to determine whether or not there is an abnormality such as damage or deterioration. An inspection object corresponds to a specific example of an object of interest. The objects to be inspected may be various facilities installed along the road, such as road signs, road markings, traffic lights, electronic bulletin boards, utility poles, electric wires, and manholes. Road signs may be classified into side signs and upward signs according to the direction of existence as seen from the camera 2 . A side sign refers to a sign placed on the right or left side of the road in a space whose height from the road surface is less than 3 m. Side signs may include so-called roadside signs, in which sign plates are installed on poles installed at the edge of the road, the center of the road, the sidewalk, the median strip, or the like. Side signs may include some or all of regulatory signs, warning signs, instructional signs, and auxiliary signs. An upper sign refers to a type of sign placed 4.7m or more above the road surface using a pillar or gate-shaped post. Upper signs may include cantilevered or mounted signs, as well as gantry-style signs. A guide sign is a specific example of the upward sign. A part of the guide sign may be divided into side signs.

When the upper marker is set to the inspection object, the cutout direction is horizontal, and the cutout region CR is set near the upper end. That is, when the upper sign is set as the inspection object, the camera ECU 1 extracts the vicinity of the upper end of the original image frame as a partial image and links and saves it, as shown in FIG. The cut width can be set to any value. Note that, as described above, the linked storage here refers to the process of linking partial images generated at other times in chronological order and storing them.

Each partial image is stored in the storage memory 15 in association with the shooting situation data. The shooting situation data here is a data set of items indicating the situation (that is, the shooting situation) at the time of partial image acquisition. For example, as shown in FIG. 27, the photographing situation data includes data on partial image number, photographing time, photographing position, traveling direction, weather, preceding vehicle, and the like. The partial image number is an identifier indicating the number of the partial image among the partial images included in the connected image or among the partial images generated during the current trip. Note that the shooting situation data for each partial image may be stored in association with the main body of the partial image data, in which case the partial image number may be omitted.

The shooting position is the position coordinates of the shooting point and is specified based on the output signal of the GNSS receiver. The shooting position may be expressed in the world geodetic system (WGS84), or may be expressed in another coordinate system. The shooting position may be represented by two-dimensional coordinates of latitude and longitude, and may include altitude information. According to the configuration including the altitude information, it is possible to suppress erroneous identification of the photographing position in a road section having an elevated road or a double-deck structure.

The direction of travel can be represented by an azimuth angle. The traveling direction may be represented by the yaw angle of the vehicle body with respect to the extending direction of the lane markings that define the lane in which the host vehicle is traveling. The direction of travel may be detected by a direction sensor, or may be specified by image recognition. The weather information may be determined from the operating state of the rain sensor or wipers, or may be specified based on weather information received from an external device. Weather can be represented in code. A code indicating fine weather can be set to 00, a code indicating rain can be set to 01, and the like.

Further, the photographing situation data may include preceding vehicle information as information indicating whether or not there is an object obstructing the field of view of the camera 2 in front of the own vehicle. The preceding vehicle information indicates whether or not a vehicle corresponding to the preceding vehicle exists within a certain distance from the own vehicle. The presence or absence of a preceding vehicle can be represented by a flag (0/1) or code. The preceding vehicle information may include the inter-vehicle distance to the preceding vehicle. The preceding vehicle information may include type information or size information of the preceding vehicle, such as whether or not the preceding vehicle was a large vehicle. The preceding vehicle information corresponds to the presence or absence of an obstacle for camera 2/image recognition, in other words, information indicating the possibility that an object to be inspected is blocked by the preceding vehicle.

The preceding vehicle information may be a flag as a simpler configuration. The camera ECU 1 may be configured to turn on the leading vehicle flag in the photographing situation data only when a large vehicle is present within a certain distance (for example, 10 m) from the host vehicle. Such a preceding vehicle flag is a flag indicating whether the field of view of the camera 2 is good or bad, and can also be called a field of view flag or an occlusion flag.

Of course, the shooting situation data need not include all of the above items, and some items may be omitted. The shooting situation data may also include the detection value of the illuminance sensor, that is, the external illuminance. The shooting situation data may be embedded in the partial image as metadata of the partial image. The shooting situation data may be converted into text and embedded in the partial image, as shown in FIG. The shooting situation data may be saved in a separate file from the partial image. In this case, it is preferable that a part of the partial image number, the shooting time, and the shooting position coordinates are embedded in each partial image in order to facilitate association with the shooting situation data.

As shown in FIG. 28, the camera ECU 1 as the inspection processing unit F47 detects breakage of the sign as the inspection object by comparing partial images taken at the same point. FIG. 28 illustrates a case where part of the upper marker is deformed. (A) of FIG. 28 shows the past image, and (B) shows the latest image. Note that the same point here is not limited to being exactly the same, and may have an error of about 0.5 m or 1.0 m.

FIG. 29 is a flowchart for explaining an example of the operation of the camera ECU 1, and can include steps S31 to S38. The processing flow shown in FIG. 29 is performed at sampling intervals, for example, in a state where the power source for running the vehicle is on and the second recording mode is set.

Step S31 is a step in which the base image acquisition unit F41 acquires an image frame for partial image extraction based on the video signal. Step S32 is a step in which the region-of-interest clipping unit F42 clips a partial image from the image frame acquired in step S31. Step S33 is a step in which the camera ECU 1 saves the partial image acquired this time in conjunction with partial images generated at other times. At that time, the shooting situation data is linked with the partial image and stored in the storage memory 15 .

Step S33 is a step of determining whether or not the partial image generated in step S32 includes an inspection object (in this case, an upper sign). This step is carried out by the inspection processing unit F47 cooperating with the recognition processing unit F45. The partial image generated in step S32 this time corresponds to the latest image.

Step S34 corresponds to a step of determining whether or not the latest image includes an object to be inspected. Camera ECU1 ends this flow, when an inspection subject is not able to be detected from the newest picture (S34 NO). On the other hand, when the inspection object can be detected from the latest image (S34 YES), the camera ECU 1 executes subsequent processing (from step S35).

Step S35 is a step of referring to the storage memory 15 and reading a partial image captured in the past at the current position as a past image. It should be noted that this flow may be ended when no past image is found as a result of past image search, such as when the vehicle passes the current position for the first time.

Step S36 is a step of comparing the latest image and the past image and determining whether or not there is a difference in the shape of the object to be inspected. The image comparison may detect only the difference in shape, or may detect the difference in color. According to the configuration for detecting the difference in color, it may be possible to detect occurrence/expansion of rust, graffiti, and the like. Difference detection based on image comparison corresponds to processing for determining whether or not an abnormality has occurred in an inspection target using a past image as a reference (sample). Therefore, the processing can also be called image diagnosis processing.

When the camera ECU 1 detects a difference by comparing the latest image and the past image (step S37 YES), the abnormality notification process is performed (step S38). The abnormality notification process may be a process of notifying the driver of an abnormality of the inspection object using the display device 6 or the speaker 7, or may be a process of transmitting a signal indicating that an abnormality of the inspection object has been detected to a specific server by radio. It may be a process of transmitting. Abnormalities of the object to be inspected may include, in addition to breakage, adhesion of dirt, inclination exceeding a predetermined value, and the like.

Note that if the preceding vehicle flag of the latest image or past image is on, there is a possibility that the inspection object is not shown in its original form due to occlusion. Under such circumstances, the camera ECU 1 may omit the image diagnosis processing when the leading vehicle flag is turned on in either the latest image or the past image. According to this configuration, when the latest image/past image is an image in which a part of the inspection object is missing due to occlusion, it is possible to reduce the possibility of erroneously determining that the inspection object has an abnormality. On the other hand, regardless of the presence or absence of a preceding vehicle, the configuration of performing image diagnosis on all the latest images in which the object to be inspected is reflected has the advantage of suppressing the possibility of overlooking equipment that may be abnormal. . The response policy of the camera ECU 1 when the preceding vehicle flag is set to ON may be configured so that the driver can change the settings.

In addition, when the camera ECU 1 detects an equipment abnormality as a result of image diagnosis processing using the latest image or the past image in which the preceding vehicle flag is set to ON, the camera ECU 1 indicates that the diagnosis result is an uncertain diagnosis result. , may be notified to the driver/server together with the diagnosis result.

In the above description, the aspect of linking and saving the partial images regardless of whether or not the inspection object is reflected in the cutout region CR has been described, but the present invention is not limited to this. The camera ECU 1 may be configured to perform linked storage only when the inspection object is captured in the cutout region CR. For example, the execution order of steps S33 and S34 may be interchanged. According to this configuration, it is possible to suppress the storage of partial images that are not related to facility inspection.

As described above, inspection objects are not limited to upper signs, but may be side signs, utility poles, and the like. When a side sign or a utility pole is set as the inspection object, the cropping direction can be set vertically, and the cropping region CR can be set near the left end or near the right end of the image frame. When a road marking such as a stop line is set as an object to be inspected, the cropping direction may be set to the horizontal, and the cropping area may be set to the lower half. The cut-out direction and cut-out region may be set so as to include a portion of the image frame in which the inspection target is assumed to appear.

<Modification (7)>
In the modified example (6) above, an aspect has been described in which image diagnosis or the like is performed when it is possible to detect that an upper marker is included in the latest image, but the conditions for performing image diagnosis are not limited to this. As shown in FIG. 30, the camera ECU 1 is configured to acquire and store a partial image as the latest image, and perform image diagnosis processing when the own vehicle reaches a pre-registered inspection point. It's okay to be there. The inspection point is a point where the object to be inspected should be captured in the cropped area CR of the camera 2 .

For convenience, the storage medium in which the list of inspection points is registered is called an inspection point storage unit. The camera ECU 1 of this modified example includes an inspection point storage unit, and performs a series of processing related to image diagnosis based on data stored in the inspection point storage unit. The inspection point storage unit may be implemented using part of the storage area of the storage 13 or may be a storage device independent of the storage 13 . The inspection point may be registered in the storage 13 of the camera ECU 1 as part of the program.

Note that the inspection point is determined according to the installation position of the inspection object. The inspection point can be a point a predetermined distance before the inspection object. Therefore, the camera ECU 1 may be provided with data indicating the facility installation location instead of the data directly indicating the position coordinates of the inspection point. In that case, the camera ECU 1 may calculate the inspection point based on the facility installation location and the angle of view of the camera 2, and perform the flow shown in FIG. The data indicating the facility installation location corresponds to the data indirectly indicating the position coordinates of the inspection point.

The flowchart shown in FIG. 30 includes steps S41 to S48. Step S41 is a step for determining whether or not the current position of the vehicle matches the inspection point. Matching here is not limited to perfect matching, and a given amount of error, such as 0.5 m or 1.0 m, can be allowed. In other words, the match here can allow approximate match. Steps S42 to S48 are the same as steps S31 to S33 and S35 to S38 described above.

By the way, in the modified examples (6) and (7), the configuration in which the camera ECU 1 executes the image diagnosis processing has been described, but the image diagnosis processing may be executed by an external server. The camera ECU 1 transmits connected image data obtained by connecting partial images acquired periodically/when an inspection object is detected/when an inspection point is reached, to an external server at a predetermined timing, and the server also transmits the connected image data. Image diagnosis processing may be performed on both.

In the above description, a computer such as the camera ECU 1 or a server performs image diagnosis automatically, but the image diagnosis may be performed visually by an operator. For example, the worker may determine whether there is an abnormality in the object to be inspected by comparing past connected images and current connected images obtained when traveling the same road section. The inspection processing unit F47 may be configured to display the latest image and the previous image taken at the same point side by side on the display device 6. FIG. The function of the inspection processing part F47 is an arbitrary element and may be omitted.

In addition, it is preferable that the past images to be compared with the latest image substantially match not only the position information but also the traveling direction (vehicle posture). In addition, it is preferable to compare images in which not only the position and posture of the own vehicle are the same, but also the weather and the time of day are the same. The inspection processing unit F47 can reduce the possibility of erroneously judging the state of the inspection object by using a past image with a high degree of similarity in photographing conditions as much as possible as a target for comparison with the latest image.

<Additional notes>
The apparatus, systems, and techniques described in the present disclosure may be implemented by a special purpose computer comprising a processor programmed to perform one or more functions embodied by the computer program. . The apparatus and techniques described in this disclosure may also be implemented using dedicated hardware logic. Additionally, the apparatus and techniques described in this disclosure may be implemented by one or more special purpose computers configured in combination with a processor executing a computer program and one or more hardware logic circuits. For example, some or all of the functions provided by the processor 11 may be implemented as hardware. Implementation of a function as hardware includes implementation using one or more ICs. A CPU, an MPU, a GPU, a DFP (Data Flow Processor), or the like can be used as a processor (arithmetic core). Also, some or all of the functions of the processor 11 may be implemented by combining multiple types of arithmetic processing units. Some or all of the functions of the processor 11 may be implemented using a system-on-chip (SoC), FPGA, ASIC, or the like. FPGA stands for Field-Programmable Gate Array. ASIC is an abbreviation for Application Specific Integrated Circuit.

Computer programs may also be stored as computer-executable instructions on a computer-readable, non-transitory tangible storage medium. A HDD (Hard-disk Drive), an SSD (Solid State Drive), a flash memory, or the like can be used as a program storage medium. The present disclosure also includes a computer program for causing a computer to operate as the vehicle recording device, and a recording medium storing the computer program.

1 camera ECU (vehicle recording device), 2 camera, 2A front camera, 2B rear camera, 2C left camera, 2D right camera, 3 input device, 4 vehicle sensor, 5 vehicle ECU, 11 processor, 12 RAM, 15 storage memory (storage medium), F1 image acquisition unit, F2 operation reception unit, F3 vehicle state acquisition unit, F4 data processing unit, F41 base image acquisition unit (image acquisition unit), F42 region of interest cutout unit (cutout unit), F43 connection processing unit, F44 storage processing unit, F45 recognition processing unit, F46 image processing unit, F47 inspection processing unit, CR clipping area, PI partial image, CI connected image, CPI camera integration partial image, RP vehicle horizontal plane, Mk marker

Claims (17)

A video acquisition unit (F1) that acquires a video signal from a camera installed in the vehicle;
an image acquisition unit (F41) that sequentially acquires image frames generated based on the video signal acquired by the video acquisition unit at predetermined sampling intervals;
a clipping unit (F42) for clipping, as a partial image (PI), a row range or a column range set as a clipping region from the image frame acquired by the image acquiring unit;
a connection processing unit (F43) that connects the plurality of partial images cut out by the cutout unit in chronological order in a direction orthogonal to a cutout direction, which is a direction in which the partial images are cut out from the image frame;
a storage processing unit (F44) for storing a connected image (CI), which is an image in which a plurality of partial images are connected, generated by the connection processing unit, in a predetermined storage medium (15); Vehicle recording device. The vehicle recording device according to claim 1,
The clipping unit laterally clips the image frame so as to include a predetermined line range in the image frame as the partial image,
The vehicular recording apparatus, wherein the connection processing unit is configured to vertically connect the partial images. The vehicle recording device according to claim 2,
A recording apparatus for a vehicle, wherein the vertical length of the partial image is set to be constant. The vehicle recording device according to claim 1,
The clipping unit vertically clips the image frame so as to include a predetermined row range in the image frame as the partial image,
The vehicular recording apparatus, wherein the connection processing unit is configured to horizontally connect the partial images. The vehicle recording device according to claim 4,
A recording apparatus for a vehicle, wherein the horizontal length of the partial image is set constant. The vehicle recording device according to claim 1,
a recognition processing unit (F45) that recognizes an object from an image formed by the video signal;
An image processing unit (F46) that generates an image in which a marker (Mk), which is an image indicating the recognition state of the object, is superimposed on the image formed by the video signal,
The image acquisition unit acquires an image on which the marker is superimposed as the image frame,
When the marker is placed on the clipped region of the image frame acquired by the image acquiring unit, the clipping unit selects a region including part or all of the marker as the partial image. A vehicle recording device configured to cut out. The vehicle recording device according to claim 1,
An image processing unit (F46) that generates a processed image that is an image obtained by subjecting the image indicated by the video signal to predetermined processing,
The image acquisition unit acquires the processed image as the image frame at the sampling interval,
The vehicular recording apparatus, wherein the clipping section is configured to clip a predetermined range of the processed image as the partial image. The vehicle recording device according to claim 7,
The video acquisition unit acquires the video signal from each of the plurality of cameras installed to capture images outside the vehicle,
The image processing unit generates a composite image by combining videos of the plurality of cameras,
The image acquisition unit acquires the composite image as the image frame at the sampling interval,
The clipping unit clips a predetermined range of the composite image as the image frame as the partial image,
The connection processing unit is configured to generate, as the connected image, an image in which the plurality of partial images are linked in chronological order based on the plurality of synthesized images generated at different times. The vehicle recording device according to claim 1,
The video acquisition unit acquires the video signal from each of a plurality of cameras installed in the vehicle,
The image acquisition unit acquires the image frame for each camera at the sampling interval,
The clipping unit generates the partial image for each camera,
The connection processing unit is
generating a camera-integrated partial image (CPI) by combining the partial images of the cameras having the same photographing time in a predetermined order in the extraction direction;
and generating the connected image by combining the camera-integrated partial images in a direction orthogonal to the extraction direction in chronological order. The vehicle recording device according to any one of claims 1,
The camera is mounted on the vehicle so that the imaging range includes a direction downward by 20° or more from the vehicle horizontal plane (RP), which is a plane perpendicular to the height direction of the vehicle,
The vehicular recording device, wherein the cut-out area is set to an area in which a road surface can be captured in the image frame. The vehicle recording device according to claim 1,
The camera is mounted so that the imaging range includes an upward direction of 10° or more from the vehicle horizontal plane (RP), which is a plane perpendicular to the height direction of the vehicle,
The vehicular recording apparatus, wherein the cut-out area is set to an area in which the sky can be captured in the image frame. The vehicle recording device according to any one of claims 1 to 11,
The recording device for vehicle, wherein the sampling interval is dynamically changed according to the driving scene or moving speed. The vehicle recording device according to any one of claims 1 to 11,
The clipping unit is set to clip a predetermined area in which a predetermined inspection object is expected to be reflected in the image frame as the partial image,
The storage processing unit
Acquiring data indicating shooting conditions based on input from an in-vehicle sensor;
and storing the data indicating the photographing situation in association with the partial image. The vehicle recording device according to claim 13,
An inspection processing unit (F47) that detects an abnormality of a predetermined inspection object based on the partial image,
The inspection processing unit performs the inspection by comparing the latest image, which is the newer partial image of the two partial images in which the photographing conditions match, with the past image, which is the older partial image. A recording device for vehicles that detects anomalies in objects. The vehicle recording device according to claim 14,
The inspection processing unit
judging by image recognition whether or not the inspection object is reflected in the partial image generated by the cutout unit;
Vehicle recording configured to try to compare the partial image with the past image as the latest image when the partial image including the inspection object is generated by the cutting unit. Device. The vehicle recording device according to claim 14,
an inspection point storage unit, which is a storage medium in which data directly or indirectly indicating an inspection point, which is a point where the inspection object is reflected in the cut-out area, is registered;
The inspection processing unit
determining whether the current position is the inspection point based on the data registered in the inspection point storage unit;
A vehicular recording apparatus configured to try to compare the partial image generated at the current position as the latest image with the past image when the current position is the inspection point. The vehicle recording device according to claim 13,
In addition to the position coordinates, the photographing situation includes at least one of the presence or absence of a preceding vehicle, the size of the preceding vehicle, the type of the preceding vehicle, and the inter-vehicle distance to the preceding vehicle.

Images