JP2023019827A - System, information processing method, and program and the like - Google Patents

Abstract

To provide a system and the like which are superior to conventional ones, with which it is possible to apply appropriate image processing that is based on the state or condition of a moving body.SOLUTION: A system includes a control unit. The control unit is constituted so as to execute each of the following steps. In an acquisition step, specific information pertaining to the state or condition of a moving body is acquired. In a first correction step, first correction processing is executed, on the basis of the specific information, upon image data that is generated by an image capturing device that the moving body has during the time corresponding to the specific information. In a display control step, the image data upon which the first correction processing is executed is reproducibly displayed.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device, an information processing method, a program, and the like.

A technique is disclosed in which, when the user selects the room emphasis button, brightness correction within the position is performed on image data representing a dark image displayed in the image display area (Patent Document 1).

JP 2019-129530 A

However, with the technique disclosed in Patent Document 1, there are only buttons on the screen for performing image processing on video data, and it is not possible to perform image processing desired by users for various scenes.

In view of the above circumstances, the present invention provides a system that is superior to the conventional system, such as a system that can perform appropriate image processing based on the state or situation of a mobile object.

The object of the present invention is not limited to this, and we have the intention to acquire rights by divisional application, amendment, etc. for the structure aiming to obtain the effect of the structure disclosed in the specification and drawings. For example, the present specification discloses a problem in which the descriptions such as "can do", "possible", etc. are replaced with "is a problem". The problems are described as independent ones, and we intend to acquire the rights for the structure for solving each problem independently by filing a divisional application, an amendment, etc. Even if the problem is implicitly grasped from the description of the specification, the applicant intends to claim part of the configuration described in this specification in an amended or divisional application. It also discloses a configuration that solves a problem that combines these independent problems, and has the intention to acquire the right.

(1) According to one aspect of the present invention, a system is provided. The system may comprise a controller. The controller may be configured to perform the following steps. The obtaining step may obtain specific information about the state or situation of the mobile object. In the first correction step, based on the specific information, a first correction process may be performed on the image data generated by the imaging device of the moving object at the time corresponding to the specific information. In the display control step, the image data subjected to the first correction process may be displayed in a reproducible manner.

In this way, the user can view an image reproduced from the image data generated by the imaging device at the time corresponding to the state or situation of the moving object after the first correction processing has been performed. .
In particular, the correcting process for the image data should be appropriate image processing. The image processing may be correction processing for correcting the image data generated by the photographing device of the moving body so that the image data is more visually recognizable for the user. In particular, correction processing should be performed constantly. According to this aspect, it is possible to perform appropriate image processing based on the state or situation of the moving body, so that it is possible to consistently present image data that is easily visible to the user regardless of the situation or scene. system can be realized.

For example, it may be provided in each aspect described below.

(2) In the above system, in the first correction step, the first correction process is performed on the image data while making the correction range variable based on the specific information. .

According to such an aspect, since it is not necessary to correct the entire range, the processing load can be suppressed. Moreover, since the preferable range is corrected, the user's consciousness can be directed to the preferable range regardless of the spatial range or the temporal range.

(3) In the system, the correction range is a region of interest set in the frame of the image data.

According to this aspect, since it is possible to narrow down the spatial region in which visibility is originally desired to be improved and perform correction, it is possible to direct the user's awareness to a preferable range on a frame-by-frame basis.

(4) In the system, the control unit is configured to further execute the following steps, a first receiving step receiving a first input by the user, and a canceling step receiving the first input and canceling the first correction process on the image data based on.

According to this aspect, when the correction process presented by the system is unnecessary, it is possible to improve the visibility by canceling it with the intention of the user's input. Images can be presented.

(5) In the system, the control unit is configured to further execute the following steps, a second receiving step receiving a second input from the user, and a second correcting step receiving the second 2, performs a second correction process on the image data.

According to this aspect, in addition to the correction processing presented by the device, it is possible to improve the visibility by adding the correction with the intention of the user's input. A combined image can be presented.

(6) In the system, the display control step displays the image data reproducibly in such a manner that the first correction process and the second correction process can be distinguished from each other.

According to this aspect, the user can grasp the type of correction processing, which is user-friendly.

(7) In the above system, the specific information is at least one of the position, direction, speed, acceleration, time, time period, weather, driving conditions, and pedal work of the moving body.

According to this aspect, since at least one of the specific states of the mobile body, namely position, speed, acceleration, running state, and pedal work, is considered, based on this information, More appropriate image processing can be performed.

(8) In the above system, the control unit is configured to further execute an obtaining step, wherein the specific information is the position of the moving body, the obtaining step obtains road network information, In the correction step 1, a first correction process is performed on the image data based on the position and the road network information.

According to this aspect, appropriate image processing can be performed according to the position and the road conditions at that time. For example, by performing appropriate image processing according to traffic volume obtained from road network information, it is possible to present an image that is easier for the user to visually recognize.

(9) In the above system, the control unit is configured to further execute an obtaining step, wherein the specific information is the position of the moving body, and the obtaining step obtains terrain information; In the correction step of (1), a first correction process is performed on the image data based on the position and the terrain information.

According to this aspect, appropriate image processing can be performed according to the position and the topography, environment, and the like at that time. For example, by applying different image processing to a busy street and a forest, it is possible to present an image that is easier for the user to visually recognize.

(10) In the system, the specific information is the position and orientation of the moving object, and in the first correction step, based on the position of the sun estimated from the position and the orientation, and performing a first correction process on the image data.

According to such an aspect, appropriate image processing can be performed in consideration of the position of the sun. For example, when the sun shines in the afternoon, the backlight tends to lower the brightness of the area that the user originally wants to see. can be done.

(11) In the system, the first correction step includes a preparation step and a processing step, and the control unit stores a first program stored in advance and a second program different from the first program. The first program causes the preparation step to be executed, and in the preparation step, parameters included in the image data are extracted, and the values of the extracted parameters are transferred to the Passed to a second program, the second program causes the processing step and the display control step to be executed, and the processing step executes the first correction processing for the parameter, The value of the parameter is changed, and the display control step reproducibly displays the image data based on the value of the parameter changed in the processing step.

According to this aspect, a separate program that is more specialized for correction processing can be used, which contributes to reduction in calculation cost and improvement in image processing performance. In particular, by passing the parameter values to the second program without duplicating the original image data, the second program suitable for image processing reads out the parameter values and follows them. It is possible to reproduce image data that has been subjected to image processing.

(12) In the system, the parameters include at least one of hue, brightness, saturation, contrast, gamma, sharpness, playback start time, playback end time, and presence/absence of loop.

According to this aspect, each parameter such as hue, brightness, saturation, contrast, gamma, sharpness, reproduction start time, reproduction end time, and presence or absence of loop is a general parameter of image data. Therefore, a more versatile image processing program can be used as the second program, and a system with high usability that is not limited to dedicated software can be realized.

(13) In the system, in the preparation step, operation information regarding the operation of the second program is passed to the second program, and in the display control step, the second program is controlled based on the operation information. Reproducible display of the image data by restricting at least part of the functions it has.

According to this aspect, the second program can be used only for correction and reproduction, and the user can avoid using unnecessary functions of the second program, and can avoid erroneous operations. It is possible to prevent or solve the problem that the user finds it difficult to use due to the large number of functions.

(14) In the system, the first program is configured to execute the display control step, and in the display control step, if the second program is not installed, a screen recommending installation of the second program is displayed. The one that displays the .

According to this aspect, even if the second program is not installed, it is possible to inform the user of the recommended second program. As a result, the user can, for example, install the second program on his/her own computer to reproduce the image data that has been appropriately subjected to the first correction process.

(15) In the system, the second program is a program stored in an external device, and the control unit is configured to call the second program via a network.

According to this aspect, even if the second program is not installed, the first correction process can be performed via the second program. For example, even a user whose computer capacity is limited or who does not have the authority to install a new program can appropriately perform the first correction process.

(16) A system comprising a control unit, wherein the control unit is configured to execute the following steps, in the recording step, the image captured by the camera is recorded as image data, and in the display control step, , wherein the image captured by the camera is displayed as a live view, and parameters for the recorded image data and the displayed image are set differently.

According to such an aspect, both visibility and customizability can be ensured by differentiating the video visually recognized as live view and the image data to be recorded.

(17) In the system, in the recording step, the video is recorded as the image data without correction, and in the display control step, the video is corrected and displayed as the live view.

According to this aspect, it is possible to prevent information loss due to processing of image data to be recorded while enhancing the visibility of the video viewed as a live view.

(18) In the system, in the recording step, the parameters of the displayed image are recorded as a parameter log in chronological order together with the image data.

According to such an aspect, it is possible to store the visibility of the video viewed as a live view as a record. Later, using this information, the visibility can be reproduced when reproducing the image data.

(19) In the system, the control unit is configured to further execute a third receiving step, the third receiving step receiving a third input from the user, and the display controlling step receiving the and reading the image data and the parameter log to reproducibly display the corrected image data based on a third input.

According to such an aspect, when the user desires, it is possible to reproduce an image with high visibility such as in live view on the spot, and it is possible to realize a system with high usability.

(20) In the above system, the control unit is configured to be able to call a third program stored in advance and a fourth program different from the third program, and in the display control step, the image When data is reproduced by the fourth program, information prompting reproduction by the third program is displayed.

According to this aspect, even if the content is played back by an unintended program, it is possible to guide the playback with a preferred program and provide appropriate information to the user.

(21) In the above system, the information prompting reproduction is displayed together with a reproduction screen of the image data.

According to this aspect, it is possible to appropriately provide information to the user that the current appearance is unfavorable and that the image data itself is fine.

(22) A method of information processing, comprising the steps of the system.

According to such a method, the functions of the system described above can be realized and provided in the form of a method.

(23) A program that causes a computer to function as a controller of the system.

According to this aspect, the functions of the system described above can be realized and provided in the form of a program.
Of course, it is not limited to this.

The inventions shown in (1) to (23) above can be combined arbitrarily. For example, a configuration may be adopted in which at least part of the configuration of at least one of the following inventions (2) is added to all or part of the configuration of the invention shown in (1). In particular, the invention shown in (1) may be added with at least a part of the configuration of at least one invention after (2). Also, arbitrary configurations may be extracted from the inventions shown in (1) to (23) and the extracted configurations may be combined. The applicant of this application intends to obtain rights to inventions including these configurations. In addition, even if there is a description of "in the case of" or "when", it is not described as a configuration limited to that case or that time. These are examples of better configurations, and we intend to acquire configurations that are not in these cases or occasions. Moreover, the parts described with order are not limited to this order. It also discloses a configuration in which some parts are deleted or the order is changed, and there is an intention to acquire the right.

According to the present invention, since appropriate image processing can be performed based on the state or situation of a moving object, a system that can constantly present image data that is easy for the user to visually recognize regardless of the situation or scene. can be realized.

In addition, the effect of the invention of the present application is not limited to this, and the effect exerted by the parts of the structure disclosed in the specification and drawings, etc. is also disclosed, and the structure exerting the effect is also obtained by divisional application, amendment, etc. Intend to acquire rights. For example, in this specification, the places described as "can do", "can be", etc. are descriptions that clearly show the effect, and even if there is no description such as "can be done", "can be done", etc. There are parts shown. Moreover, even without such a description, there is an effect that can be grasped by the configuration.

FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system according to Embodiment 1. As illustrated in FIG. FIG. 2 is a diagram illustrating the configuration of the vehicle according to the first embodiment. FIG. 3 is a diagram showing an example of the external configuration of the photographing device. FIG. 4 is a diagram showing the configuration of the bracket. FIG. 5 is a block diagram showing the electrical configuration of the imaging device. 6 is a diagram illustrating an example of a hardware configuration of a server according to the first embodiment; FIG. 7 is a diagram illustrating an example of a hardware configuration of an information display terminal according to the first embodiment; FIG. 8 is a diagram illustrating an example of a functional configuration of a server according to the first embodiment; FIG. 9 is a diagram illustrating an example of a functional configuration of an information display terminal according to the first embodiment; FIG. FIG. 10 is an activity diagram showing the flow of information processing by the server according to the first embodiment. FIG. 11 is an activity diagram showing the flow of information processing of the information display terminal according to the first embodiment. 12 is a diagram illustrating an example of a screen before image processing, displayed by the display control unit according to the first embodiment; FIG. 13 is a diagram illustrating an example of a screen after image processing, displayed by the display control unit according to the first embodiment; FIG. 14 is a diagram illustrating an example of a screen after image processing, displayed by a display control unit according to Modification 1 of Embodiment 1; FIG. 15 is a diagram illustrating an example of a screen before image processing, displayed by a display control unit according to Modification 2 of Embodiment 1; FIG. 16 is a diagram illustrating an example of a screen before image processing, displayed by a display control unit according to Modification 3 of Embodiment 1; FIG. 17 is a diagram illustrating an example of a screen before image processing, displayed by a display control unit according to Modification 4 of Embodiment 1; FIG. 18 is a diagram illustrating an example of a screen before image processing, displayed by a display control unit according to Modification 5 of Embodiment 1; FIG. 19 is a diagram illustrating an example of a screen before image processing, displayed by a display control unit according to Modification 6 of Embodiment 1; FIG. 20 is a diagram illustrating an example of a screen before image processing, displayed by a display control unit according to Modification 7 of Embodiment 1; FIG. 21 is a diagram illustrating an example of a screen before image processing, displayed by a display control unit according to Modification 8 of Embodiment 1; FIG. 22 is a diagram illustrating an example of a screen before image processing, displayed by a display control unit according to Modification 9 of Embodiment 1; FIG. 23 is a diagram illustrating an example of a screen after image processing, displayed by a display control unit according to Modification 10 of Embodiment 1; FIG. 24 is a diagram illustrating an example of a screen after image processing, displayed by a display control unit according to Modification 11 of Embodiment 1; FIG. 25 is a diagram illustrating an example of a screen after image processing, displayed by a display control unit according to Modification 12 of Embodiment 1; FIG. 26 is a diagram illustrating an example of a screen after image processing, displayed by a display control unit according to Modification 13 of Embodiment 1; FIG. 27 is a diagram illustrating an example of a screen after image processing, displayed by a display control unit according to Modification 14 of Embodiment 1; FIG. 28 is a diagram illustrating an example of a screen after image processing, displayed by a display control unit according to Modification 15 of Embodiment 1; FIG. 29 is a diagram illustrating an example of a screen displayed by a display control unit according to Modification 16 of Embodiment 1; FIG. 30 is a diagram illustrating an example of a screen after image processing, displayed by a display control unit according to Modification 17 of Embodiment 1; FIG. 31 is a diagram illustrating an example of a screen displayed by a display control unit according to Modification 18 of Embodiment 1; FIG. 32 is a diagram illustrating an example of a screen before image processing, displayed by a display control unit according to Modification 19 of Embodiment 1; FIG. 33 is a diagram illustrating an example of a screen displayed by a display control unit according to Modification 20 of Embodiment 1; FIG. 34 is a diagram illustrating an example of a screen displayed by a display control unit according to Modification 21 of Embodiment 1; FIG. 35 is a diagram illustrating an example of a screen displayed by a display control unit according to Modification 22 of Embodiment 1; FIG. 36 is a diagram illustrating an example of a screen displayed by a display control unit according to Modification 23 of Embodiment 1; FIG. 37 is a schematic diagram illustrating a system configuration of an information processing system according to a modification of the second embodiment; FIG. 38 is a schematic diagram illustrating a functional configuration of an information processing terminal according to a modification of Embodiment 2; FIG. FIG. 39 is a functional block diagram realized by the storage section 22 and the control section 23. As shown in FIG. FIG. 40 is an activity diagram showing the flow of information processing according to the fourth embodiment. FIG. 41 is a schematic diagram showing an example of a spatial correction range of the first correction process. FIG. 42 is a schematic diagram showing an example of the temporal correction range of the first correction process. FIG. 43 is a schematic diagram showing additional information 43 attached to the reproduced screen 4 when the first correction process and the second correction process are executed. 44 is an activity diagram illustrating the flow of information processing according to Modification 3 of Embodiment 4. FIG. FIG. 45 is a functional block diagram realized by the control unit 33. As shown in FIG. FIG. 46 is an activity diagram showing the flow of information processing according to the fifth embodiment. FIG. 47 is a schematic diagram showing aspects of image data IM displayed as a live view and image data IM recorded so as to be reproducible later. FIG. 48 is a schematic diagram showing a mode in which additional information 43 prompting reproduction of the recommended third program P3 is superimposed on the reproduction screen 4. As shown in FIG.

EMBODIMENT OF THE INVENTION Hereinafter, the form for implementing this invention is demonstrated with reference to drawings. The embodiment shown below is one embodiment that provides the present invention, and the content of the present invention should not be construed as being limited based on the following description.

By the way, the program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable medium (Non-Transitory Computer-Readable Medium), or may be downloaded from an external server. It may be provided as possible, or may be provided so that the program is activated on an external computer and the function is realized on the client terminal (so-called cloud computing).

Further, in the present embodiment, the term “unit” may include, for example, a combination of hardware resources implemented by circuits in a broad sense and software information processing that can be specifically realized by these hardware resources. . In addition, various information is handled in the present embodiment, and these information are, for example, physical values of signal values representing voltage and current, and signal values as binary bit aggregates composed of 0 or 1. It is represented by high and low, or quantum superposition (so-called quantum bit), and communication and operation can be performed on a circuit in a broad sense.

A circuit in a broad sense is a circuit implemented by appropriately combining at least circuits, circuits, processors, memories, and the like. Application Specific Integrated Circuits (ASICs), programmable logic devices (e.g., Simple Programmable Logic Devices (SPLDs), Complex Programmable Logic Devices (CPLDs), and field It includes a programmable gate array (Field Programmable Gate Array: FPGA).

[Embodiment 1]
[System configuration of information processing system]
An example of the information processing system 3000 will be described below. FIG. 1 is a diagram showing an example of the system configuration of an information processing system 3000 according to the first embodiment. As shown in FIG. 1, the information processing system 3000 comprises a vehicle 3400, a server 3700, a plurality of information display terminals 3800A, 3800B, 3800C and a network 3900. FIG. Vehicle 3400 is configured to be able to communicate with server 3700 via network 3900 . More specifically, vehicle 3400 transmits information possessed by vehicle 3400 to server 3700 via network 3900 . A plurality of information display terminals 3800A, 3800B, and 3800C are configured to be able to communicate with server 3700 via network 3900. FIG. More specifically, a plurality of information display terminals 3800A, 3800B, and 3800C acquire information in server 3700 via network 3900. FIG. Here, the information display terminals 3800A, 3800B, and 3800C may be devices that have a role as a UI (User Interface), and specific examples include personal computers, smart phones, tablets, and the like.

[Overall Configuration of Vehicle 3400]
FIG. 2 is a diagram illustrating the configuration of vehicle 3400 according to the first embodiment. FIG. 2 shows a schematic diagram of the vehicle 3400 viewed from the side. A vehicle 3400 has an imaging device 3100 as a first imaging device and an imaging device 3200 as a second imaging device. The vehicle 3400 is, for example, an internal combustion engine vehicle having an engine as a drive source, a hybrid vehicle having an engine and a travel motor as drive sources, an electric vehicle having a travel motor as a drive source, or the like. The vehicle 3400 is, for example, a four-wheeled automobile, but may be any vehicle on which the imaging device 3100 and the imaging device 3200 can be arranged. The vehicle may be, for example, a large transportation vehicle with four or more wheels such as an automobile, a bus, or a truck, a two-wheeled vehicle such as a motorcycle or a bicycle, or other vehicles. The vehicle may be, for example, a transportation vehicle such as a train, a monorail, or a linear motor car.

The imaging device 3100 is a front camera arranged on the front side of the vehicle 3400 . The photographing device 3100 is attached, for example, at a predetermined position on the front side of the cabin of the vehicle 3400, and photographs the front of the vehicle 3400 through the windshield as the photographing direction. The imaging device 3100 is a drive recorder. More specifically, the photographing device 3100 has a function of photographing, a function of recording image data representing the photographed image, and a function of recording image data obtained from the photographing device 200 .

The imaging device 3200 is a rear camera arranged on the rear side of the vehicle 3400 . The photographing device 3200 is attached, for example, at a predetermined position on the rear side of the cabin of the vehicle 3400, and photographs the rear side of the vehicle through the rear glass. The photographing device 3200 has a function of photographing and a function of outputting image data representing the photographed image to the photographing device 3100 .

The imaging device 3100 and the imaging device 3200 are connected via a cable 3300 . A cable 3300 is a wired communication path that connects the imaging device 3100 and the imaging device 3200 . The cable 3300 has, for example, a power line that supplies power for operation from the imaging device 3100 to the imaging device 3200, and a signal line for transmitting various signals between the imaging devices 3100 and 3200. . The imaging device 3200 operates by being supplied with power from the imaging device 3100 via the cable 3300 . Note that the imaging devices 3100 and 3200 may be connected by a wireless communication channel of Wi-Fi (registered trademark), Bluetooth (registered trademark), or other standards, instead of a wired communication channel. Also, the imaging device 3100 may be used without being connected to the imaging device 200 through communication.

[External configuration of imaging device 3100]
FIG. 3 is a diagram showing an example of the external configuration of the imaging device 3100. As shown in FIG. FIG. 3A is a diagram of the photographing device 3100 as seen from the front and obliquely upper right direction. FIG. 3B is a diagram of the photographing device 3100 viewed obliquely from the upper right on the rear side. The imaging device 3100 has a housing 3101 . The housing 3101 has a rectangular parallelepiped shape that is longer in the horizontal direction than in the vertical direction and has a relatively small thickness. The housing 3101 has a top surface 31011 facing upward when attached to the vehicle 3400, a first side surface 31012, a second side surface 31013, and a third side surface 31014 located opposite the second side surface 31013. and a fourth side 31015 located opposite the first side 31012 .

A joint rail 3102 and a camera jack 3191 are provided on the upper surface 31011 . The joint rail 3102 can attach and detach a bracket (for example, a bracket 3600 described later) for attaching the imaging device 3100 to a predetermined attachment position of the vehicle 3400 . The mounting position may be, for example, the windshield of the vehicle 3400 (for example, near the upper end of the windshield), the rearview mirror of the vehicle 3400, the ceiling of the vehicle interior, or the like. When the photographing device 3100 is attached to the vehicle 3400 , the first side surface 31012 faces the front side of the vehicle 3400 . At this time, the second side surface 31013 faces rightward when viewed from the rear side of the vehicle 3400 . The third side surface 31014 faces left when viewed from the rear side of the vehicle 3400 . The fourth side 31015 faces the rear side of the vehicle 3400 .

Camera jack 3191 is a terminal to which one end of cable 3300 is connected. The camera jack 3191 may correspond to, for example, the USB Type C standard, and may be a terminal for the imaging device 3100 to communicate with the imaging device 3200 according to the Ethernet standard.

A photographing lens 3151 , a sound emitting hole 3103 and a microphone hole 3104 are provided on the first side surface 31012 . A photographing lens 3151 is a lens for condensing light included in the photographing unit (the photographing unit 3015 to be described later) provided in the photographing apparatus 3100 . The sound emitting hole 3103 is provided above the photographing lens 3151, and is a hole through which the sound output by the sound output unit (the sound output unit 3014 described later) of the photographing device 3100 is transmitted from the inside of the housing 3101 to the outside. . A microphone hole 3104 is provided below the photographing lens 3151 and is a hole through which external sound is transmitted from the outside of the housing 3101 to the inside. The sound that has passed through the housing 3101 is input to a microphone (a microphone 3121 to be described later) of the imaging device 3100 .

An event recording button 3122 is provided on the second side surface 31013 . The event recording button 3122 is an operation means for instructing the start of recording (recording) of the image captured by the imaging unit 3015 or the end of the recording. When the event recording button 3122 is operated by the user while event recording is not being performed, the imaging device 3100 starts event recording. Event recording will be described later in detail. The event record button 3122 is provided on the second side 31013 facing the driver's seat for easy operation by the driver of the right-hand drive vehicle 3400 .

A terminal 3192 and a storage medium insertion port 3181 are provided on the third side surface 31014 . A terminal 3192 is a terminal for receiving power supply from an external device. Terminal 3192 is, for example, a DC jack. Terminal 3192 is connected to a connector on one end of a power supply cord (for example, cigar plug cord). A connector on the other end side of the power cord is connected to a power supply terminal (for example, a cigar socket) provided on the vehicle 3400 side, for example.

Terminal 3192 may be connected to an OBDII adapter connectable to an OBDII (“II” is the Roman numeral for “2”) connector of vehicle 3400 . The OBDII connector, also called a failure diagnosis connector, is a terminal that is connected to an ECU (Engine Control Unit) of a vehicle and outputs various vehicle information every predetermined period (for example, every 0.5 seconds). By connecting the terminal 3192 to the OBDII connector using the OBDII adapter, the imaging device 3100 can receive power for operation and acquire vehicle information.

Vehicle information is information about the state of vehicle 3400 . The vehicle information includes, for example, the speed of the vehicle 3400 (vehicle speed), engine speed, engine load factor, throttle degree, ignition timing, remaining fuel ratio, intake manifold pressure, intake air amount (MAF), injection opening time, engine Cooling water temperature (cooling water temperature), temperature of air taken into the engine (intake air temperature), air temperature outside the vehicle (outside air temperature), amount of remaining fuel in the fuel tank (remaining fuel amount), fuel flow rate, instantaneous fuel consumption, At least one or more of accelerator opening, blinker information (operation of left and right blinkers (on/off)), brake opening, steering angle of steering wheel, gear position, door opening/closing state, etc. may be used.

A storage medium insertion port 3181 is an insertion port for inserting a storage medium 3500 as an external storage means into the imaging device 3100 . A storage medium 3500 is a storage medium in which images captured by the imaging device 3100 or the imaging device 3200 are recorded, and is an SD card, for example. The SD card includes any form such as SD memory card, miniSD card, and microSD card. The storage medium 3500 may further store a program for a viewer (for example, a dedicated viewer) for reproducing the stored images on an information display terminal such as a personal computer.

An operation unit 3123 , a display surface 3131 , and a light emitting unit 3021 are provided on the fourth side surface 31015 . The operation unit 3123 has a first button 31231 , a second button 31232 , a third button 31233 and a fourth button 31234 . A first button 31231 , a second button 31232 , a third button 31233 , and a fourth button 31234 are arranged vertically along one right side of the display surface 3131 . Functions assigned to these buttons include, for example, the following functions.

The first button 31231 functions as a button for switching images when pressed for a long time, and functions as a button for instructing formatting of the storage medium 3500 when pressed for a short time. The image displayed on the display surface 3131 is, for example, one or both of the image currently captured by the imaging device 3100 and the image currently captured by the imaging device 3200 . Formatting the storage medium 3500 is to initialize the storage medium 3500, for example, to erase data such as images stored in the storage medium 3500, and to put the storage medium 3500 into a state in which the imaging apparatus 3100 can use the storage medium 3500. (for example, to enable recording and reading of images), writing setting information indicating the contents of operation settings to the storage medium 3500, and setting the storage medium 3500 to a specific file state; is grasped as at least one of

A second button 31232 is a button for displaying a selection screen for selecting an image to be reproduced by the imaging device 3100 . A third button 31233 is a button for displaying a menu regarding settings of the imaging device 3100 and the imaging device 3200 . A fourth button 31234 is a button for instructing start and stop of image recording. For example, when the fourth button 31234 is pressed for a short time during recording by the constant recording function, which will be described later, the recording is temporarily stopped. When the fourth button 31234 is pressed for a short time during the pause, image recording by the constant recording function resumes. When the fourth button 31234 is long pressed, the frame rate for recording images can be changed.

A display surface 3131 is an area where an image displayed by a display unit (a display unit 3013 to be described later) of the imaging device 3100 is displayed. The display surface 3131 is, for example, a rectangular or square area. A touch sensor 3124 for detecting a user's touch operation is provided over the display surface 3131 .

The light emitting part 3021 is provided above the first button 31231 and emits light in a predetermined color.

Note that the imaging device 3200 may also have a function as a drive recorder, and may have the same configuration as the imaging device 3100, for example. Also, as the imaging device connected to the imaging device 3100 by communication, one or a plurality of imaging devices that capture images in other directions may be used instead of or in addition to the imaging device 3200 . As other directions, there are directions such as diagonally rearward right, diagonally rearward left, and vehicle width direction (side) of vehicle 3400 .

[Configuration of Bracket 3600]
FIG. 4 is a diagram showing the configuration of bracket 3600. As shown in FIG. FIG. 4A is a diagram of the bracket 3600 viewed obliquely from the upper right side on the front side. FIG. 4B is a side view of the bracket 3600. FIG. FIG. 4(C) is a diagram of the bracket 3600 viewed obliquely from the lower right on the front side. A bracket 3600 is an example of a mounting member for mounting the photographing device 3100 to the vehicle 3400 .

Bracket 3600 is an attachment member using a ball joint mechanism. In bracket 3600 , flat base portion 3610 has mounting surface 3611 to be attached to the windshield of vehicle 3400 . The base portion 3610 is inclined at a predetermined angle with respect to the support of the ball stud 3620 . An adhesive member such as a double-sided tape is attached to the mounting surface 3611, and it is attached to the windshield of the vehicle 3400 or the like via the adhesive member.

The ball stud 3620 is a portion of the base portion 3610 that rises from the surface opposite to the mounting surface 3611 . A ball portion 3621 of a ball stud 3620 is attached to the socket portion 3630 . A nut 3640 is removably attached around the socket portion 3630 . A screw groove is formed on the outer periphery of the socket portion 3630 . A nut 3640 is attached to the outer circumference of the socket portion 3630 so as to fit into the thread groove. With the ball portion 3621 of the ball stud 3620 mounted in the socket portion 3630, before the nut 3640 is tightened, the socket portion 3630 rotates in a desired direction along the peripheral surface of the ball portion 3621, and the base portion rotates. The attitude and position of the 3650 can be displaced. When the nut 3640 is tightened, the attitude and position of the base portion 3650 are fixed.

The base portion 3650 is formed in a pair with the socket portion 3630 and is a part for mounting on the photographing device 3100 . The base portion 3650 has a pair of guide rails 3651 on its bottom surface. The pair of guide rails 3651 are configured to be slidable along the joint rails 3102 of the photographing device 3100 . A claw-shaped tip portion 3652 is provided at the tip portion of the base portion 3650 . The bracket 3600 is attached to the imaging device 3100 by hooking the front end portion 3652 of the joint rail 3102 near the front end of the imaging device 3100 .

[Electrical configuration of imaging device 3100]
FIG. 5 is a block diagram showing the electrical configuration of the imaging device 3100. As shown in FIG. A control unit 3011 controls each unit of the imaging device 3100 . The control unit 3011 is, for example, a computer including a processor 3111 and memory 3112 . The processor 3111 includes, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application-Specific Integrated Circuit), and an FPGA (Field Programmable Graphic etc.). The memory 3112 is a main storage device including, for example, RAM (Random Access Memory) and ROM (Read Only Memory). The processor 3111 temporarily stores the program read from the ROM of the memory 3112 in the RAM. The RAM of memory 3112 provides a working area for processor 3111 . The processor 3111 performs arithmetic processing while temporarily storing data generated during execution of the program in the RAM, thereby performing various controls. The timer 3113 is, for example, a real-time clock. The clock unit 3113 may be mounted on the motherboard of the processor 3111 or may be externally attached to the processor 3111 .

The input unit 3012 receives input of information from the user. The input unit 3012 has, for example, the microphone 3121, event recording button 3122, operation unit 3123, and touch sensor 3124 described above. The microphone 3121 converts sound incident through the microphone hole 3104 or the like into an electric signal. Microphone 3121 is, for example, a condenser microphone. The touch sensor 3124 detects the position touched by the user on the display surface 3131 . The touch sensor 3124 is, for example, a capacitive type.

A display unit 3013 displays an image on a display surface 3131 . The display unit 3013 is, for example, a liquid crystal display (LCD).

The audio output unit 3014 outputs audio. This sound includes, for example, notification sound, BGM, voice message, and the like. The audio output unit 3014 has, for example, an audio processing circuit and a speaker.

The imaging unit 3015 takes an image and generates image data obtained by the imaging. The imaging device is, for example, a CMOS (Complementary MOS) or a CCD (Charge Coupled Device). The imaging unit 3015 generates a color (multicolor) image including, for example, red (R), green (G), and blue (B) color components.

A communication unit 3016 communicates with an external device. The communication unit 3016 has a communication circuit for wirelessly communicating with an external device by, for example, Wi-Fi (registered trademark), Bluetooth (registered trademark), other wireless LAN (Local Area Network) communication, or short-range wireless communication. . The communication unit 3016 may have, for example, a communication circuit for performing communication conforming to mobile communication system standards such as LTE (Long Term Evolution), 4G, and 5G.

The sensor unit 3017 has various sensors. The sensor unit 3017 has, for example, at least one of an acceleration sensor, a gyro sensor, an air pressure sensor, and an illuminance sensor. The acceleration sensor is, for example, a three-axis acceleration sensor that measures longitudinal, lateral, and vertical accelerations of the vehicle. A gyro sensor is a sensor that measures the tilt of the imaging device 3100 . Accelerometers and gyro sensors may be used to estimate the position of vehicle 3400 by autonomous navigation, for example, when signals from GNSS satellites cannot be received. An air pressure sensor measures air pressure. An atmospheric pressure sensor is used, for example, to detect a difference in height and determine whether it is a highway or a general road. The illuminance sensor is a sensor that detects the illuminance indicating the brightness of the interior of the vehicle around the imaging device 3100 . The illuminance sensor is used, for example, to adjust the brightness of display on the display unit 3013 .

The reader/writer 3018 functions as a medium holding unit that holds the storage medium 3500 inserted into the photographing apparatus 3100 through the storage medium insertion port 3181 . The reader/writer 3018 writes data to the storage medium 3500 and reads data from the storage medium 3500 . The reader/writer 3018 may hold only one storage medium 3500, but may be configured to hold two or more storage media 3500 at the same time.

The terminal portion 3019 has a terminal for electrical connection with an external device. The terminal section 3019 has the camera jack 3191 and the terminal 3192 described above. As a device connected to the terminal portion 3019, an external battery may be used so that the imaging device 3100 and the imaging device 3200 can operate without power supply from the vehicle 3400 side. A device connected to the terminal portion 3019 may be, for example, a device having a function of assisting the user in driving safely. As such a device, for example, a device having a function of photographing the driver (for example, the face) and detecting and notifying the state of the driver exemplified by inattentiveness and drowsy driving, or an obstacle around the vehicle 3400 (For example, devices used for vehicle detection for Forward Vehicle Collision Warning Systems (FCWS)). The device connected to the terminal portion 3019 may also be an in-vehicle device such as a radar detector, a laser detector, a car navigation device, or a display device.

The position information acquisition unit 3020 acquires position information indicating the position of the imaging device 3100 (more specifically, the current position). The position of the photographing device 3100 should be equated with the position of the vehicle 3400 in which the photographing device 3100 is arranged, the position of the photographing device 3200 arranged in the vehicle 3400, and the position of the driver and other persons riding in the vehicle 3400. can be done. The position information acquisition unit 3020 acquires the position information (latitude information and longitude information) of the imaging device 3100 based on a signal from a GPS (Global Positioning System), which is one of GNSS (Global Navigation Satellite System), for example. ). The position information acquisition unit 3020 may also use Michibiki as a QZSS (Quasi-Zenith Satellite System).

Light emitting unit 3021 emits light in a predetermined color. The light emitting section 3021 has, for example, a light emitting diode.

A power control unit 3022 controls power supply to each unit of the imaging device 3100 and to the imaging device 3200 . The power control unit 3022 has, for example, a power switch and a power control circuit. The power control unit 3022 supplies electric power supplied from the vehicle 3400 side via the terminal unit 3019 to each unit of the imaging device 3100 and the imaging device 3200 . The power control unit 3022 may further include a secondary battery, a button battery, or an electric double layer capacitor (also called a supercapacitor) as a power storage unit.

The imaging device 3100 may further have an auxiliary storage device exemplified by a flash memory (eg, eMMC, SSD) as internal storage means. Various storage media such as optical storage media, magnetic storage media, and semiconductor storage media can be used as auxiliary storage devices.

[Image recording function of imaging device 3100]
The imaging device 3100 has one or more of the following functions as image recording functions. The image recording function is a function of recording an image captured by the imaging device 3100 as image data in a predetermined file format. The image data is preferably image data in a moving picture format, for example, MPEG (Moving Picture Experts Group) format (eg, MPEG2, MPEG4), AVI, MOV, WMV, and the like. It is assumed that the image data includes Exif (Exchangeable image file format) information such as shooting date and time information and shooting location information.

[Always recording function]
The constant recording function (also referred to as constant recording function) is a function of continuously (that is, constantly) recording images captured by one or both of the imaging device 3100 and the imaging device 3200 while the imaging device 3100 is in operation. be. When executing the constant recording function, the control unit 11 stores image data captured from the start of the drive source of the vehicle 3400 to the stop. The control unit 3011 detects the activation of the driving source by turning on the power source (for example, the accessory power source) of the vehicle 3400, for example. The control unit 3011 detects stoppage of the drive source when the vehicle 3400 is powered off, for example.

[Event recording function]
The event recording function is a function of recording an image captured by one or both of the imaging device 3100 and the imaging device 3200 in response to the occurrence of a specific event. An event is an event in which an image captured by the imaging device 3100 or the imaging device 3200 should be recorded. collision, etc. The control unit 3011 determines that an event has occurred based on the acceleration measured by the acceleration sensor of the sensor unit 3017, for example. More specifically, the control unit 3011 determines that an event has occurred when the measured acceleration exceeds a predetermined threshold or exhibits a predetermined temporal change. The condition for determining the occurrence of an event is not limited to this. Based on the vehicle information, the control unit 3011 may determine that an event has occurred when the state of the vehicle 3400, such as vehicle speed and steering state, satisfies a predetermined condition. The control unit 3011 analyzes the image captured by the image capturing unit 3015 or the image capturing device 3200, and an event occurs when the vehicle 3400 or another vehicle is detected to be driving dangerously (for example, driving in a rush, approaching, or approaching abnormally). can be determined. The control unit 3011 also determines that an event has occurred when the event record button 3122 is operated.

When the control unit 3011 determines that an event has occurred, the control unit 3011 records in the storage medium 3500 images captured during a predetermined period before and after the occurrence of the event (hereinafter referred to as “event recording period”). For example, the control unit 3011 temporarily records images captured by the imaging unit 3015 and the imaging device 3200 in the memory 3112 (eg, RAM), and reads the images from the memory 3112 when determining that an event has occurred. Images during the event recording period may be recorded in the storage medium 3500 . The control unit 3011, for example, makes one file 40 seconds of images in total, 20 seconds before the occurrence of the event and 20 seconds after the occurrence of the event. The event recording period is an example, and may differ depending on the type of event, or may be changeable by the user. The control unit 3011 may record an image made up of a plurality of files in the storage medium 3500 for each occurrence of one event. The control unit 3011 further associates the values (eg, acceleration in each direction of three axes) measured by the sensor unit 3017 during the event recording period and the position information acquired by the position information acquisition unit 3020 with the image, It may be recorded on the storage medium 3500 .

[Parking monitoring function]
The parking monitoring function is a function of recording images captured by one or both of the imaging device 3100 and the imaging device 3200 while the vehicle 3400 is parked. The parking monitoring function is a function for monitoring the inside of the parked vehicle 3400 or the outside around the vehicle 3400 . When vehicle 3400 is powered off, control unit 3011 receives power from an external battery and records an image in storage medium 3500 . Regarding whether or not the vehicle 3400 is parked, the control unit 3011 detects, for example, that the power source of the vehicle 3400 is turned off, that the power source of the vehicle 3400 is turned off, and that the power supply from the external battery is started. that the vehicle speed is 0 km / h or a predetermined speed or less, and that the position information acquired by the position information acquisition unit 3020 is predetermined position information (for example, position information of home, work, or parking lot); is determined based on one or more of

The parking monitoring function may have a time-lapse mode and a motion detection mode. More specifically, when the time-lapse mode is selected by the user, the control unit 3011 records images by thinning the frame rate more than other image recording functions such as the constant recording function and the event recording function. For example, while the frame rate of other image recording functions is 20 to 30 frames/second, the frame rate of the time-lapse mode is 1 frame/second. The moving body detection mode is a mode for recording an image in response to detection of a moving body. More specifically, when the moving body detection mode is selected by the user and the moving body is detected from changes in the images captured by the imaging devices 3100 and 3200, the control unit 3011 detects the motion before and after the detection. Images captured during a predetermined period are recorded in the storage medium 3500 . The frame rate may be the same frame rate as the constant recording function, the event recording function, and the like.

As the imaging devices 3100 and 3200, imaging devices that capture celestial images such as omnidirectional and semi-spherical images may be used. Further, as the imaging device 3100, an imaging device that does not have the display unit 3013 may be used. Further, the housing of the photographing device 3100 does not have to be rectangular parallelepiped, and may be cylindrical, for example.

[Hardware Configuration of Server 3700]
The hardware configuration of the server 3700 will be described below with reference to FIG.

FIG. 6 is a diagram showing an example of the hardware configuration of the server 3700 according to the first embodiment. In Embodiment 1, the server 3700 serves as an image processing device. Server 3700 includes a control unit 3710, a storage unit 3720, and a communication unit 3730 as a hardware configuration. Control unit 3710, storage unit 3720, and communication unit 3730 are electrically connected to each other via a communication bus.

The control unit 3710 is composed of a microcomputer including a CPU and semiconductor memory, and controls the overall operation of the server 3700 . In particular, control unit 3710 realizes various functions related to server 3700 by reading out and executing a predetermined program stored in storage unit 3720 . That is, the control unit 3710 executes processing based on a predetermined program stored in the storage unit 3720, so that the functional configuration shown in FIG. 8 and the processing of the activity diagram shown in FIG. Realized. Note that the control unit 3710 is not limited to a single unit. The server 3700 may be configured to include multiple controllers 3710 .

The storage unit 3720 is configured by any one of HDD (Hard Disk Drive), RAM (Random Access Memory), ROM (Read Only Memory), SSD (Solid State Drive), or any combination thereof. The storage unit 3720 stores image information sent from the vehicle 3400, information other than the image sent from the vehicle 3400, various data related to various programs including part of the image processing program according to the first embodiment, and the control unit 3710. stores data and the like used when executing processing based on the program.

Communication unit 3730 is a so-called network interface that is communicably connected to vehicle 3400 and information display terminals 3800A, 3800B, and 3800C via network 3900 such as the Internet and LAN. In Embodiment 1, the communication unit 3730 of the server 3700 that performs image processing is an image processing device that can communicate with the communication unit 3830 of the information display terminal 3800 . The main information that the server 3700 receives via the communication unit 3730 includes image information captured by the imaging device 3100 of the vehicle 3400, image information captured by the imaging device 3200, information on the state of the vehicle 3400 (speed of the vehicle 3400 ( vehicle speed), acceleration, engine speed, engine load factor, throttle opening, ignition timing, percentage of remaining fuel, intake manifold pressure, intake air volume (MAF), injection opening time, engine cooling water temperature (cooling water temperature) ), temperature of the air taken into the engine (intake air temperature), air temperature outside the vehicle (outside air temperature), amount of remaining fuel in the fuel tank (remaining fuel amount), fuel flow rate, instantaneous fuel consumption, accelerator opening, blinker information (left and right turn signal operation (on / off)), brake opening, steering wheel rotation angle, gear position, door open/close status information, position information, etc.), date and time information, air pressure, illuminance, image processing instructions, image Examples include a download request instruction, a request instruction for information other than images, and the like. The main information that the server 3700 transmits via the communication unit 3730 includes image information before executing image processing, image information after executing image processing, information on the image processing method, information on the state of the vehicle 3400 ( Vehicle 3400 speed (vehicle speed), acceleration, engine speed, engine load factor, throttle opening, ignition timing, percentage of remaining fuel, intake manifold pressure, intake air amount (MAF), injection opening time, engine cooling water Temperature (cooling water temperature), temperature of air taken into the engine (intake air temperature), air temperature outside the vehicle (outside air temperature), amount of remaining fuel in the fuel tank (remaining fuel amount), fuel flow rate, instantaneous fuel consumption, accelerator opening , turn signal information (left and right turn signal operation (on/off)), brake opening, steering wheel rotation angle, gear position, information on door open/close status, position information, etc.), date and time information, atmospheric pressure, illuminance, etc. be done.

[Hardware Configuration of Information Display Terminal 3800]
Next, the hardware configuration of the information display terminal 3800 will be described with reference to FIG.

FIG. 7 is a diagram showing an example of the hardware configuration of the information display terminal 3800 according to the first embodiment. The information display terminal 3800 includes a control section 3810, a storage section 3820, a communication section 3830, an input section 3840 and an output section 3850 as a hardware configuration. Control unit 3810, storage unit 3820, communication unit 3830, input unit 3840 and output unit 3850 are electrically connected to each other via a communication bus.

The control unit 3810 is composed of a microcomputer including a CPU and a semiconductor memory, and controls the entire operation of the information display terminal 3800 . In particular, control unit 3810 realizes various functions related to information display terminal 3800 by reading out and executing a predetermined program stored in storage unit 3820 . That is, the control unit 3810 executes processing based on a predetermined program stored in the storage unit 3820, thereby realizing the functional configuration shown in FIG. 9 and the processing of the activity diagram shown in FIG. be done. Note that the control unit 3810 is not limited to a single unit. Information display terminal 3800 may be configured to include a plurality of control units 3810 .

The storage unit 3820 is configured by any one of HDD, SSD, RAM, ROM, etc., or any combination thereof. Storage unit 3820 stores a program for exchanging data related to image processing with server 3700, data used when control unit 3810 executes processing based on the program, and the like.

The communication unit 3830 is a so-called network interface that is communicably connected to the network 3900 in order to transmit and receive various data between the information display terminal 3800 and the server 3700 . That is, the main information received by the information display terminal 3800 via the communication unit 3830 includes image information before executing image processing, image information after executing image processing, information on the image processing method, and information on the vehicle 3400. State information (vehicle 3400 speed (vehicle speed), acceleration, engine speed, engine load factor, throttle opening, ignition timing, remaining fuel ratio, intake manifold pressure, intake air flow rate (MAF), injection opening time, Temperature of engine cooling water (cooling water temperature), temperature of air taken into the engine (intake air temperature), air temperature outside the vehicle (outside air temperature), amount of remaining fuel in the fuel tank (remaining fuel amount), fuel flow rate, instantaneous fuel consumption , accelerator opening, turn signal information (left and right turn signal operation (on/off)), brake opening, steering wheel rotation angle, gear position, door open/close status information, etc.), date and time information, location information, atmospheric pressure, Illuminance and the like can be mentioned. Main information transmitted by the information display terminal 3800 via the communication unit 3830 includes an image processing instruction, an image download request instruction, and an information request instruction other than the image.

Input unit 3840 receives an operation to information display terminal 3800 . Examples of the input unit 3840 include a mouse, keyboard, touch panel, microphone, buttons, and the like.

The output unit 3850 outputs the result of the operation to the input unit 3840. FIG. Examples of output unit 3850 include a display, speakers, lights, buttons, and the like.

[Functional Configuration of Server 3700]
The functional configuration of the control unit 3710 of the server 3700 will be described below with reference to FIG.

FIG. 8 is a diagram showing an example of the functional configuration of the server 3700 according to the first embodiment. As shown in FIG. 8, the server 3700 includes an image processing unit 3711, a communication control unit 3712, and a storage control unit 3713 as functional configurations.

The image processing unit 3711 performs image processing on the image stored in the storage unit 3720 or transmitted from the information display terminal 3800 based on a request from the information display terminal 3800 . As a result, the server 3700 that can communicate with the information display terminal 3800 can execute image processing. In the first embodiment, the image processing unit 3711 performs various image processing based on requests from the information display terminal 3800 . However, the server 3700 may have an image processing unit for each image processing. Further, instead of software of the server 3700 performing image processing, dedicated hardware (for example, GPU, etc.) of the server 3700 may perform image processing. When dedicated hardware performs image processing, one hardware of the server 3700 may be configured to perform a plurality of image processes, or a plurality of hardware of the server 3700 corresponding to each of the plurality of image processes may be configured. Each of them may be configured to perform image processing.

Communication control unit 3712 controls information exchange with vehicle 3400 and information display terminal 3800 via network 3900 .

The storage control unit 3713 stores information in the storage unit 3720 and acquires the stored information.

[Functional Configuration of Information Display Terminal 3800]
The functional configuration of the control unit 3810 of the information display terminal 3800 will be described below with reference to FIG.

FIG. 9 is a diagram showing an example of the functional configuration of the information display terminal 3800 according to the first embodiment. As shown in FIG. 9, the information display terminal 3800 includes a display control section 3811, a communication control section 3812 and a storage control section 3813 as functional configurations.

The display control section 3811 controls information to be output to the output section 3850 of the information display terminal 3800 . More specifically, the display control unit 3811 controls to display a screen including information about an image.

A communication control unit 3812 controls exchange of information with the server 3700 via the network 3900 . More specifically, the communication control unit 3812 acquires images captured by the imaging device 3100 and images captured by the imaging device 3200 from the server 3700 .

The storage control unit 3813 stores information in the storage unit 3820 and acquires the stored information.

[Outline of information processing]
The information processing flow of the information processing system 3000 according to the first embodiment will be described with reference to FIGS. 10 and 11. FIG.

[Information processing of server 3700]
First, the flow of information processing of the server 3700 will be described. FIG. 10 is an activity diagram showing the flow of information processing of the server 3700 according to the first embodiment. The flow of information processing will be outlined below along this activity diagram.

At A4001, the communication control unit 3712 waits for reception of information from the vehicle 3400. FIG. When communication control section 3712 determines that information has been received from vehicle 3400, it advances the process to A4002. The communication control unit 3712 receives, as information from the vehicle 3400, identification information identifying the vehicle 3400, image information captured by the imaging device 3100, image information captured by the imaging device 3200, vehicle information of the vehicle 3400, and other information. It is described as receiving information (eg, information about the time the image was taken).

At A4002, the storage control unit 3713 causes the storage unit 3720 to store the received image information, vehicle information, and other information. More specifically, memory control unit 3713 stores image information, vehicle information, and other information in a predetermined area of storage unit 3720 in association with received identification information for identifying vehicle 3400 .

In A4003, the image processing unit 3711 analyzes image information, vehicle information, and other information. In the first embodiment, the image processing unit 3711 analyzes the acquired image information, for example, 07:22:10 on April 29, 2021 to 07:24:18 on April 29, 2021, the vehicle 3400 determines that the vehicle is traveling in fog. The image processing unit 3711 stores the analysis result for each time information in a predetermined area of the storage unit 3720 together with image information, vehicle information, and other information in association with identification information for identifying the vehicle 3400 .
In this embodiment, the server 3700 analyzes the image information, but the information display terminal 3800 that receives the image information may analyze the image information.

At A<b>4004 , the communication control unit 3712 transmits image information, vehicle information, analysis results, and other information to the information display terminal 3800 based on the instruction from the information display terminal 3800 . More specifically, the communication control unit 3712 displays image information, vehicle information, and other information after the designated date and time information regarding the vehicle identified by the identification information instructed from the information display terminal 3800. Send to the display terminal 3800 .

At A<b>4005 , the communication control unit 3712 waits for reception of an image processing request from the information display terminal 3800 . When the communication control unit 3712 determines that it has received an image processing request from the information display terminal 3800, it advances the process to A4006. For example, the communication control unit 3712 receives from the information display terminal 3800 a request to perform fog correction for the entire range of the image as an image processing request.

In A4006, the image processing unit 3711 executes image processing on the image based on the image processing request. For example, the image processing unit 3711 performs fog correction on the entire image based on the image processing request. Fog correction is image processing that adjusts the brightness of an image so as to improve poor visibility due to fog.

At A<b>4007 , the communication control unit 3712 transmits the image processing result of the image processing unit 3711 to the information display terminal 3800 .

[Information processing of information display terminal 3800]
Next, the flow of information processing of the information display terminal 3800 will be described. FIG. 11 is an activity diagram showing the flow of information processing of the information display terminal 3800 according to the first embodiment. The flow of information processing will be outlined below along this activity diagram.

In A4011, the display control unit 3811 displays an activation button for the external video player on the viewer. By selecting the activation button on the viewer, the communication control unit 3812 activates the external video player in the server 3700 . At this time, the communication control unit 3812 transmits to the server 3700 the file name of the moving image, video effect settings, parameters for disabling at least part of the operation unit of the external moving image player, and the like as “command line arguments”. Also, at this time, the communication control unit 3812 may transmit the reproduction start time, the reproduction end time, the presence or absence of a loop, and the like. Henceforth, the communication control part 3812 controls an external video player by sending a control command. At this time, the communication control unit 3812 preferably transmits an instruction to disable at least part of the operation unit of the external video player.

At A<b>4012 , the communication control unit 3812 waits to receive information from the server 3700 . When the communication control unit 3812 determines that information has been received from the server 3700, it advances the process to A4013. The communication control unit 3812 will be described as receiving image information, vehicle information, analysis information analyzed by the image processing unit 3711 and other information as information from the server 3700 . More specifically, in the first embodiment, the communication control unit 3812 receives image information, acceleration information, vehicle speed information, position information, analysis information, date and time information, etc. from the server 3700. FIG.

In A4013, the display control unit 3811 displays the received image information, vehicle information, analysis information, and other information. More specifically, in the first embodiment, the display control unit 3811 displays image information, acceleration information, vehicle speed information, position information, analysis information, date and time information, and the like.

In A4014, the display control unit 3811 determines whether or not the designation of the image processing range has been received. When the display control unit 3811 determines that the designation of the image processing range has been received, the process proceeds to A4015. When the display control unit 3811 determines that the designation of the image processing range has not been received, the process proceeds to A4016.
Description will be given on the assumption that the entire range of the image is subject to image processing when the range of image processing is not specified. That is, although omitted in FIG. 11 for the sake of simplification, when the range specification of image processing is not accepted, "the whole image" is held as range information. In the description of the first embodiment, it is assumed that "the whole image" is held as the range information.

In A4015, the communication control unit 3812 holds the image processing range information selected by the operator of the information display terminal 3800 via the screen or the like.

In A4016, the display control unit 3811 waits for acceptance of an instruction to execute image processing selected from among a plurality of image processing by the user via the screen. In the first embodiment, the display control unit 3811 will be described as having received an instruction to perform fog correction among a plurality of image processes from the user via the screen.

When the user issues an instruction to execute image processing, the communication control unit 3812 transmits an instruction to execute image processing to the server 3700 in A4017. In the first embodiment, the communication control unit 3812 transmits to the server 3700 an instruction to perform fog correction on the image.

At A<b>4018 , the communication control unit 3812 waits for reception of the resulting image processed by the server 3700 . Upon receiving the image, the communication control unit 3812 advances the process to A4019. In the first embodiment, the communication control unit 3812 receives the fog-corrected image processed by the server 3700 .

In A4019, the display control unit 3811 displays the information acquired from the server 3700 on the screen. In the first embodiment, the display control unit 3811 displays the fog-corrected image acquired from the server 3700 on the screen.

In this embodiment, the PC Browser (PC application), which is a viewer on the information display terminal 3800, controls the external video player in the server 3700. FIG. Examples of external video players include free software such as VLC, image processing services in the cloud, and image sharpening services. When installing the viewer of this embodiment, the display control unit 3811 asks whether to install an external video player as well. The storage control unit 3813 installs the external video player when receiving an instruction to install the external video player. At the time of installation of the external video player or immediately after starting the external video player, the display control unit 3811 displays guidance of a button for starting the external video player and cautions when using the external video player. The display control unit 3811 has the following points to note when using an external player: ``When the button of the external video player is operated, the display state may deviate from the viewer's display state''; After closing the external moving image player, the external moving image player should be started again from the viewer." or the like may be displayed.

When using a system on the cloud, the storage control unit 3813 may be configured to have a function of storing an ID or the like for automatic login to a cloud image sharpening service in the storage unit 3820 . Also, the communication control unit 3812 has a function of automatically transmitting information, a function of uploading information, a function of determining whether conversion is completed, and automatically downloading the information when the conversion is completed, or notifying that the download is possible. may be configured. Further, the display control unit 3811 preferably has a function of synchronizing the downloaded video with the pre-conversion video and playing it back.

Further, the display control unit 3811 performs control so as to display that the content can be reproduced finely by using a dedicated viewer when the content is reproduced using a non-recommended viewer. When a dark image is recorded, it is corrected and displayed clearly when played back with a dedicated viewer, but the played back image may be dark when played back with a standard player for an OS such as Windows (registered trademark). Therefore, for example, the display control unit 3811 provides black belt areas at the top and bottom of the video when playing back with a non-recommended viewer, and displays in the black belt areas, "This video will be played back cleanly with a dedicated viewer." control to At this time, the display control unit 3811 may display a code (for example, a QR code (registered trademark)) that refers to a URL where the viewer can be installed together with the character string. As another form, the display control unit 3811 may display a character string at the beginning of the video when the video is played back with a non-recommended viewer, such as "This video will be played back cleanly with a dedicated viewer." A code (for example, a QR code (registered trademark)) that refers to a URL where the viewer can be installed may be displayed.

[Details of information processing]
Next, details of the information processing described above will be described.

FIG. 12 is a diagram showing an example of a screen before image processing, displayed by the display control unit 3811 of the first embodiment. The display control unit 3811 controls to display information in the server 3700 on the viewer 4100 . Viewer 4100 is configured to display various function calls and information. More specifically, the viewer 4100 includes a file button 4101, a display button 4102, a play button 4103, a tool button 4104, and an information button 4105, and these buttons call and refer to various functions. becomes possible. Viewer 4100 also includes a video display area 4110, an acceleration graph display area 4120, a map image display area 4130, a date and time display area 4150, a vehicle state display area 4160, and a speed graph display area 4170. It is possible to refer to the image being reproduced and the information about the image by the area of . Further, the viewer 4100 includes an image display area 4110, a fog correction button 4140, an HDR button 4141, a sharpening button 4142, an other processing button 4143, a partial processing button 4144, a stream display button 4145, and operation buttons. 4146 and a playlist display area 4180, and the display control unit 3811 requests the server 3700 to perform image processing, image reproduction, and the like in accordance with the operation of these buttons. That is, the viewer 4100 is configured to be able to receive a plurality of user's operation instructions. At this time, each of the plurality of operation instructions instructs display of a different processing result regarding the image.

The file button 4101 is configured to be able to execute processing related to various file operations. More specifically, when the file button 4101 is selected, the display control unit 3811 displays file-related processes such as "file reference", "overwrite save", and "save as" on the screen in a pull-down format. Execution of the selected process is requested to the control unit 3810 or the like by displaying and selecting a process related to any file.

Display buttons 4102 enable operations related to the display of various video display areas 4110 . More specifically, when the display button 4102 is selected, the display control unit 3811 displays processing related to display such as “display side by side”, “display in line”, and “minimize” in a pull-down format. The selected process is executed by selecting the process related to the display.

A playback button 4103 enables execution of processing related to playback of various images. More specifically, when the play button 4103 is selected, the display control unit 3811 performs "play next playlist", "play previous playlist", "play from next image processing", " Processing related to reproduction such as "reproduce from previous image processing" is displayed in a pull-down format, and the selected processing is executed by selecting one of the processing related to display.

Tool buttons 4104 can invoke various functions. More specifically, when the tool button 4104 is selected, the display control unit 3811 displays a pull-down format such as "macro" or "user setting", and when any process is selected, the selection is made. It requests the control unit 3810 or the like to execute the process.

An information button 4105 can refer to various information. More specifically, when the tool button 4104 is selected, the display control unit 3811 displays buttons related to information such as "version information" in a pull-down format. Display the information that the user can refer to.

A video display area 4110 is an area in which images captured by the imaging devices 3100 and 3200 and images processed by the image processing unit 3711 are displayed. Also, the display format of the video display area 4110 can be arbitrarily changed according to the user's instruction. More specifically, in the video display area 4110, a plurality of images after image processing by different methods may be displayed side by side based on an instruction from the user, or an image after image processing and an image before image processing may be displayed. may be displayed side by side. Furthermore, arbitrary information such as an acceleration graph display area 4120 may be displayed in the image display area 4110 .

Acceleration graph display area 4120 is an area in which the acceleration of the vehicle in which the image is captured is displayed in three dimensions. In the acceleration graph display area 4120, the X axis is formed in the depth direction, the Y axis is formed in the horizontal direction, and the Z axis is formed in the vertical direction. Here, the X axis is the axis that indicates the acceleration in the front and rear directions of the vehicle, the Y axis is the axis that indicates the acceleration in the lateral direction of the vehicle, and the Z axis is the axis that indicates the acceleration in the vertical direction of the vehicle. be. Acceleration toward the front of the vehicle (positive acceleration) is graphed along the upper X-axis, and acceleration toward the rear of the vehicle (negative acceleration) is graphed along the lower X-axis. Acceleration to the right of the vehicle is plotted on the upper Y-axis, and acceleration to the left of the vehicle is plotted on the lower Y-axis. The upward acceleration of the vehicle is graphed leftward on the Z-axis, and the downward acceleration of the vehicle is graphed rightward on the Z-axis.

The map image display area 4130 is an area in which an image of a map near the shooting location of the image displayed in the image display area 4110 is displayed. Data representing the position of the vehicle is transmitted from the information display terminal 3800 to a map server (not shown) existing on the network 3900, and data representing a map near the position of the vehicle is transmitted from the map server to the information display terminal 3800. , a map image is displayed in the map image display area 4130 . At this time, a current location icon 4131 is displayed on the map image display area 4130 .

Viewer 4100 includes fog correction button 4140 . When the fog correction button 4140 is selected, the image processing unit 3711 performs fog correction on the image displayed in the image display area 4110 . More specifically, when the fog correction button 4140 is selected, the communication control unit 3812 of the information display terminal 3800 instructs the server 3700 to remove fog from the image. The server 3700 that has received the instruction uses the image processing unit 3711 to perform fog removal processing on the image. The communication control unit 3712 of the server 3700 transmits the defogged image to the information display terminal 3800 . The display control unit 3811 controls to display the image display area 4110 including the image subjected to the fog removal process. In other words, image correction can be performed by selecting a button on the PC Browser (PC application). At this time, when the fog correction button 4140 is operated again while fog correction is being performed, the image processing unit 3711 can perform control to cancel the fog correction.

The display control unit 3811 can also perform control so that the degree of fog correction is adjusted and displayed according to the number of times the fog correction button 4140 is selected or the selection time. More specifically, the display control unit 3811 controls to increase the degree of fog correction according to the number of times the fog correction button 4140 is pressed, Depending on the situation, the degree of fog correction is strengthened and displayed. That is, the display control unit 3811 receives an operation instruction for the fog correction button 4140 selected by the user, receives an instruction to display the processing result regarding the image corresponding to the operation instruction, and receives the image display area 4110 including the processing result regarding the image. is displayed. The fog correction button 4140 is an example of an object. An operation of selecting the fog correction button 4140 is an example of an object operation.

Viewer 4100 includes an HDR (High Dynamic Range) button 4141 . The image processing unit 3711 performs HDR processing on the image displayed in the video display area 4110 when the HDR button 4141 is selected. Here, the HDR process is a process of synthesizing a plurality of photographs with different brightnesses so as not to cause blown-out highlights or blocked-up shadows. More specifically, when the HDR button 4141 is selected, the communication control unit 3812 of the information display terminal 3800 instructs the server 3700 to HDR process the image. The server 3700 that has received the instruction uses the image processing unit 3711 to synthesize a plurality of photographs with different brightnesses and perform HDR processing. A communication control unit 3712 of the server 3700 transmits the HDR-processed image to the information display terminal 3800 . The display control unit 3811 controls to display the video display area 4110 including the HDR-processed image.

In addition, the display control unit 3811 receives an instruction to display the processing result regarding the image corresponding to the button selected by the user, and controls to display the screen including the processing result regarding the image. More specifically, the image processing unit 3711 receives a fog correction execution instruction by selecting the fog correction button 4140, receives an HDR processing execution instruction by selecting the HDR button 4141, and displays the processing result in the video display area 4110. control to display. Here, like the fog correction button 4140 and the HDR button 4141, buttons related to image processing are different for different processing results related to images. A fog correction button 4140 and an HDR button 4141 are examples of objects. As a result, arbitrary image processing corresponding to the buttons can be executed according to the user's selection. In addition, when displaying a plurality of buttons, the display control unit 3811 can control not to display them at the same time. More specifically, the display control unit 3811 can control the fog correction button 4140 and the HDR button 4141 not to be displayed at the same time.

Further, at this time, when the fog correction button 4140 and the HDR button 4141 are selected via the screen, the display control unit 3811 instructs the display of the first processing result regarding the image corresponding to the fog correction button 4140. , an instruction to display a second processing result regarding the image corresponding to the HDR button 4141, and controls to display a screen including the first processing result regarding the image and the second processing result regarding the image. You may do so. More specifically, when the fog correction button 4140 and the HDR button 4141 are selected via the viewer 4100, the display control unit 3811 displays the fog correction process result for the image corresponding to the fog correction button 4140. and an instruction to display the processing result of the HDR processing for the image corresponding to the HDR button 4141, and display a screen including the processing result of the fog correction processing for the image and the processing result of the HDR processing for the image. to control. As a result, a plurality of arbitrary image processes can be executed simultaneously. The fog correction button 4140 is an example of a first object. HDR button 4141 is an example of a second object.

Viewer 4100 includes a sharpen button 4142 . The image processing unit 3711 performs sharpening processing on the video displayed in the video display area 4110 when the sharpening button 4142 is selected. More specifically, when the sharpening button 4142 is selected, the communication control unit 3812 of the information display terminal 3800 instructs the server 3700 to perform the sharpening process. The server 3700 that has received the instruction sharpens the image using the image processing unit 3711 . The communication control unit 3712 of the server 3700 transmits the sharpened image to the information display terminal 3800 . The display control unit 3811 controls to display the image display area 4110 including the sharpened image.

Viewer 4100 includes other processing button 4143 . The display control unit 3811 controls to display various image processing options on the viewer 4100 when the other processing button 4143 is selected. Here, the options for image processing include, for example, an image processing adjustment area 6290, which will be described later. When this image processing option is selected, the communication control unit 3812 of the information display terminal 3800 instructs the server 3700 to perform image processing. The server 3700 that has received the instruction performs image processing using the image processing unit 3711 . The communication control unit 3712 of the server 3700 transmits the processed image to the information display terminal 3800 . The display control unit 3811 controls to display the image display area 4110 including the image that has undergone the image processing.

Viewer 4100 includes partial processing button 4144 . The display control unit 3811 controls display so that a part of the video display area 4110 can be selected by selecting the partial processing button 4144 . When this image processing option is selected, the communication control unit 3812 of the information display terminal 3800 instructs the server 3700 to perform image processing on part of the image. Upon receiving the instruction, the server 3700 uses the image processing unit 3711 to process a part of the image. The communication control unit 3712 of the server 3700 transmits the processed image to the information display terminal 3800 . The display control unit 3811 controls to display a video display area 4110 including an image partially subjected to image processing. Here, the display control unit 3811 uses the image before image processing and the image after image processing, and controls to display a part of them in the video display area 4110 including the image on which image processing has been performed. may Note that the method for performing the partial processing may be another aspect. For example, the display control unit 3811 receives designation of a correction target range for the image display area 4110 . The display control unit 3811 further provides a range registration check box. The display control unit 3811 may set the specified range as the correction target range while the correction button is pressed with the range registration check box checked. At this time, the display control unit 3811 may provide a button for temporarily canceling the correction while the range remains corrected.

Viewer 4100 includes stream display button 4145 . When the stream display button 4145 is selected, the display control unit 3811 controls to display a plurality of images side by side in the video display area 4110 .

Each button displayed in viewer 4100 may have other forms. For example, each button displayed on the viewer 4100 may be a radio button, check box, toggle button, or the like instead of a push button. Further, for example, each button has been described as a soft button displayed on the viewer 4100, but it may be provided in a place other than the viewer 4100, and may be a physical button such as a physical switch.

The date and time display area 4150 is an area in which the recording date and time of the image displayed in the image display area 4110 is displayed. Also, here, the date and time display area 4150 may be configured to display the current date and time.

Vehicle status display area 4160 is an area in which information such as vehicle speed at the time of video recording is displayed. A vehicle state display area 4160 displays the state of the vehicle to which the photographing devices 3100 and 3200 that record the images displayed in the image display area 4110 are attached.

Speed graph display area 4170 is an area where the speed of the vehicle is graphed and displayed. In addition to velocity, acceleration may also be graphed and displayed in velocity graph display area 4170 .

A play list display area 4180 is an area where information of saved moving image data is displayed. The display control unit 3811 accesses the video data in the server 3700 and displays it in the video display area 4110 by selecting the information of the video data in the playlist display area 4180 . At this time, the video data may be stored in the information display terminal 3800 as well as in the server 3700, or may be stored in an external storage device.

FIG. 13 is a diagram showing an example of a screen after image processing, displayed by the display control unit 3811 of the first embodiment. The display control unit 3811 controls the information processed in the server 3700 to be displayed on the viewer 4200 . The image display area 4210 of the viewer 4200 in the first embodiment is an image obtained by fogging the image display area 4110 of FIG. 12 by selecting the fog correction button 4140 . That is, the video display area 4210 includes an image after image processing. At this time, the fog correction button 4240 is displayed in a different color from the other buttons so that it can be seen that it has been selected. Furthermore, at this time, the storage control unit 3813 stores the image captured by the imaging device 3100 in the storage area. As a result, the information display terminal 3800 can store an image before image processing in the storage unit 3820 while displaying an image after image processing in the video display area 4210 .

In the first embodiment, an example has been described in which the storage control unit 3813 controls to store an image before image processing, and the display control unit 3811 controls to display an image after image processing. The image before image processing may be reproduced and displayed using the image after image processing. More specifically, the storage control unit 3713 controls the storage unit 3720 to store the image after image processing and the image processing performed on the image. Next, the image processing unit 3711 controls to reproduce the image before the image processing based on the information of the executed image processing for the image after the image processing. After that, the display control unit 3811 controls to display the reproduced image before image processing. Also, at this time, the display control unit 3811 may display which data the image displayed in the video display area 4210 is. More specifically, the display control unit 3811 determines whether the data being displayed is the original data, the data after image processing, or the data reproduced from the data after image processing to the data before image processing. You may make it display whether it is data.

[Modification 1]
Modification 1 of Embodiment 1 will be described.
FIG. 14 is a diagram showing an example of a screen after image processing, displayed by the display control unit 3811 of Modification 1 of Embodiment 1. As shown in FIG. An image display area 4310 of the viewer 4300 in Modification 1 includes a fog-corrected area 4311 in which part of the image display area 4310 has been subjected to fog correction, and an area different from the fog-corrected area 4311 has been subjected to a sharpening process. and a region 4313 . As described above, the image processing related to the fog correction button 4340 and the image processing related to the sharpening button 4342 of the modified example 1 are configured so that the image processing target area in the image is different. The fog correction button 4340 is an example of a first object. A sharpen button 4342 is an example of a second object. The information display terminal 3800 of Modification 1 can execute arbitrary image processing on different regions.
For example, the user selects a predetermined area (area corresponding to the fog correction area 4311 ) in the image display area 4110 and selects the fog correction button 4340 . Also, the user selects a predetermined area (area corresponding to the sharpening area 4313 ) in the video display area 4110 and selects the sharpening button 4342 . By making such a selection, a screen such as that shown in FIG. 14 can be displayed.
In addition, the display control unit 3811 can display what kind of processing has been performed in which region by displaying the processing content in each processing region. That is, the display control unit 3811 can display the processing content display unit 4312 in the fog correction area 4311 so that the fog correction area 4311 has performed fog correction. In addition, the display control unit 3811 can display the processing content display unit 4314 in the sharpening area 4313 so that the sharpening process is performed. Furthermore, the display control unit 3811 can display that these processes are being executed by changing the colors of the fog correction button 4340, sharpening button 4342, and partial processing button 4344. FIG.
In the example of modification 1, fog correction and sharpening have been described as examples of image processing for different regions, but the present invention is not limited to this, and other image processing may be used. Also, even if the same image processing (for example, fog correction) is performed, different degrees of processing may be performed on different regions.

[Modification 2]
Modification 2 of Embodiment 1 will be described.
FIG. 15 is a diagram showing an example of a screen before image processing, displayed by the display control unit 3811 according to Modification 2 of Embodiment 1. As shown in FIG. Objects displayed on the viewer 4400 in Modification 2 are a sunny correction button 4440 , a cloudy correction button 4441 , a rain correction button 4442 , and a fog correction button 4443 . A fine weather correction button 4440, a cloudy correction button 4441, a rain correction button 4442, and a fog correction button 4443 are examples of objects related to weather-related image processing. In Modified Example 2, the viewer uses weather information to perform different image corrections for fine weather, rain, and fog.
The display control unit 3811 receives instructions for image quality correction such as gamma correction, color correction (color enhancement, color balance), edge enhancement, etc. corresponding to the weather by selecting each button. For example, the display control unit 3811 performs image processing such as gamma correction, color correction (color enhancement, color balance), and edge enhancement on the video so as to erase rain by selecting the rain correction button 4442. Request to 3711. In addition, the display control unit 3811 performs gamma correction, color correction (color enhancement, color balance), and edge enhancement to make the fog transparent so that the other side of the fog can be seen by selecting the fog correction button 4443 . The image processing unit 3711 of the server 3700 is requested to perform image processing such as the above. In addition, the display control unit 3811 provides the button names such as an operation button for erasing a snowstorm, a button for making fog transparent so that the other side of the fog can be seen, a button for erasing rain, and the like. You may display the button name which correct|amends. Furthermore, the display control unit 3811 allows the viewer 4400 to display not only "clear", "cloudy", "rain" and "fog", but also "sunny", "lightly cloudy", "smoke", "dust storm", Objects related to blizzard, drizzle, sleet, snow, hail, hail, thunder, etc. may be newly provided or alternatively provided. Instead of providing each object for each weather, an all-around automatic correction may be performed by automatically detecting these weather conditions and automatically correcting them. When the all-around automatic correction is configured to be possible, the display control unit 3811 displays a button for performing/not performing image correction on the screen. If the display control unit 3811 determines that there is a limit to the automatic correction of the almighty, it creates some kind of state. For example, the display control unit 3811 displays another correction means. As a result, it is possible to perform appropriate image processing in accordance with the weather.

[Modification 3]
Modification 3 of Embodiment 1 will be described.
FIG. 16 is a diagram illustrating an example of a screen before image processing, displayed by the display control unit 3811 according to the third modification of the first embodiment. In the image display area 4510 of the viewer 4500 in the third modification, an image captured by the image capturing device 3100 of the vehicle 3400 traveling on the foggy road surface is displayed.
At this time, the display control unit 3811 performs control to emphasize and display the fog correction button 4540 as an appropriate correction method. That is, the display control unit 3811 changes the buttons displayed on the screen based on the state of the image. Accordingly, appropriate image processing can be performed based on the information in the image. As one mode of changing the button, the display control unit 3811 can newly display the button. Further, as another mode of changing the button, the outline of the button can be emphasized. Furthermore, as still another mode of changing the buttons, the buttons of the display control unit 3811 can be illuminated and displayed. For example, when the vehicle 3400 travels in the city or in the woods, the brightness changes depending on the location and/or direction of travel. , the outline of the brightness correction button may be emphasized, or the brightness correction button may be illuminated and displayed.

[Modification 4]
Modification 4 of Embodiment 1 will be described.
FIG. 17 is a diagram illustrating an example of a screen before image processing, displayed by the display control unit 3811 according to Modification 4 of Embodiment 1. As illustrated in FIG. In the image display area 4610 of the viewer 4600 in Modification 4, an image photographed by the photographing device 3100 of the vehicle 3400 immediately before entering the tunnel is displayed. At this time, the current location icon 4631 in the map image display area 4630 indicates that the vehicle 3400 is about to enter the tunnel.
A communication control unit 3812 of the information display terminal 3800 acquires road network information related to the position information of the vehicle 3400 from a map server (not shown) via the network 3900 . Then, the display control unit 3811 controls to display a tunnel correction button 4640 suitable for image processing inside the tunnel. That is, the display control unit 3811 performs control to display on the screen a button suggesting image processing for the image based on the road network information. Here, by selecting the tunnel correction button 4640, the image processing unit 3711 automatically performs an appropriate type of correction on the tunnel position. Specifically, when the tunnel correction button 4640 is selected, the image processing unit 3711 adjusts the brightness so as to correct the darkness of the tunnel, or reduces the effects of lighting provided in the tunnel. , image processing is performed to reduce the influence of the lighting equipment of the vehicle ahead. Therefore, appropriate image processing can be performed based on the road network information. Here, the road network information is information about road conditions. More specifically, in addition to tunnel position information, speed regulation information, road width information, weekday 12-hour traffic information, travel This information includes speed information (for a certain section, the distance of the section is divided by the time required to travel, and the required time includes stops due to traffic lights, traffic jams, etc.), lane number information, and the like.

[Modification 5]
Modification 5 of Embodiment 1 will be described.
FIG. 18 is a diagram illustrating an example of a screen before image processing, displayed by the display control unit 3811 according to the fifth modification of the first embodiment. The sensor unit 3017 of Modification 5 measures acceleration information of the vehicle 3400 . That is, the vehicle information of Modified Example 5 includes acceleration information measured by the sensor unit 3017 . In the image display area 4710 of the viewer 4700 in Modification 5, an image captured by the imaging device 3100 of the rapidly accelerating vehicle 3400 is displayed. At this time, the acceleration graph display area 4720 is a graph showing a large slope in the upward direction of the X-axis, and the speed graph display area 4770 is a graph showing a rapid increase in speed. At this time, the viewer 4700 may be provided with an acceleration display section 4721 such as "1 G or more detected".
The display control unit 3811 controls to display an acceleration correction button 4740 suitable for image processing when the vehicle 3400 accelerates. That is, the display control unit 3811 controls to display the acceleration correction button screen based on the information from the sensor unit 3017 that detects the acceleration of the vehicle 3400 having the imaging device 3100 . Here, when the acceleration correction button 4740 is selected, image processing for correcting screen shake and/or poor visibility due to sudden acceleration is performed. Accordingly, appropriate image processing can be performed based on the information from the acceleration sensor of the moving body. The imaging device 3100 is an example of an imaging unit. Vehicle 3400 is an example of a mobile object. Sensor unit 3017 is an example of an acceleration sensor. An acceleration correction button is an example of an object that suggests image processing for an image.
The display control unit 3811 also receives one or more acceleration correction settings from the user. Specifically, the display control unit 3811 accepts one or more acceleration correction settings by the user's input of a numerical value for acceleration correction, dragging to the acceleration graph display area 4720 or the speed graph display area 4770, or the like. At this time, the display control unit 3811 displays a band of a different color for each type of correction in the acceleration graph display area 4720 or the velocity graph display area 4770, or displays the dragged area.

[Modification 6]
Modification 6 of Embodiment 1 will be described.
FIG. 19 is a diagram showing an example of a screen before image processing, displayed by the display control unit 3811 according to Modification 6 of Embodiment 1. As shown in FIG. In the video display area 4810 of the viewer 4800 in Modification 6, the video captured by the imaging device 3100 of the vehicle 3400 is displayed. At this time, the current location icon 4831 in the map image display area 4830 where the vehicle 3400 is traveling indicates the location where the vehicle 3400 is traveling on the map.
The display control unit 3811 can set a processing area 4832 on the map image display area 4830 based on a user operation or the like, and controls to display the fog correction button 4840 when the current location icon 4831 enters this area. do. That is, the display control unit 3811 performs control to display on the screen a button or the like suggesting image processing for an image based on the position information of the vehicle 3400 having the imaging device 3100 . Accordingly, appropriate image processing can be performed based on the position information of the vehicle 3400. FIG.
As described above, the processing area 4832 may be set based on a user operation or the like, or may be set automatically. For example, if it is set to display an image processing button that brightens the image when entering a tunnel, the display control unit 3811 determines that the vehicle 3400 has entered the tunnel based on the position information of the vehicle 3400, and displays the image. control to display the image processing button to brighten the image on the screen.

[Modification 7]
Modification 7 of Embodiment 1 will be described.
FIG. 20 is a diagram showing an example of a screen before image processing, displayed by the display control unit 3811 according to Modification 7 of Embodiment 1. As shown in FIG. In the image display area 4910 of the viewer 4900 in Modification 7, an image captured by the imaging device 3100 of the vehicle 3400 on a cloudy and then rainy road is displayed. At this time, weather information 4911 for that road can be displayed on the viewer 4900 . In this embodiment, the viewer 4900 displays weather information 4911 about cloudy and rainy weather.
The communication control unit 3812 obtains weather information for the route traveled by the vehicle 3400 from a weather information providing server (not shown) based on the location information of the vehicle 3400 and the date and time information included in the image captured by the imaging device 3100. get. Then, the display control unit 3811 performs control to display a button suggesting image processing based on the location information of the vehicle 3400 and the weather information corresponding to the location information. That is, the display control unit 3811 controls to display on the screen an object suggesting image processing for the image based on the positional information of the vehicle 3400 having the imaging device 3100 and the weather information of the date and time when the image was taken. In Modified Example 7, image correction can be applied according to the time period recorded by the viewer. For example, the image processing unit 3711 can apply different image corrections in the morning, at night, and in the evening based on the instruction from the communication control unit 3812 and the weather information 4911 .
In the example of modification 7, the display control unit 3811 controls the viewer 4900 to display the rain correction button 4941 based on the position information of the vehicle 3400 and the weather information. Here, when the rain correction button 4941 is selected, image processing for correcting darkness caused by rain clouds and poor visibility caused by rain is executed. Accordingly, appropriate image processing can be performed based on the location information of the vehicle 3400 and the weather information of the date and time at that location. Here, the weather information includes not only cloudy weather and rain, but also fine weather, fine weather, slightly cloudy weather, haze, dust storm, blizzard, drizzle, sleet, fog, snow, hail, hail, and thunder.

[Modification 8]
Modification 8 of Embodiment 1 will be described.
FIG. 21 is a diagram showing an example of a screen before image processing, displayed by the display control unit 3811 according to the eighth modification of the first embodiment. In the video display area 5010 of the viewer 5000 in Modification 8, the video captured by the camera 3100 of the vehicle 3400 approaching the intersection is displayed. At this time, the current location icon 5031 in the map image display area 5030 indicates that the vehicle 3400 is approaching the intersection. Speed graph display area 5070 shows how vehicle 3400 decelerates as vehicle 3400 approaches an intersection.
The display control unit 3811 controls to display the brake correction button 5040 according to the deceleration of the vehicle 3400 . That is, the display control unit 3811 performs control to display on the screen an object suggesting image processing for the image based on the operation information of the vehicle 3400 having the imaging device 3100 . Here, by selecting the brake correction button 5040, blurring of the field of view due to deceleration can be reduced, and the influence of the brake lamps on the image display area 5010 can be reduced. Accordingly, appropriate image processing can be performed based on the operation information of the vehicle 3400. FIG. Here, the operation information of the vehicle 3400 is information related to operation among the vehicle information. In addition to the brake, the vehicle information includes operation of the blinker, reverse, and other information that can be acquired by OBD (On-board diagnostics).

[Modification 9]
Modification 9 of Embodiment 1 will be described.
FIG. 22 is a diagram showing an example of a screen before image processing, displayed by the display control unit 3811 according to the ninth modification of the first embodiment. In the image display area 5110 of the viewer 5100 in Modification 9, the image captured by the image capturing device 3100 of the vehicle 3400 is displayed. At this time, information about the date and time is displayed in the date and time display area 5150 . The example in FIG. 22 indicates that the time is night.
The display control unit 3811 controls to display the night correction button 5140 based on the date and time information, which is the image attribute information. That is, the display control unit 3811 performs control to display on the screen a button suggesting image processing for the image based on the date and time when the image was captured. Here, by selecting the night correction button 5140, it is possible to correct the darkness of the night and display what is around. Accordingly, it is possible to perform appropriate image processing based on the date and time when the image was taken.
Here, the night correction button is an example of a button that suggests image processing for an image based on the date and time.
It should be noted that the automatic correction may be performed by combining the corrections of Modifications 3 to 9 described above. A combination of corrections of Modifications 3 to 9 may be set using an OR condition, or may be set using an AND condition.

[Modification 10]
Modification 10 of Embodiment 1 will be described.
FIG. 23 is a diagram showing an example of a screen after image processing, displayed by the display control unit 3811 of Modification 10 of Embodiment 1. As shown in FIG. In the video display area 5210 of the viewer 5200 in Modification 10, an image captured by the image capture device 3100 of the vehicle 3400 traveling on a foggy road surface is displayed. Also, in a fog correction area 5211 that is part of the image display area 5210, the result of fog correction is displayed. That is, in the fog correction area 5221, the result of image processing performed on part of the image captured by the imaging device 3100 is displayed. When an instruction to perform image processing on the fog correction area 5221 is given, the image processing unit 3711 performs image processing on part of the image captured by the imaging device 3100 . A fog correction area 5211 is an area set according to a user operation or the like before performing image processing. In other words, the part of the image is the set area out of the entire area of the image captured by the imaging device 3100 .
More specifically, when the partial treatment button 5245 is selected, the display control unit 3811 sets the range of image processing by specifying the range on the screen. Next, when the fog correction button 5240 is selected, the display control unit 3811 determines that the set range is the fog correction area 5211 where fog correction is performed. Then, the communication control unit 3812 of the information display terminal 3800 requests the server 3700 via the network 3900 to perform fog correction on the fog correction area 5211 . Upon receiving the request, the image processing unit 3711 performs fog correction on the fog correction area 5211 . The communication control unit 3812 acquires the result of fog correction for the fog correction area 5211 . The display control unit 3811 displays the result of image processing performed on the fog correction area 5211 . Accordingly, image processing can be performed on a portion of the captured image. Also, this makes it possible to set an area for which image processing is to be performed.

[Modification 11]
Modification 11 of Embodiment 1 will be described.
FIG. 24 is a diagram showing an example of a screen after image processing, displayed by the display control unit 3811 according to the eleventh modification of the first embodiment. In the image display area 5310 of the viewer 5300 in Modification 11, as an example, an image captured by the imaging device 3100 of the vehicle 3400 in the middle of the night is displayed. Also, in a night correction area 5311 that is part of the image display area 5310, the result of night correction is displayed. Further, the date and time display area 5350 indicates that the time is night. When the date and time display area 5350 is night, the night correction area 5311 displays the result of night correction. That is, image processing for an image is image processing for a part of an image captured by the imaging device 3100, and the part of the image is an image captured by the imaging device 3100 during a set time zone. This is an image.
The display control unit 3811 sets the range of image processing based on the operation of the partial treatment button 5345 and the date and time information of the image. Next, based on the operation of the night correction button 5340, the display control unit 3811 determines that the range is the night correction area 5311 where correction is performed at night time. Furthermore, the communication control unit 3812 instructs the server 3700 via the network 3900 to perform image processing on the night correction area 5311 . Next, the image processing section 3711 executes image processing on the night correction area 5311 . Furthermore, the communication control unit 3812 acquires the result of image processing performed on the night correction area 5311 . Finally, the display control unit 3811 displays the result of image processing performed on the night correction area 5311 . Accordingly, it is possible to appropriately perform image processing on a partial area of an image captured during the set time period.
In Modified Example 11, the partial treatment button 5345 and the night correction button 5340 have been described as examples. It is also possible to instruct to execute the set image processing for the images shot during the time period.

[Modification 12]
Modification 12 of Embodiment 1 will be described.
FIG. 25 is a diagram showing an example of a screen after image processing, displayed by the display control unit 3811 of Modification 12 of Embodiment 1. As shown in FIG. The location information acquisition unit 3020 of Modification 12 acquires the location information of the vehicle 3400 . That is, the other information of Modification 12 includes the position information acquired by the position information acquisition unit 3020 . In the image display area 5410 of the viewer 5400 in the twelfth modification, an image captured by the imaging device 3100 of the vehicle 3400 is displayed. Also, in a fog correction area 5411 that is part of the image display area 5410, the result of fog correction is displayed. Further, a current location icon 5431 in the map image display area 5430 indicates the location where the vehicle 3400 travels on the map. A processing area 5432 is displayed in the map image display area 5430 . When the current location icon 5431 is within the processing area 5432, the result of fog correction is displayed in the fog correction area 5411. FIG. That is, the image processing for an image is image processing for a part of the image captured by the imaging device 3100, and the part of the image means that the position information of the vehicle 3400 having the imaging device 3100 is within a set range. This is a photographed image.
First, the display control unit 3811 sets the range of image processing based on the operation of the partial treatment button 5445 and the position information. Next, based on the operation of the fog correction button 5440, it is determined that the range is the fog correction area 5411 where fog correction is performed at a predetermined position. Furthermore, the communication control unit 3812 instructs the server 3700 via the network 3900 to perform image processing on the fog correction area 5411 . Next, the image processing section 3711 executes image processing on the fog correction area 5411 . Furthermore, the communication control unit 3812 acquires the result of image processing performed on the fog correction area 5411 . Finally, the display control unit 3811 displays the result of image processing performed on the fog correction area 5411 . Accordingly, it is possible to appropriately perform image processing on a partial area of an image captured by vehicle 3400 moving in a set position.
In the modification 12, the partial treatment button 5445 and the fog correction button 5440 have been described as an example. It is also possible to instruct execution of set image processing on an image captured at the input latitude and longitude.
Note that input of position information is not limited to latitude information and longitude information. It is also possible to instruct to execute image processing.
In addition, the display control unit 3811 displays the history route information of the user's vehicle on the viewer 5300, and the set image processing is applied to the image captured at the position within the range selected by the user on the displayed history route. may be instructed to execute

[Modification 13]
Modification 13 of Embodiment 1 will be described.
FIG. 26 is a diagram showing an example of a screen after image processing, displayed by the display control unit 3811 of Modification 13 of Embodiment 1. As shown in FIG. In the image display area 5510 of the viewer 5500 in the thirteenth modification, an image captured by the imaging device 3100 of the vehicle 3400 on a cloudy and then rainy road is displayed. Also, in a fog correction area 5511 that is part of the image display area 5510, the result of fog correction is displayed. In addition, the viewer 5500 can display weather information 5512 for that road. Here, the viewer 5500 displays weather information 5512 relating to rain after cloudy weather. If the weather at the date and time of the reproduced image is cloudy, the result of cloudiness correction is displayed in the cloudiness correction area 5511 . That is, the image processing for an image is image processing for a part of the image captured by the imaging device 3100, and the part of the image is the weather at the date and time when the image at the position of the vehicle 3400 having the imaging device 3100 was captured. It is the image which shows the weather with which the information was set.
First, the communication control unit 3812 of the information display terminal 3800, based on the location information of the vehicle 3400 and the date and time information included in the image captured by the image capturing device 3100, determines whether the vehicle 3400 is detected by a weather information providing server (not shown). Get weather information for the route you traveled. Next, the display control unit 3811 sets the range of image processing based on the operation of the partial treatment button 5545, time information, weather information, and the like. Further, based on the operation of the fogging correction button 5540, the range is determined to be the fogging correction area 5511 where the fogging correction is performed at a predetermined position. Furthermore, the communication control unit 3812 instructs the server 3700 via the network 3900 to perform image processing on the fogging correction area 5511 . Next, the image processing section 3711 performs image processing on the fogging correction area 5511 . Furthermore, the communication control unit 3812 acquires the result of image processing performed on the fogging correction area 5511 . Finally, the display control unit 3811 displays the result of image processing performed on the fogging correction area 5511 . As a result, it is possible to appropriately perform image processing on a partial region of the image captured based on the weather at the position of the vehicle 3400 . Here, the weather information includes not only cloudy weather and rain, but also fine weather, fine weather, slightly cloudy weather, haze, dust storm, blizzard, drizzle, sleet, fog, snow, hail, hail, and thunder.

[Modification 14]
Modification 14 of Embodiment 1 will be described.
FIG. 27 is a diagram illustrating an example of a screen after image processing, displayed by the display control unit 3811 according to the fourteenth modification of the first embodiment. The sensor unit 3017 of Modification 14 measures acceleration information of the vehicle 3400 . That is, the vehicle information of Modified Example 14 includes acceleration information measured by the sensor unit 3017 . In the video display area 5610 of the viewer 5600 in the fourteenth modification, a video captured by the imaging device 3100 of the vehicle 3400 that is rapidly accelerating is displayed. At this time, the acceleration graph display area 5620 is a graph in which a large slope can be seen in the upward direction of the X-axis, and the speed graph display area 5670 is a graph in which the speed is rapidly increasing. At this time, the viewer 5600 may be provided with an acceleration display section 5621 such as "1 G or more detected". Also, in an acceleration correction area 5611 that is part of the video display area 5610, the result of acceleration correction is displayed. Furthermore, when the acceleration sensor of vehicle 3400 detects 1 G or more, the result of acceleration correction is displayed in acceleration correction area 5611 . That is, the image processing for the image is image processing for a part of the image captured by the imaging device 3100, and the part of the image is an image within the threshold for which the sensor information of the vehicle 3400 having the imaging device 3100 is set. is.
First, the communication control unit 3812 of the information display terminal 3800 acquires image information, acceleration information of the vehicle 3400 and the like from the server 3700 via the network 3900 . Next, the display control unit 3811 sets the range of image processing based on the operation of the partial treatment button 5645, the acceleration information of the vehicle 3400, and the like. Further, based on the operation of the acceleration correction button 5640, it is determined that the range is the acceleration correction area 5611 in which acceleration correction is performed at a predetermined position. Next, the communication control unit 3812 instructs the server 3700 via the network 3900 to perform image processing on the acceleration correction area 5611 . Furthermore, the image processing unit 3711 performs image processing on the acceleration correction area 5611 . Next, the communication control unit 3812 acquires the result of image processing performed on the acceleration correction area 5611 . Finally, the display control unit 3811 displays the result of image processing performed on the acceleration correction area 5611. FIG. Accordingly, it is possible to appropriately perform image processing on a partial area of an image captured based on sensor information of the vehicle 3400 . In addition to the acceleration of the vehicle 3400, the sensor information includes the speed of the vehicle 3400 (vehicle speed), engine speed, engine load factor, throttle opening, ignition timing, percentage of remaining fuel, intake manifold pressure, intake air amount (MAF), injection opening time, temperature of engine cooling water (cooling water temperature), temperature of air taken into the engine (intake air temperature), air temperature outside the vehicle (outside air temperature), amount of remaining fuel in the fuel tank (remaining fuel amount), fuel flow rate, instantaneous fuel consumption, accelerator opening, turn signal information (left and right turn signal operation (on/off)), brake opening, steering angle of steering wheel, gear position, and information on door open/close status, etc. Contains information.
It should be noted that the automatic correction may be performed by combining the corrections of Modifications 10 to 14 described above. A combination of corrections from modified examples 10 to 14 may be set using an OR condition, or may be set using an AND condition.

[Modification 15]
Modification 15 of Embodiment 1 will be described.
FIG. 28 is a diagram showing an example of a screen after image processing, displayed by the display control unit 3811 of Modification 15 of Embodiment 1. As shown in FIG. In the image display area 5710 of the viewer 5700 in the fifteenth modification, an image captured by the imaging device 3100 of the vehicle 3400 is displayed.
The display control unit 3811 performs control so as to set an automatic fog correction area 5711 as a range in which correction is applied from preset settings, and to set a manual fog correction area 5713 as a range to apply correction from user's operation. In addition, the display control unit 3811 causes the processing content display unit 5712 to display "automatic processing" for the automatic fog correction area 5711, and the processing content display unit 5714 to display "manual processing" for the manual fog correction area 5713. control to display. That is, the screen includes information as to whether the range for which image processing is to be performed has been automatically selected or manually selected. As a result, it is possible to distinguish between the manually added range and the automatically added range of the executed image processing.

[Modification 16]
Modification 16 of Embodiment 1 will be described.
FIG. 29 is a diagram showing an example of a screen displayed by the display control unit 3811 of Modification 16 of Embodiment 1. As shown in FIG. In the video display area 5810 of the viewer 5800 in Modification 16, an original image display portion 5811 before image processing, a processing 1 result display portion 5813 after image processing, a processing 2 result display portion 5815 after image processing, and an image A processing 3 result display portion 5817 after processing is displayed as a plurality of different streams. An image captured by the image capturing device 3100 of the vehicle 3400 is displayed on each display unit. That is, the screen includes an image captured by the imaging device 3100 and an image after one or more image processes.
The display control unit 3811 controls to display a plurality of images by selecting the stream display button 5845 . Further, here, the display control unit 3811 may arrange and display a plurality of images in chronological order. Further, here, the display control unit 3811 displays the original display unit 5812 such as "original" on the original image display unit 5811, and displays the contents of the processing 1 such as "processing 1" on the processing 1 result display unit 5813. 5814 is displayed, the processing 2 content display section 5816 such as "processing 2" is displayed in the processing 2 result display section 5815, and the processing 3 content display such as "processing 3" is displayed in the processing 3 result display section 5817. It controls to display the part 5818 . Thereby, before and after image processing can be compared and displayed.
Further, the display control unit 3811 displays the original image display unit 5811, the process 1 result display unit 5813, the process 2 result display unit 5815, and the process 3 result display unit 5817 as multiple streams. Furthermore, the display control unit 3811 selectively displays an original image display unit 5811, a processing 1 result display unit 5813, a processing 2 result display unit 5815, and a processing 3 result display unit 5817 as a write target, a playback target, or the like. At this time, the display control unit 3811 can control the selected display unit to reproduce or output as an image file. In modification 16, the display control unit 3811 opens a plurality of processing 1 result display units 5813, processing 2 result display units 5815, and processing 3 result display units 5817, which are candidate video windows, and displays the original image display unit 5811, which is the source video window. Play in sync with the time. The display control unit 3811 accepts the selection of the best one among them by the user's operation of the selection button or the like. The storage control unit 3813 stores the selection result and the relationship between the information of the original video in the storage unit 3820 . Furthermore, the storage control unit 3813 causes the learning model to perform machine learning of the user's image processing tendency based on the stored selection result and the relationship with the information of the original video, and stores the result in the storage unit 3820. do. The display control unit 3811 adjusts image processing based on the results of machine learning. That is, the display control unit 3811 adjusts the image processing parameters based on the learning information regarding the selection of one or more images after image processing. Thereby, the selected image processing result can be learned, and subsequent image processing can be performed more appropriately. Also, the display control unit 3811 may perform image processing on a part of the image.

[Modification 17]
Modification 17 of Embodiment 1 will be described.
FIG. 30 is a diagram showing an example of a screen after image processing, displayed by the display control unit 3811 according to the seventeenth modification of the first embodiment. In the video display area 5910 of the viewer 5900 in Modification 17, the video captured by the imaging device 3100 of the vehicle 3400 is displayed.
When the fog correction button 5940 is selected, the display control unit 3811 controls to display the fog correction image processing result in the image display area 5910 . Further, the display control unit 3811 controls the image display area 5910 so that the processing content display unit 5911 displays “Fog correction in progress”. That is, the screen includes information about what kind of image processing was performed on the image captured by the image capturing device 3100 . This makes it possible to confirm what kind of image processing has been performed.

[Modification 18]
Modification 18 of Embodiment 1 will be described.
FIG. 31 is a diagram showing an example of a screen displayed by the display control unit 3811 of Modification 18 of Embodiment 1. As shown in FIG. Recording information in the storage unit 3720 or the storage unit 3820 is displayed in the playlist display area 6080 of the viewer 6000 in the eighteenth modification.
The storage control unit 3813 stores information about what kind of image processing was performed on the image captured by the image capturing device 3100 in the storage unit 3820 as log information. At this time, the storage control unit 3813 divides and records the log information into static log information that does not change during shooting and dynamic log information that changes depending on the video state during shooting. can be controlled as follows. More specifically, the storage control unit 3813 stores one static file for one file and one dynamic file for at least one change point. Control to record. Further, in Modified Example 18, a function is provided for recording according to the user's selection of recording with or without fogging removal processing, and information that enables determination of which one was selected by the viewer. may be provided with a function to record The storage control unit 3713 of the server 3700 can perform similar processing on the storage unit 3720 . The display control unit 3811 can control to read and process these pieces of log information at different timings. In addition, the storage control unit 3813 is configured to record drive recorder parameters relating to image quality, such as drive recorder image sensors and ISP parameters described later, in association with drive recorder video data shot with the parameters. good too. At this time, the display control unit 3811 may perform playback using this parameter during playback with a viewer or drive recorder. The display control unit 3811 may have a function of switching whether to perform playback without using this parameter when performing playback using this parameter.
Also, at this time, the display control unit 3811 can display information about what kind of image processing has been performed on the image captured by the imaging device 3100 . That is, the display control unit 3811 performs control to display what timing and what image processing was performed on the image captured by the image capture device 3100, as in the playlist 6081 . This makes it possible to confirm what kind of image processing has been performed.
Furthermore, when playing back the playlist 6081, the display control unit 3811 may display so as to select whether to play back the image before image processing or play back the image after image processing based on the log information. Further, at this time, the display control unit 3811 may be configured to convert an image after image processing into an image before image processing based on the log information and output the image to the screen. At this time, the image processing unit 3711 performs image processing so as to restore the original image based on the log information regarding the image processing in the storage unit 3720 and the image after the image processing. Next, the communication control unit 3812 acquires the image restored to the original image. Furthermore, the display control unit 3811 displays the image restored to the original image. As a result, the image after image processing can be restored to the original image before image processing.

[Modification 19]
Modification 19 of Embodiment 1 will be described.
FIG. 32 is a diagram showing an example of a screen before image processing, displayed by the display control unit 3811 according to the nineteenth modification of the first embodiment. In the image display area 6110 of the viewer 6100 in Modification 19, an image captured by the imaging device 3100 of the vehicle 3400 running in the fog is displayed.
An image processing unit 3711 of the server 3700 analyzes the image and detects fog. Next, the communication control section 3712 transmits the fog detection result to the information display terminal 3800 . After that, the display control unit 3811 controls the detection information display unit 6111 to display "fog detected" based on the information indicating that fog is detected. Also, at this time, the display control unit 3811 controls to display the fog correction button 6140 based on the information that fog is detected. That is, the display control unit 3811 changes the buttons included in the screen according to the object detected from the image. This allows the image processing method to be changed based on the object in the image.

[Modification 20]
Modification 20 of Embodiment 1 will be described.
FIG. 33 is a diagram showing an example of a screen displayed by the display control unit 3811 of Modification 20 of Embodiment 1. As shown in FIG. The viewer 6200 in Modification 20 includes an image processing adjustment area 6290 . The display control unit 3811 accepts adjustment of image quality parameters by user's operation of the image processing adjustment area 6290 . The image processing adjustment area 6290 includes a hue adjustment object 6291, a brightness adjustment object 6292, a contrast adjustment object 6293, a saturation adjustment object 6294, a gamma value adjustment object 6295, a sigma value adjustment object 6296, and a radius adjustment object 6296. An object 6297 and a variation adjustment object 6298 are included. Selection of the hue adjustment object 6291 executes image processing for adjusting the hue of the image. Also, by selecting the brightness adjustment object 6292, image processing for adjusting the brightness of the image is executed. Furthermore, selection of the contrast adjustment object 6293 executes image processing for adjusting the contrast of the image. Also, by selecting the saturation adjustment object 6294, image processing for adjusting the saturation of the image is executed. Further, selection of the gamma value adjustment object 6295 executes image processing for adjusting the gamma value of the image. Also, selection of the adjust sigma value object 6296 performs image processing to adjust the sigma value of the image. Furthermore, when a part of the image is processed, selection of the radius adjustment object 6297 executes image processing for adjusting the boundary line between the area to be image processed and the area not to be image processed. By selecting the variation amount adjustment object 6298, image processing for adjusting the number of images per unit time is executed. That is, the screen includes an adjustment object that adjusts the degree of image processing. In Modified Example 20, each adjustment object is displayed in a slider format, but radio buttons, check boxes, pull-down menus, and the like may be used instead of sliders.
When the display control unit 3811 receives an operation of the operation unit provided for these adjustment objects, the display control unit 3811 adjusts the degree of image processing according to the received operation. For example, in modification 20, when the hue adjustment object 6291 is moved to the left end, the display control unit 3811 requests the image processing unit 3711 to perform image processing to set the hue to 0. Also, when the hue adjustment object 6291 is moved to the center, the display control section 3811 requests the image processing section 3711 to perform image processing to set the hue to 50. FIG. Furthermore, when the hue adjustment object 6291 is moved to the right end, the display control section 3811 requests the image processing section 3711 to perform image processing to set the hue to 100. FIG. The same applies to brightness adjustment object 6292, contrast adjustment object 6293, saturation adjustment object 6294, gamma value adjustment object 6295, sigma value adjustment object 6296, radius adjustment object 6297, and variation amount adjustment object 6298. That is, the display control section 3811 adjusts the degree of image processing according to the operation of the adjustment object. This makes it possible to adjust the degree of image processing such as brightness.
Also, the adjustment object can be configured to adjust while the image is being recorded or played back. At this time, the storage control unit 3813 performs control so that the timing of operation of the adjustment object and the timing of recording the image or the timing of reproducing the image are stored in association with each other. This makes it possible to know at what timing the degree of image processing was adjusted.
Furthermore, the storage control unit 3813 stores the relationship between the adjustment result and the information of the original video in the storage unit 3820 . Furthermore, the storage control unit 3813 causes the learning model to perform machine learning on the tendency of the user's image processing based on the relationship between the stored adjustment result and the information of the original video, and stores the result in the storage unit 3820. do. The display control unit 3811 adjusts image processing based on the results of this machine learning. That is, the display control unit 3811 adjusts the degree of image processing based on learning information about the degree of adjustment of past image processing. As a result, it is possible to learn the adjustment result of the degree of image processing, and to perform subsequent image processing more appropriately. Also, the display control unit 3811 may perform image processing on a part of the image.
Further, the display control unit 3811 may be configured to instruct processing in the direction of refraining from further correcting the image quality parameter corrected by the drive recorder in the viewer. On the other hand, the display control unit 3811 may be configured to instruct the viewer to perform more correction for image quality parameters that have not been corrected or cannot be corrected by the drive recorder.

[Modification 21]
Modification 21 of Embodiment 1 will be described.
FIG. 34 is a diagram showing an example of a screen displayed by the display control unit 3811 of Modification 21 of Embodiment 1. As shown in FIG. The viewer 6300 in Modification 21 includes an image processing adjustment area 6390 . The image processing adjustment area 6390 includes an image fixing instruction portion 6391 , a sharpness fixing instruction portion 6392 , a banding noise removal fixing instruction portion 6393 , and a film graining fixing instruction portion 6394 . That is, the screen includes a fixed instruction section that prohibits adjustment by the adjustment object. A fixed instruction part is an example of a fixed instruction object.
The display control unit 3811 prohibits changing the operation of the adjustment object when a fixation instruction is given to the fixation instruction unit that fixes the adjustment object. In other words, the display control section 3811 can fix the adjustment of the degree of image processing by operating these fixing instruction sections. For example, in Modified Example 21, by checking the check box of the image fixation instruction section 6391, the display control section 3811 controls the "hue", "brightness", "contrast", and "saturation" in the image adjustment area. For "degrees" and "gamma", the degree of adjustment is fixed. At this time, the display control unit 3811 displays the image quality adjustment result of the fixed adjustment object, that is, the display control unit 3811 prohibits adjustment by the adjustment object according to the operation of the fixed instruction object. This makes it possible to fix the degree of image processing. At this time, the display control unit 3811 displays the “hue”, “brightness”, “contrast”, “saturation”, and “gamma” so that the adjustments are fixed. For example, the display control unit 3811 displays the operation part of the adjustment object in a light color so that it cannot be operated, or hides the operation part of the adjustment object when the check box for the operation of the fixed instruction part is checked. , make it inoperable.

[Modification 22]
Modification 22 of Embodiment 1 will be described.
FIG. 35 is a diagram showing an example of a screen displayed by the display control unit 3811 of Modification 22 of Embodiment 1. As shown in FIG. A file button 6401 is expanded on the viewer 6400 in the modified example 22, so that "reference file", "overwrite save", "save as", "write process result", and "write position information" are displayed. A state in which options for image output of "delete and export" and "close" are displayed is displayed. That is, the screen includes an image output object.
When the file button 6401 is selected, the display control unit 3811 performs "reference file", "overwrite save", "save as", "write process result", and "delete position information". Controls to display a menu related to file processing such as "Export" and "Close". By selecting a process for any file, that process is performed. Here, when “write after deleting position information” is selected, the control unit 3810 deletes the position information linked to the image information and outputs the image information. In other words, the control unit 3810 deletes the position information of the image and outputs the image according to the operation of the image output object. As a result, it is possible to perform an operation to delete the position information associated with the image. Here, when “write after deleting position information” is selected, control unit 3810 may write out the video for posting with the position information deleted. Here, the video for posting is a video converted into a video posting format. At this time, the control unit 3810 may automatically set the title, thumbnail, etc. of the video for posting. In addition, when “write processing result” is selected, the control unit 3810 outputs the image information without deleting the position information associated with the image information. Here, when “write processing result” is selected, control unit 3810 may write out a video for posting without deleting the position information. Furthermore, when outputting image information, the control unit 3810 may be configured to output only an image before image processing, may output an image after image processing, or may output an image before image processing. and an image after image processing may be output at the same time. Further, when using the writing function for posting, the control unit 3810 may be configured to have a function of storing the original video and the processed video in one file and uploading the file. At this time, the video to be written out may be a stream in which both videos are arranged side by side, or may be a video stream in which both videos are arranged chronologically and connected. Also, at this time, the control unit 3810 may embed and write out a display indicating that each video is the “original video” and the “processed video”. Furthermore, the control unit 3810 may be configured to have a function of selecting a locational range for writing a video by using a rectangle or the like when writing the video. At this time, it is preferable to correct the image quality only within the selected range and then re-encode and write out the data.

[Modification 23]
Modification 23 of Embodiment 1 will be described.
FIG. 36 is a diagram showing an example of a screen displayed by the display control unit 3811 of Modification 23 of Embodiment 1. As shown in FIG. In the image display area 6510 of the viewer 6500 in the modification 23, the image captured by the image capturing device 3100 of the vehicle 3400 running behind the other vehicle 6511 is displayed. Video display area 6510 includes license plate information 6512 that can identify other vehicle 6511 by performing HDR conversion processing based on operation of HDR button 6541 .
The display control unit 3811 controls to display a personal information processing button 6540 when the video display area 6510 contains information that can identify an individual, such as the license plate information 6512 . The display control unit 3811 performs control to mask and display the license plate information 6512 based on the operation of the personal information processing button 6540 . In other words, when the processed image contains information that can identify an individual, the display control unit 3811 requests execution of mask processing for the information that can identify an individual. As a result, it is possible to appropriately perform image processing so that information that can identify an individual is not included. Here, the information that can identify an individual is information including name, date of birth, face, license plate, information linked to a specific individual, and the like.

[Modification 24]
In the above-described embodiment, for example, as shown in FIG. 12, an example in which buttons corresponding to each of a plurality of image processes are displayed on the screen of the information display terminal 3800 has been described. However, the information display terminal 3800 may display a smaller number of buttons than each of the plurality of image processes. Then, the display control unit 3811 may request image processing to the image processing unit 3711 or the like according to the set operation corresponding to the button. More specifically, the display control unit 3811 displays, for example, one button on the screen, and when the button is clicked, requests the image processing unit 3711 or the like to perform the first image processing. When the button is double-clicked, the image processing unit 3711 or the like is requested to perform the second image processing, and when the button is triple-clicked, the image processing unit 3711 or the like is requested to perform the third image processing. good too.
According to Modification 24, it is possible to provide a screen on which the user can instruct desired image processing with a small number of buttons.

[Embodiment 2]
In the first embodiment, the server 3700 performs image processing and the information display terminal 3800 displays the image.
In the modified example, a case where the information processing terminal 6800 performs image processing and displays the image will be described with reference to FIGS. 37 and 38. FIG.

[System configuration of information processing system 6600]
FIG. 37 is a schematic diagram showing the system configuration of an information processing system 6600 according to the modification of the second embodiment. As shown in FIG. 37, the information processing system 6600 includes a vehicle 3400, a plurality of information processing terminals 6800D, 6800E, 6800F, and a network 3900. FIG. Vehicle 3400 is configured to be able to communicate with a plurality of information processing terminals 6800D, 6800E, 6800F via network 3900 . More specifically, vehicle 3400 transmits information possessed by vehicle 3400 to a plurality of information processing terminals 6800D, 6800E, and 6800F via network 3900. FIG. A plurality of information processing terminals 6800D, 6800E, 6800F are configured to be able to communicate with vehicle 3400 via network 3900 . More specifically, the plurality of information processing terminals 6800D, 6800E, 6800F acquire information of the plurality of information processing terminals 6800D, 6800E, 6800F via the network 3900. FIG.

[Functional Configuration of Information Processing Terminal 6800]
FIG. 38 is a schematic diagram showing the functional configuration of an information processing terminal 6800 according to the modification of the second embodiment. The hardware configuration of the information processing terminal 6800 is the same as that of the information display terminal 3800 of the first embodiment. That is, the control unit of information processing terminal 6800 executes processing based on the program stored in the storage unit of information processing terminal 6800, thereby realizing the functional configuration of information processing terminal 6800 shown in FIG. .
As shown in FIG. 38, the information processing terminal 6800 includes a display control unit 6811, a communication control unit 6812, a storage control unit 6813, and an image processing unit 6814 as functional configurations.

A display control unit 6811 controls information to be output to an output unit (not shown) of the information processing terminal 6800 . More specifically, the display control unit 6811 controls to display a screen including information about an image.

A communication control unit 6812 controls exchange of information with the server 3700 via the network 3900 .

The storage control unit 6813 stores information in the storage unit of the information processing terminal 6800 and acquires the stored information.

The image processing unit 6814 executes image processing on the images in the storage unit of the information processing terminal 6800 based on the operation instruction via the screen. Accordingly, the information processing terminal 6800 can execute image processing. In the second embodiment, the image processing unit 6814 performs various image processing based on requests through the screen of the information display terminal 3800 and the like. However, the information processing terminal 6800 may have an image processing unit for each image processing. Further, instead of software of the information processing terminal 6800 performing image processing, dedicated hardware (for example, GPU, etc.) of the information processing terminal 6800 may perform image processing. When dedicated hardware performs image processing, one hardware of the information processing terminal 6800 may be configured to perform a plurality of image processes, or a plurality of information processing terminals 6800 corresponding to each of the plurality of image processes may be configured. may be configured to perform image processing respectively.

According to the second embodiment, image processing can be performed on the information processing terminal side.

[Embodiment 3]
In the third embodiment, an example in which image processing is performed within the imaging device 3100 of the vehicle 3400 will be described.

The control unit 3011 records the image captured by the imaging unit 3015 in the storage medium 3500 . The control unit 3011 causes the processor 3111 to perform image processing on the recorded image. At this time, the control unit 3011 records in the storage medium 3500 what kind of image processing was performed and at what timing. At this time, the control section 3011 can transmit the information recorded in the storage medium 3500 via the network 3900 .

The control unit 3011 performs control to write out an image that has undergone image processing by performing a predetermined operation during playback in the imaging device 3100 . The control unit 3011 gives a predetermined notification while the image processed in the imaging device 3100 is being written. More specifically, the control unit 3011 controls the display unit 3013 to display a message "Please do not turn off the engine" while the image processed in the imaging device 3100 is being written, Control is performed such that a voice saying “Please do not turn off the engine” is output from the voice output unit 3014 . These notifications can be continuously (repeatedly) performed.

Further, in the third embodiment, the control unit 3011 may have an image processing function capable of removing fog. At this time, the control unit 3011 removes the fog from the target image by the user's operation. Next, the control unit 3011 records the image after performing the fog removal process by the user's operation. Furthermore, the control unit 3011 may be configured to reproduce and reproduce the image before performing the fog removal process. This reproduction may be executed by the control unit 3011 recording an image before the fog removal process in association with data after the fog removal process, or by recording the image before the fog removal process. may be recorded in association with the image after defogging, and may be executed using this information. For example, if there is a large difference between what the driver saw and the corrected image, the image will be different from what the driver originally saw, which would be a disadvantage for the driver if he or she were to cause an accident. However, it is possible to grasp the degree of fog removal by the fog removal process.

The control unit 3011 may also have a function of performing recording according to the user's selection of whether to perform recording with or without performing the fog removal process. Furthermore, the control section 3011 may have a function of displaying these information on an information display terminal. The control unit 3011 may have a function of recording information that can be determined on the viewer of the information display terminal as to whether the recording was performed with the fog removal processing performed or the recording was performed without performing the processing. . Information may be transferred from the drive recorder to the information display terminal at this time via the network 3900 or using an information recording medium such as an SD card.

According to the third embodiment, image processing can be performed on the vehicle side.

[Embodiment 4]
Next, Embodiment 4 will be described. Embodiment 4 may be implemented by the information processing system 3000 shown in FIG. FIG. 39 is a functional block diagram realized by the storage section 22 and the control section 23. As shown in FIG. The controller 23 shown in FIG. 39 may be, for example, the controller 3810 provided in the information display terminals 3800A, 3800B, 3800C, and the like. The storage unit 22 shown in FIG. 39 may be, for example, the storage unit 3820 provided in the information display terminals 3800A, 3800B, 3800C, and the like.

The storage unit 22 stores various information defined by the above description. For example, it can be used as a storage device such as a solid state drive (SSD) for storing various programs related to the information display terminal 3800 executed by the control unit 23, or as a temporary storage device related to program calculation. It can be implemented as a memory such as a Random Access Memory (RAM) that stores information (arguments, arrays, etc.) required for the . A combination of these may also be used. In particular, the storage unit 32 may store, as various programs related to the information display terminal 3800 executed by the control unit 33, for example, a first program P1 and a second program P2 that serve as viewers of the image data IM. .

The control unit 23 may function as an acquisition unit 231 , a reception unit 232 , a correction unit 233 , a cancellation unit 234 and a display control unit 235 by executing various programs stored in the storage unit 22 . That is, information processing by software stored in the storage unit 22 may be specifically realized by the control unit 23 which is an example of hardware, and may be executed as each function unit included in the control unit 23 . Each functional unit related to the control unit 23 will be described below.

The acquisition unit 231 may be configured to acquire various information. For example, the acquisition unit 231 may acquire specific information IF1 regarding the state or situation of the moving body as the acquisition step. This will be explained in more detail later.

The reception unit 232 may be configured to receive various types of information. For example, the reception unit 232 may receive a first input from the user as the first reception step. The accepting unit 232 may accept a second input from the user as a second accepting step. This will be explained in more detail later.

The correction unit 233 may be configured to perform correction processing. For example, as a first correction step, the correction unit 233 performs the first correction to the image data IM generated by the imaging device 3100 of the moving body at the time corresponding to the specific information IF1 based on the specific information IF1. processing may be performed. Note that the first correction processing may include automatic correction processing by a program such as artificial intelligence, for example. As a second correction step, the correction unit 233 may perform a second correction process on the image data IM based on the second input. Note that the second correction process may include any manual correction process based on user input, for example. Furthermore, the correction unit 233 may perform a preparation step and a processing step. This will be explained in more detail later.

The cancellation unit 234 may be configured to cancel the correction processing by the correction unit 233 . For example, as the canceling step, the canceling unit 234 may cancel the first correction process on the image data IM based on the first input. This will be explained in more detail later.

The display control unit 235 is configured to control various visual information to be displayed on a display capable of displaying. Here, the visual information may be information itself generated in a form that can be visually recognized by the user, such as screens, images, icons, texts, etc., or rendering information for displaying screens, images, icons, texts, etc. There may be. For example, as the display control step, the display control unit 235 may reproducibly display the image data IM on which the first correction process has been performed. This will be explained in more detail later.

[Outline of information processing]
Next, an information processing method of the information processing system 3000 according to the fourth embodiment will be described. The information processing method includes steps of an information display terminal 3800, which is an example of an information processing device. FIG. 40 is an activity diagram showing the flow of information processing according to the fourth embodiment. This information processing will be outlined below along this activity diagram.

First, when the moving object is running, the photographing device 3100 may record the state of the running as image data IM (activity A101). The image data IM may be, for example, moving image format data having a predetermined frame rate. The frame rate of the imaging device 3100 is not particularly limited, but is, for example, 10 fps or higher, preferably 30 fps or higher, more preferably 60 fps or higher. , 60, 70, 80, 90, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 210, 220, 230, 240, 250 fps, and any of the numerical values exemplified here or within a range between the two. The generated image data IM may be displayed on the display unit 3013 of the imaging device 3100 as a live view. The recording of the image data IM may be continuously performed until the moving body finishes running.

Subsequently, the information display terminal 3800 connected to the imaging device 3100 via the network may receive a series of image data IM generated by continuous recording in activity A101 (activity A102). That is, the reception unit 332 may receive the image data IM from the imaging device 3100 . Of course, instead of the network, the receiving unit 332 may receive the image data IM via a storage medium such as an SD card (registered trademark). In the following description, for the sake of simplification, an example of communication via a network will be used, but it is not necessarily limited to this.

Subsequently, the acquisition unit 231 may acquire specific information IF1 regarding the state or situation of the mobile object as an acquisition step (activity A103). Specifically, the control unit 23 may read the dedicated program stored in the storage unit 22 to acquire the specific information IF1 and store it in the storage unit 22 . The specific information IF1 may be included in the image data IM received by the reception unit 332, or may be received as separate information from the imaging device 3100 or an external device. Preferably, the specific information IF1 may be at least one of the position, orientation, speed, acceleration, time, time period, weather, running condition, and pedal work of the moving body. For example, an appropriate type of correction may be automatically applied to the tunnel position, or different image corrections may be applied for sunny, rainy and foggy conditions. Also, different image corrections may be applied in the morning, at night, and in the evening. good. Optimal parameters for each may be prepared in advance. More specifically, it is implemented as a button to correct the image according to the purpose, such as an operation button to erase the snowstorm, a button to make the fog transparent so that you can see the other side of the fog, a button to erase the rain, etc. good too. Furthermore, as an almighty automatic correction, automatic correction may be made by appropriately combining these.

According to this aspect, it is possible to perform more appropriate image processing in consideration of at least one of the specific states of the moving body, namely, position, speed, acceleration, traveling state, and pedal work. Therefore, usability can be further improved.

Subsequently, as a first correction step, based on the specific information IF1, the correction unit 233 converts the image data IM generated by the imaging device 3100 of the moving object at the time corresponding to the specific information IF1 into the first image data IM. Correction processing may be performed (activity A104). Specifically, the control unit 23 may read the dedicated program stored in the storage unit 22 to determine the correction range for the image data IM based on the specific information IF1. That is, preferably, as the first correction step, the correction unit 233 performs the first correction process on the image data IM while changing the correction range, which is the range to be corrected, based on the specific information IF1. good. According to such an aspect, since it is not necessary to correct the entire range, the processing load can be suppressed. In addition, since the preferable range is corrected, a function with high usability can be realized. Conversely, the correction unit 233 may perform the first correction process on the image data IM while changing the range that should not be corrected based on the specific information IF1. Furthermore, the first correction process may be executed as appropriate while both the range to be corrected and the range not to be corrected are variable.

After that, as a display control step, the display control unit 235 may reproducibly display the image data IM on which the first correction processing has been performed (activity A105). Such a display is preferably capable of being reproduced as moving image data by displaying the frames of the image data IM frame by frame.

During display control by the display control unit 235, the reception unit 232 may receive a first input from the user as a first reception step. Control unit 33 may be configured to be able to switch between the presence or absence of correction processing for image data IM based on the first input (activity A106). For example, if the image data IM has undergone correction processing, the cancellation unit 234 may cancel the first correction processing on the image data IM based on the first input as a cancellation step. . For example, when the correction process for the image data IM has been canceled, the correction unit 233 may restart the first correction process for the image data IM based on the first input. According to this aspect, when the correction processing presented by the device is unnecessary, the visibility can be improved by canceling the correction processing at the user's will, and the usability can be further improved. For example, this can be done by providing a button for temporarily canceling the correction while the range remains corrected.

To summarize the above, the information display terminal 3800, which is an example of the information processing apparatus according to this embodiment, includes the control unit 23. FIG. The control unit 23 includes the following units. As a first correction step, based on the specific information IF1, the correction unit 233 performs a first correction process on the image data IM generated by the imaging device 3100 of the moving object at the time corresponding to the specific information IF1. Execute. As a display control step, the display control unit 235 reproducibly displays the image data IM on which the first correction processing has been performed. According to this aspect, it is possible to perform appropriate image processing based on the state or situation of the moving object, thereby improving usability.

[Details of information processing]
Next, the details of the information processing outlined above will be described. FIG. 41 is a schematic diagram showing an example of a spatial correction range of the first correction process. FIG. 42 is a schematic diagram showing an example of the temporal correction range of the first correction process. That is, the first correction process may be applied with a limited location range or time range.

In the mode of FIG. 41, the image data IM is displayed by the display control unit 235, and this is shown as the playback screen 4. FIG. The correction range is the region of interest R set in the frame of the image data IM. The playback screen 4 has a correction area 41 and a non-correction area 42 . Region of interest R is illustrated as correction region 41 . Specifically, the correction area 41 may be determined according to various information included in the specific information IF1.

For example, the size of the correction area 41 may be changed according to the speed or acceleration of the moving object. For example, when the moving body turns left, the correction process may be performed so that the left side of the moving body is visualized in order to prevent a collision accident. The shape and size of the correction area 41 are absolutely arbitrary, and may be rectangular, circular, or elliptical, for example. The shape may be changeable based on user input. According to such an aspect, it is possible to narrow down the area in which the visibility is originally desired to be improved and to apply the correction, thereby improving the usability.

Further, for example, appropriate correction processing may be performed so that the position of the sun can be grasped from the time of day or the time zone so that the user can easily recognize the position even in backlight conditions. In the mode of FIG. 42, a correction time 51 is shown in the seek bar 5 positioned below the playback screen 4 and indicating the playback time of the image data IM. The seek bar 5 may correspond to a concept such as a timeline of video editing software. For example, the correction time 51 is a time zone in which the sun shines in the afternoon, and a time zone in which the photographing device 3100 shoots against the light depending on the traveling direction of the moving body. In other words, the specific information IF1 is the position and orientation of the mobile object. As a first correction step, the correction unit 233 may perform a first correction process on the image data IM based on the position of the sun estimated from the position and orientation. According to this aspect, it is possible to perform appropriate image processing in consideration of the position of the sun, thereby further improving usability.

Also, the various correction processes described above may be performed as arbitrary manual corrections by the user, which is an example of the second correction process. In other words, the reception unit 232 receives a second input from the user as a second reception step, and the correction unit 233 performs a second input to the image data IM based on the second input as a second correction step. 2 correction process may be executed. According to this aspect, in addition to the correction processing presented by the device, it is possible to improve the visibility by adding the correction according to the user's intention, and it is possible to further improve the usability.

In addition, in relation to the first correction process and the second correction process, the display control unit 235 enables the image data IM to be reproduced in a manner in which the first correction process and the second correction process can be distinguished. should be displayed. In other words, it is preferable that the manually added range and the automatically added range are displayed so as to be distinguishable. FIG. 43 is a schematic diagram showing additional information 43 attached to the reproduced screen 4 when the first correction process and the second correction process are executed. As shown in FIG. 43 , additional information 43 such as “under automatic correction” or “under manual correction” may be added to the upper right of the playback screen 4 . Although the additional information 43 is a text annotation here, an illustration or an icon may be used instead. According to this aspect, the user can grasp the type of correction processing, which is user-friendly.

The method of correction processing is not particularly limited, and at least one of brightness, dynamic range, contrast, gamma, saturation, hue, etc. may be corrected as appropriate. Moreover, it is preferable that the correction is performed based on the ease of visual recognition by human eyes.

Conversely, image data IM that is difficult for the user to visually recognize may be realized. For example, from the viewpoint of personal information protection, when an unspecified number of people are captured, a function of applying a mosaic to the faces of such people may be realized. Mosaic may be applied at least one of when recording, when recognizing a man, and when recognizing a woman. Also, a function of applying a mosaic to license plates of other cars may be realized. The target to be mosaiced is preferably determined by artificial intelligence or other methods. Such mosaic processing is an example of correction processing for making an image dirty after making it clean. Mosaic is an example of "not visible" and may be implemented in other ways.

In addition, unlike correction by an ISP (Imaging Signal Processor) that removes aberrations, noise, etc. based on optical elements and optical systems, this correction processing is a correction processing that is further performed on the image data IM obtained through the ISP. be. Specifically, gamma correction, color enhancement, color correction such as color balance, edge enhancement, and the like may be included. Conversely, the original image data IM refers to the image data IM obtained through correction by the ISP. Such correction processing may be performed after correction by the ISP and before encoding into an appropriate file format such as H264. may be implemented.

[Modification 1]
Next, Modification 1 of Embodiment 4 will be described. In this modified example 1, in addition to the specific information IF1, the acquisition unit 231 may acquire various information other than the specific information IF1 from an external device. In Modification 1, for example, the specific information IF1 may be the position of the mobile object. The acquisition unit 231 acquires the road network information IF2 as an acquisition step, and the correction unit 233 performs a first correction to the image data IM based on the position and the road network information IF2 as a first correction step. processing may be performed. Specifically, the control unit 23 reads out the dedicated program stored in the storage unit 22 to acquire the road network information IF2, and furthermore, the first correction process is executed using this.

The road network information IF2 includes, for example, the speed limit, road width, 12-hour traffic volume on weekdays, and travel speed (for a certain section, divide the distance of the section by the time required to travel. It has various information such as speed (including stoppage due to traffic jam, etc.) and number of lanes as attribute data. Based on this information, the correction unit 233 may determine an appropriate correction type or correction range and execute the first correction process. According to this aspect, it is possible to perform appropriate image processing according to the position and the road conditions at that time, so that usability can be further improved.

[Modification 2]
Next, Modification 2 of Embodiment 4 will be described. In this modified example 2, as in modified example 1, the acquisition unit 231 may acquire various information other than the specific information IF1 from an external device in addition to the specific information IF1. In Modified Example 2, for example, the specific information IF1 may be the position of the mobile object. The acquisition unit 231 acquires the terrain information IF3, and the correction unit 233 performs the first correction process on the image data IM based on the position and the terrain information IF3 as a first correction step. good. Specifically, the control unit 23 reads out the dedicated program stored in the storage unit 22 to acquire the terrain information IF3, and furthermore, the first correction process is executed using this.

The topography information IF3 may be acquired from an external server related to an organization that issues map data, for example. The terrain information IF3 may be information that includes information about terrain such as urban areas, suburbs, highways, general roads, gravel roads, mountains, and inside tunnels. Based on this information, the correction unit 233 may determine an appropriate correction type or correction range and execute the first correction process. For example, in a downtown area, the influence of neon light is strong, so correction processing may be performed to mitigate this. For example, if it is in a forest, a correction process that emphasizes the brightness as a whole may be performed. According to this aspect, it is possible to perform appropriate image processing according to the position and the topography, environment, etc. at that time, so that usability can be further improved.

[Modification 3]
Next, Modification 3 of Embodiment 4 will be described. In this modified example 3, the correction unit 233 may perform a preparation step and a processing step. Further, the control unit 23 may be configured to be able to call a first program P1 stored in advance and a second program P2 different from the first program P1. In particular, the first program P1 may be a player application recommended for reproducing the image data IM, and the second program P2 may be a general-purpose player application having correction processing functions. In other words, if too many functions are incorporated into the first program P1, which is a standard viewer, it will become bloated. be able to. 44 is an activity diagram illustrating the flow of information processing according to Modification 3 of Embodiment 4. FIG. This information processing will be outlined below along this activity diagram.

First, when the reception unit 232 of the information display terminal 3800 receives an input for reproducing a moving image from the user, the control unit 23 may call the first program P1 stored in the storage unit 22 (activity A201). Subsequently, the image data IM may be read into the first program P1 (activity A202). Subsequently, as an operation based on the first program P1, the acquisition unit 231 may extract and acquire a plurality of parameters PT included in the image data IM (activity A203).

If the second program P2 is not pre-installed in the information display terminal 3800, the installation of the second program P2 may be suggested. In other words, if the second program P2 is not installed, the display control unit 235 may display a screen recommending its installation. The recommended screen preferably includes the name of the second program P2, and more preferably includes the URL of its installation destination. According to this aspect, even if the second program is not installed, the first correction process can be performed via the second program.

Specifically, when installing the first program P1, which is a PC viewer, it is possible to ask whether or not to install an external player, and to install the external player if there is an instruction to install it. At the time of installation or immediately after starting, there may be guidance for a button to start the external player. A caution when using an external player may be displayed. When using an external player, it is important to note that the display state of the first program P1, which is the PC viewer, may deviate from the display state of the first program P1, which is the PC viewer, when the buttons of the external player are operated. It may be included that P2 may be closed and the second program P2, which is an external player, may be started again from the first program P1, which is a PC viewer.

If the second program P2 is pre-installed in the information display terminal 3800, the control unit 23 may call the second program P2 stored in the storage unit 22 (activity A205). in short. An external player activation button or the like may be provided on the UI of the first program P1, which is the viewer, and by pressing this button, the second program P2 such as VLC may be activated. As an operation based on the second program P2, the acquisition unit 231 may acquire a plurality of parameters PT included in the image data IM (activity A206). Here, the parameter PT may include at least one of hue, brightness, saturation, contrast, gamma, sharpness, playback start time, playback end time, and presence/absence of loop. According to this aspect, since each parameter is a general one possessed by image data, a more versatile image processing program can be used as the second program, and an apparatus with high usability can be provided. can be realized. Furthermore, as an operation based on the second program P2, the acquisition unit 231 may acquire operation information IF4 regarding the operation of the second program P2 (activity A207).

Specifically, libvlc. dll or the like into the viewer, the file name of the moving image, the video effect settings, etc. may be passed as "command line arguments" and started.

vlc. exe 20200101090000.exe avi --hue=0 --brightness=1.5 --contrast=0.6 --saturation=1.9 --gamma=0.3 --sharpen-sigma=0.9

Also, the playback start time, playback end time, presence/absence of looping, etc. may be transferred. A parameter may be passed that disables at least part of the controls of the second program P2, which is an external player. In addition to this, a control command may be sent from the first program P1, which is the viewer, to control the second program P2, which is the external player. For example, if the second program P2 is a VLC player, it may be controlled from the side of the first program P1, which is a viewer, using http, telnet, or the like.

In other words, the first program P1 may cause preparatory steps to be performed. Then, as a preparatory step, the correction unit 233 may extract the parameter PT included in the image data IM and pass the value of the extracted parameter PT to the second program P2. In addition, as a preparatory step, the correction unit 233 may pass operation information IF4 regarding the operation of the second program P2 to the second program P2.

Subsequently, as a processing step, the correction unit 233 may change the value of the parameter PT by executing a first correction process on the parameter PT (activity A208). Then, the display control section 235 may reproducibly display the image data IM based on the value of the parameter PT changed in the processing section. According to this aspect, a separate program that is more specialized for correction processing can be used, which contributes to reduction in calculation cost and improvement in image processing performance.

Further, preferably, the display control unit 235 may restrict at least part of the functions of the second program P2 based on the operation information IF4 to display the image data IM reproducibly (activity A209 ). According to this aspect, the second program can be used only for correction and reproduction, and the user does not have to use unnecessary functions of the second program, and the device has high usability. can be realized.

[Modification 4]
Next, Modification 4 of Embodiment 4 will be described. By the way, in the third modified example described above, when the second program P2 is not installed, a screen recommending the installation of the second program P2 is displayed. In this modification 4, the second program P2 can be used via the network without installing the second program P2 in the information display terminal 3800. FIG. In other words, the second program P2 may be a program stored in an external device. Then, the control unit 23 may be configured to call the second program P2 via the network. According to this aspect, even if the second program is not installed, the first correction process can be performed via the second program in the form of a cloud service.

Specifically, in the first program P1, there is a function of storing an ID for automatic login to a cloud image sharpening service, a function of automatic sending, a function of uploading, and checking whether conversion is completed. It is preferable to have a function of automatically downloading the image after it is determined and the conversion is completed, a function of notifying that the download is possible, a function of synchronizing the downloaded image with the image before conversion, and reproducing the image.

[Modification 5]
Next, Modification 5 of Embodiment 4 will be described. There are various types of correction processes described above, and different correction processes may be superimposed. Different correction processes may be applied to one reproduced screen 4 at the same time, or different correction processes may be applied to different times. Different correction processes are preferably determined by needs and situations.

In terms of the former, for example, when the photographing device 3100 using a fish-eye lens is used, the deck board of the moving object is often reflected near the center of the image data IM, and this portion is often unnecessary as information. Therefore, the processing load as a whole can be suppressed by not performing correction processing on such a central portion. Further, since the visibility of the information in the uncorrected portion is poor, some additional information 43 may be embedded in this position. For example, original image data IM before correction may be embedded, or a recommended player or the like may be presented.

In terms of the latter, for example, at the moment of a traffic accident, when a person jumps out from behind an object, the focus is placed on the person gradually coming forward from a dark place, and correction processing is performed so that the user can appropriately recognize it. good. Different processing may be performed inside the vehicle and outside the vehicle.

[Modification 6]
Next, Modification 6 of Embodiment 4 will be described. A sharpening process may be performed to sharpen a part of an image captured by an image capturing device that captures images using a fisheye lens (for example, one or both of the image capturing devices 3100 and 3200). Sharpening processing is an example of correction processing for correcting a captured image. In particular, it is preferable to perform a sharpening process for sharpening a part of an image captured by an imaging device using a circular fisheye lens (hereinafter referred to as a "circumferential fisheye image"). In this case, a processing pattern may be adopted in which shape conversion (geometric conversion) is performed after sharpening processing is performed on the circular fisheye image as it is. In particular, the shape conversion may be a process of projecting onto a rectangle or the like, in other words, a process of converting to a plane image. By doing so, it is possible to simplify processing in a configuration in which a plurality of portions are extracted from a circular fisheye image and displayed on a rectangular screen. Alternatively, when displaying an image by extracting a portion from a circular fisheye image, if the image has a function of changing the area to be extracted during playback, it is possible to improve the followability to the area changing operation. In addition, in cases where multiple extraction areas overlap (for example, when multiple rectangular areas overlap, such as the shape of a “turn” or the shape of an “eye”), a function is provided to reduce the sharpening process for overlapping parts. Good. In this way, the processing can be simplified by the amount of reduction in redundant sharpening processing.

In the processing pattern for extracting a part of the circular fisheye image, transforming the shape, and then sharpening, it is preferable not to perform the sharpening process on areas other than the extracted area. By doing so, the processing load can be reduced because the sharpening processing range is limited.

A part of the circular fisheye image captured by the circular fisheye lens may be cut out, shape-converted (for example, projected into a rectangle, etc.), and a sharpening process may be performed to sharpen the part of the shape-converted image. Such processing can be referred to as a double narrowing of the sharpening range.

The sharpening process should not be performed on a low-performance device such as a PC. A device such as a PC with low performance may be switched automatically or manually by the user. Low performance means that the processing capability of the device is below a predetermined specification, and in particular, sharpening processing should not be performed on a device such as a PC that does not have a GPU.

A function of writing out the image after the sharpening process and a function of uploading a writing function for posting to SNS such as YouTube (registered trademark) may be provided.

[Modification 7]
Next, Modification 7 of Embodiment 4 will be described. In the above-described embodiment, the recorded image data IM is described as a moving body that is running. In such a case, the parking monitoring function of the imaging device 3100 may perform correction processing different from that during running.

[Embodiment 5]
Next, Embodiment 5 will be described. Embodiment 5 may be implemented by the information processing system 3000 shown in FIG. FIG. 45 is a functional block diagram realized by the control unit 33. As shown in FIG. The control unit 33 shown in FIG. 45 may be, for example, the control unit 3011 provided in the photographing device 3100 mounted on the vehicle 3400 . The storage unit 32 shown in FIG. 45 may be, for example, a memory 3112 provided in the imaging device 3100 or the like.

The storage unit 32 stores various information defined by the above description. This may be, for example, a storage device such as a solid state drive (SSD) for storing various programs related to the imaging device 3100 executed by the control unit 33, or a temporary storage device related to program calculation. It can be implemented as a memory such as a Random Access Memory (RAM) that stores the necessary information (arguments, arrays, etc.). A combination of these may also be used. In particular, the storage unit 32 may store a third program P3 and a fourth program P4 as various programs related to the imaging device 3100 executed by the control unit 33 .

The control unit 33 may function as an acquisition unit 331 , a reception unit 332 , a recording unit 333 , a correction unit 334 and a display control unit 335 by executing various programs stored in the storage unit 32 . That is, information processing by software stored in the storage unit 32 may be specifically realized by the control unit 33 as an example of hardware, and may be executed as each function unit included in the control unit 33 . Each functional unit related to the control unit 33 will be described below.

The acquisition unit 331 is configured to acquire various information.

The reception unit 332 is configured to receive various information.

The recording unit 333 is configured to record data on a storage medium. For example, the recording unit 333 records the image captured by the camera as image data IM. This will be explained in more detail later.

The correction unit 334 is configured to perform correction processing.

The display control unit 335 is configured to control various visual information to be displayed on a display capable of displaying. Here, the visual information may be information itself generated in a form that can be visually recognized by the user, such as screens, images, icons, texts, etc., or rendering information for displaying screens, images, icons, texts, etc. There may be. For example, the display control unit 335 may display the image captured by the camera as a live view. This will be explained in more detail later.

Next, an information processing method of the information processing system 3000 according to the fifth embodiment will be described. In Embodiment 5, the control unit 33 may be configured to be able to call a third program P3 stored in advance and a fourth program P4 different from the third program P3. In particular, the third program P3 may be a player application recommended for reproducing the image data IM, and the fourth program P4 may be a general-purpose player application. FIG. 46 is an activity diagram showing the flow of information processing according to the fifth embodiment. This information processing will be outlined below along this activity diagram.

While the moving body is running, the following activities A301 to A305 are sequentially repeated. First, while the moving body is running, the recording unit 333 may record the video imaged by the imaging unit 3015 of the imaging device 3100 as the image data IM (activity A301). Specifically, the image data IM may be sequentially stored in the storage unit 32 by the control unit 33 reading out the dedicated program stored in the storage unit 32 .

The image data IM may be, for example, moving image format data having a predetermined frame rate. The frame rate of the imaging device 3100 is not particularly limited, but is, for example, 10 fps or higher, preferably 30 fps or higher, more preferably 60 fps or higher. , 60, 70, 80, 90, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 210, 220, 230, 240, 250 fps, and any of the numerical values exemplified here or within a range between the two.

Subsequently, the processing of activities A302 and A303 for displaying the image data IM on the display unit 3013 of the photographing device 3100, and the processing of activities A304 and A305 for later reproducing the image data IM on the information display terminal 3800. are executed in parallel.

Specifically, the image data IM is recorded, and each frame is subjected to the first correction process (activity A302) and displayed in live view (activity A303). In other words, the display control unit 335 may display the image captured by the camera as a live view. Here, the values of the parameters PT for the recorded image data IM and the displayed video may be set differently.

FIG. 47 is a schematic diagram showing aspects of image data IM displayed as a live view and image data IM recorded so as to be reproducible later. As shown in FIG. 47, when the recording unit 333 records the image data IM, the recording unit 333 records the image as the image data IM without correcting the image, while the display control unit 335 corrects the image. to display it as a live view. According to this aspect, it is possible to prevent information loss due to processing of image data to be recorded while enhancing the visibility of the video viewed as a live view.

On the other hand, when the image data IM displayed as a live view is recorded, the acquisition unit 331 may extract and acquire various parameters PT included in the image data IM (activity A304). Then, the recording unit 333 may record the parameter PT of the displayed video as a parameter log PL along with the image data IM in chronological order (activity A305). According to such an aspect, it is possible to preserve the visibility of the video viewed as a live view.

After the moving object has completed running, the image data IM may be reproduced via the information display terminal 3800 connected to the imaging device 3100 via the network. When the image data IM is reproduced on the information display terminal 3800, the reception unit 332 of the information display terminal 3800 receives a third input for reproducing the image data IM from the user as a third reception step (activity A306 ). Subsequently, with the third input as a trigger, the control unit 23 may read the image data IM (activity A307).

Subsequently, when the program of the third program P3 is installed in the information display terminal 3800, the control unit 23 may read the parameter log PL as an operation based on the third program P3 (activity A308). Subsequently, as an operation based on the third program P3, the correction unit 233 refers to the value of the parameter log PL, and based on this, the value of each parameter of the original image data IM is appropriately changed for the first correction processing. may be performed (activity A309). After that, as an operation based on the third program P3, the corrected image data IM may be displayed by the display control unit 235 (activity A311).

In other words, the display control unit 335 may read the image data IM and the parameter log PL based on the third input, and display the corrected image data IM reproducibly. According to such an aspect, when the user desires, a video with high visibility can be reproduced on the spot, and a device with high usability can be realized.

On the other hand, when the program of the third program P3 is not installed in the information display terminal 3800, as an operation based on the fourth program P4, the control unit 23 displays the additional information 43 recommending reproduction with the third program P3. may be generated (activity A310). Thereafter, as an operation based on the fourth program P4, the image data IM may be displayed by the display control unit 235 in a form in which the additional information 43 is superimposed on the image data IM as it is recorded (activity A311). In other words, the display control unit 335 preferably displays information prompting reproduction by the third program P3 when the image data IM is reproduced by the fourth program P4. According to this aspect, even if the content is played back by an unintended program, it is possible to guide the playback with a preferred program and provide appropriate information to the user.

FIG. 48 is a schematic diagram showing a mode in which additional information 43 prompting reproduction of the recommended third program P3 is superimposed on the reproduction screen 4. As shown in FIG. As shown in FIG. 48, the information prompting reproduction is preferably displayed together with the reproduction screen of the image data IM. According to this aspect, it is possible to appropriately provide information to the user that the current appearance is unfavorable and that the image data itself is fine.

In addition to the character string, a QR code (registered trademark) that encodes the URL of the third program P3 is embedded and recorded, or a screen image that describes these is inserted at the beginning of the playback image file. You may record.

To summarize the above, the imaging device 3100 as an example of the information processing device according to this embodiment includes the control unit 33 . The control unit 33 includes the following units. The recording unit 333 records the image captured by the camera as image data IM. The display control unit 335 displays the image captured by the camera as a live view. Here, the parameters PT of the recorded image data IM and the displayed video are set differently. According to such an aspect, both visibility and customizability can be ensured by differentiating the video visually recognized as live view and the image data to be recorded.

Furthermore, in other words, various parameters PT related to image quality, such as parameters of the image sensor and ISP of the photographing device 3100 such as a drive recorder, are recorded in association with the video data of the drive recorder, and when reproduced on the viewer and the main body. , it is preferable to perform playback using this parameter.

[Modification 1]
Next, Modification 1 of Embodiment 5 will be described. In this modified example 1, unlike the fifth embodiment, the image data IM to be recorded is image data IM that has undergone correction processing and has high visibility, in substantially the same manner as that used for live view. On the other hand, if it is desired to reproduce the original image data IM before correction later, the information display terminal 3800 or the like can be used to apply correction processing to the image data IM, thereby reproducing the image data IM. may be generated. Such correction processing is preferably exact inverse transform processing, but correction processing presumed to be inverse transform may be applied.

[Modification 2]
Next, Modification 2 of Embodiment 5 will be described. In this modified example 2, the parameters PT are divided into two types and recorded. The parameters PT include those that are constant throughout the entire time series of the image data IM and those that change on a frame-by-frame basis. The former is recorded in units of image data IM, and the latter is recorded in units of frames of image data IM, so that the processing load can be suppressed and correction processing can be performed efficiently.

In other words, regarding the parameter PT, fixed static parameters that do not change during shooting and dynamic parameters that change depending on the video state during shooting are separately recorded. , one static one for the file, and one dynamic one at the change point. Desirably, record over multiple frames, such as all frames.
Each may be read and processed separately.

[others]
A program may be provided that causes a computer to function as a control unit of an information processing device, such as the control unit 33 in the imaging device 3100 or the control unit 23 in the information display terminal 3800 .

Since there is a problem that it is necessary to have a mechanism for switching which of the different algorithms such as fog, HDR, sharpening, etc. is applied to the image currently being reproduced, even if the following aspects are adopted for the correction processing good.
1. The position and direction of the sun are obtained from GPS, and the content of correction applied to the image is changed or varied.
2. Based on GPS location information and map information, it is determined whether the location is in the city or in the woods, and the applied algorithm is changed.
3. If the operation of turning on the fog lights is detected, a fog algorithm is applied.
4. If the difference in contrast is small and the image is white, it is determined to be fog and a fog removal algorithm is applied.
5. When there is a lot of black, it is determined to be dark and brightened.
6. CLAHE, gamma, and contrast are varied as a set.

A function may be provided to write the video recorded by the imaging device 3100 such as a drive recorder to a file different from the video recorded with the video image quality-corrected by the imaging device 3100 .

The drive recorder may have a function of writing out an image corrected in image quality by performing a predetermined operation during playback on the drive recorder main body.

The notification that the engine should not be turned off while the image corrected for image quality is being written may be continuously notified, particularly preferably repeatedly by voice.

The viewer is provided with an operation unit for adjusting image quality parameters that can be operated by the user, such as radio buttons, check boxes, pull-down menus, sliders, etc., and the result of image quality adjustment according to the operation of the operation unit while playing back the recorded video. is displayed, and if a fixation instruction is given to a fixation instruction section that fixes the operation section in the displayed state, for example, a check box that gives a fixation instruction, the operation of the operation section is prohibited from being changed. The image quality adjustment result of the operation unit may be displayed.

The viewer is provided with operation units for adjusting image quality parameters that can be operated by the user, such as radio buttons, check boxes, pull-down menus, and sliders. A recording function may be provided. A function of adjusting the image quality according to the operation of the recorded user operation unit and playing back may be provided.

The imaging device 3100 such as a drive recorder or the information display terminal 3800 in which the first program P1 serving as a viewer is installed has a function of writing video for posting with position information deleted and a writing function for posting without deleting the position information. function may be provided.

It displays the recorded video after correction as multiple streams with different correction states, and has a selection function that allows the user to select the stream to be written or played back from the multiple streams on the display screen. good too. For example, a plurality of candidate video windows may be opened and reproduced in synchronism with the original video window, and the user may select the best one from among them using a selection button or the like.

Image quality adjustment parameters and the like for playback of another video may be determined using the relationship between the user's selection result and selection state and the information of the original video at that time. Here, the relationship may be recorded, or may be recorded after being learned by a learning model.

It is also possible to provide a function of selecting a locational range for writing the video with a rectangle or the like when writing the video, and perform image quality correction only within the selected range before re-encoding and writing.

For example, it is preferable to provide a writing function for posting to SNS such as YouTube, and a function to store and upload the original video and the processed video in one file. A stream in which both images are arranged side by side may be used, or a video stream in which both images are arranged one behind the other in terms of time and connected may be used. At this time, it is preferable to embed an indication indicating that each image is an "original image" and a "processed image" before writing.

In the case of an image captured by an omnidirectional or semispherical camera, it is preferable to process the displayed image by expanding the image and performing coordinate transformation on a part of the image. By doing so, the processing load can be reduced. Alternatively, for example, while the donut-shaped portion including the outermost portion is processed, the hole portion of the donut, that is, the inner circular region, may not be processed or may be processed differently.

Furthermore, the following aspects can also be provided.

In the information processing apparatus, the screen includes an object, and the control unit receives an operation instruction for the object selected by the user, and issues an instruction to display a processing result regarding the image in response to the operation instruction. An information processing device that receives and controls to display a screen containing a processing result regarding the image.

According to such an aspect, any image processing corresponding to the operation of the object can be executed by the user's selection.

In the information processing device, the screen includes an object, the object differs for each different processing result of the image, and the control unit controls the processing result of the image corresponding to the object selected by the user. , and controls to display a screen including a processing result of the image.

According to such an aspect, any image processing corresponding to the object can be executed by the user's selection.

In the information processing apparatus, when a first object and a second object are selected through the screen, the control unit outputs a first processing result regarding the image corresponding to the first object. receiving an instruction to display and an instruction to display a second processing result regarding the image corresponding to the second object, and performing the first processing result regarding the image and the second processing regarding the image; An information processing device that controls to display a screen containing results.

According to this aspect, a plurality of arbitrary image processes can be executed simultaneously.

In the information processing device, the image processing related to the first object and the image processing related to the second object are different in an image processing target region in the image.

According to this aspect, it is possible to simultaneously perform a plurality of arbitrary image processes on different regions.

In the information processing apparatus, the object displayed on the screen is an object related to image processing related to weather.

According to such an aspect, it is possible to perform appropriate image processing corresponding to the weather.

In the information processing apparatus, the control unit changes an object to be displayed on the screen based on the state of the image.

According to this aspect, it is possible to perform appropriate image processing based on the information in the image.

In the information processing apparatus, the control unit controls to display an object suggesting image processing for the image on the screen based on road network information.

According to this aspect, appropriate image processing can be performed based on the road network information.

In the information processing device, the control unit controls to display on the screen an object suggesting image processing for the image based on information from an acceleration sensor that detects acceleration of a moving body having the imaging unit. processing equipment.

According to this aspect, it is possible to perform appropriate image processing based on the information from the acceleration sensor of the moving body.

In the information processing apparatus, the control unit performs control to display on the screen an object suggesting image processing for the image based on position information of a moving body having the imaging unit.

According to such an aspect, it is possible to perform appropriate image processing based on the position information of the moving body.

In the information processing apparatus, the control unit displays on the screen an object suggesting image processing for the image based on location information of a moving body having the image capturing unit and weather information at the date and time when the image was captured. Information processing device that controls to

According to this aspect, it is possible to perform appropriate image processing based on the location information of the moving body and the weather information of the date and time at that location.

In the information processing apparatus, the control unit performs control to display an object suggesting image processing for the image on the screen based on operation information of a moving object having the imaging unit.

According to such an aspect, it is possible to perform appropriate image processing based on the operation information of the moving object.

In the information processing apparatus, the control unit performs control to display on the screen an object suggesting image processing for the image based on the date and time when the image was captured.

According to this aspect, it is possible to perform appropriate image processing based on the date and time when the image was taken.

In the information processing device, the image processing for the image is image processing for a part of the image captured by the imaging unit.

According to this aspect, it is possible to perform image processing on part of the captured image.

In the information processing apparatus, the part of the image is a set area out of the entire area of the image captured by the imaging unit.

According to such an aspect, it is possible to set the area where the image processing is performed.

In the information processing device, the part of the image is an image captured in a set time zone among the images captured by the imaging unit.

According to this aspect, it is possible to appropriately perform image processing on a partial area of an image captured during the set time period.

In the information processing device, the part of the image is an image captured within a range in which the position information of the moving body having the imaging unit is set.

According to such an aspect, it is possible to appropriately perform image processing on a partial area of an image captured by a moving body moving in a set position.

In the information processing apparatus, the part of the image is an image indicating the weather set with weather information of the date and time when the image was taken at the position of the moving body having the image capturing unit.

According to such an aspect, it is possible to appropriately perform image processing on a partial area of an image captured based on the weather at the position of the mobile object.

In the information processing device, the part of the image is an image within a threshold set by the sensor information of the moving body having the imaging unit.

According to this aspect, it is possible to appropriately perform image processing on a partial area of an image captured based on the sensor information of the moving object.

In the information processing apparatus, the screen includes information as to whether the range in which the image processing is performed has been automatically selected or manually selected.

According to this aspect, it is possible to identify whether the range of image processing that has been performed has been automatically set or manually selected.

In the information processing apparatus, the screen includes an image captured by the capturing unit and an image after one or more image processes.

According to such an aspect, it is possible to compare and display before and after image processing.

In the information processing apparatus, the control unit adjusts image processing parameters based on learning information related to selection of the one or more image-processed images.

According to this aspect, it is possible to learn the selected image processing result and perform subsequent image processing more appropriately.

In the information processing device, the screen includes an image after image processing, and the control unit stores the image captured by the imaging unit in a storage area.

According to this aspect, the original image can be stored while displaying the image after the image processing on the screen.

In the information processing device, the screen includes information about what kind of image processing has been performed on the image captured by the imaging unit.

According to such an aspect, it is possible to confirm what kind of image processing has been performed.

In the information processing apparatus, the control unit stores information on what kind of image processing was performed on the image captured by the imaging unit as log information in a storage area.

According to such an aspect, it is possible to confirm what kind of image processing has been performed.

In the information processing apparatus, the control unit converts an image after image processing into an image before image processing based on the log information, and outputs the image to the screen.

According to such an aspect, the image after image processing can be restored to the original image before image processing.

In the information processing apparatus, the control unit changes an object included in the screen according to an object detected from the image.

According to this aspect, it is possible to change the image processing method based on the object in the image.

In the information processing device, the screen includes an adjustment object for adjusting the degree of the image processing, and the control unit adjusts the degree of the image processing according to an operation of the adjustment object. Device.

According to this aspect, the degree of image processing such as brightness can be adjusted.

In the information processing device, the control unit associates and stores the timing of operation of the adjustment object and the recording timing of the image or the reproduction timing of the image.

According to such an aspect, it becomes possible to know at what timing the degree of image processing was adjusted.

In the information processing device, the screen includes a fixed instruction object that prohibits adjustment by the adjustment object, and the control unit prohibits adjustment by the adjustment object in accordance with an operation of the fixed instruction object. Information processing equipment.

According to such an aspect, the degree of image processing can be fixed.

In the information processing device, the control unit adjusts the degree of image processing based on learning information about the degree of adjustment of the image processing in the past.

According to this aspect, it is possible to learn the adjustment result of the degree of image processing, and to perform subsequent image processing more appropriately.

In the information processing apparatus, the screen includes an image output object, and the control unit deletes the position information of the image and outputs the image in accordance with an operation of the image output object. Device.

According to this aspect, it is possible to perform an operation to delete the position information associated with the image.

In the information processing device, if the image that has undergone image processing contains information that can identify an individual, the control unit requests execution of mask processing for the information that can identify the individual. Information processing equipment.

According to this aspect, it is possible to appropriately perform image processing so that information that can identify an individual is not included.

In the information processing apparatus, the control unit executes image processing based on an operation instruction through the screen.

According to such an aspect, the information processing device can execute image processing.

An information processing apparatus, which is an image processing apparatus communicable with the information processing apparatus, comprising a control section, wherein the control section executes image processing based on a request from the information processing apparatus.

A program for causing a computer to function as a control section of the information processing apparatus.

Of course, it is not limited to this.

It should be noted that the scope of the present invention is not limited to the configurations or limitations expressly set forth herein, and encompasses within its scope any combination of the various aspects of the invention disclosed herein. While the claimed features of the invention are identified in the appended claims, any configuration not presently specified in the claims is disclosed herein. We intend to claim such configurations in the future.

The present invention is not limited to the configurations described in the above-described embodiments. The constituent elements of the above-described embodiments and modifications may be arbitrarily selected and combined. In addition, arbitrary constituent elements of each embodiment and modifications, arbitrary constituent elements described in Means for Solving the Invention, or constituent elements embodying arbitrary constituent elements described in Means for Solving the Invention and may be configured in any combination. We intend to acquire the rights for these as well in the amendment of the present application or in the divisional application. Even if there is a description of "in the case of" or "when", it is not described as a configuration limited to that case or that time. We disclose and intend to reserve these occasions and occasions. Moreover, the parts described with order are not limited to this order. It also discloses a configuration in which some parts are deleted or the order is changed, and there is an intention to acquire the right.

In addition, the applicant intends to acquire rights to the entire design or partial design by converting the design registration application. Although the entire device is drawn in solid lines in the drawing, the drawing includes not only the overall design but also the partial design claimed for a part of the device. For example, it is a drawing that includes a part of the device as a partial design regardless of the member, as well as a partial design of a part of the member of the device. The part of the device may be a part of the device or a part of the member. We intend to acquire rights not only for the overall design, but also for the partial design in which any part of the solid line part of the drawing is a broken line part. In addition, the modules, members, parts, etc. inside the housing of the device that are shown in the drawings are all subject to trade independently. It has the intention of making changes and obtaining rights.

11 : Control unit 22 : Storage unit 23 : Control unit 231 : Acquisition unit 232 : Reception unit 233 : Correction unit 234 : Cancellation unit 235 : Display control unit 32 : Storage unit 33 : Control unit 331 : Acquisition unit 332 : Reception unit 333 : Recording unit 334 : Correction unit 335 : Display control unit 4 : Playback screen 41 : Correction area 42 : Non-correction area 43 : Additional information 5 : Seek bar 51 : Correction time IF1 : Specific information IF2 : Road network information IF3 : Terrain information IF4 : Motion information IM : Image data P1 : First program P2 : Second program P3 : Third program P4 : Fourth program PL : Parameter log PT : Parameter R : Region of interest 3000 : Information processing system 3011 : Control Unit 3013: Display unit 3014: Audio output unit 3015: Imaging unit 3111: Processor 3100: Imaging device 3200: Imaging device 3400: Vehicle 3500: Recording medium 3700: Server 3710: Control unit 3711: Image processing unit 3712: Communication control unit 3713 : Storage control unit 3720: Storage unit 3730: Communication unit 3800: Information display terminal 3810: Control unit 3811: Display control unit 3812: Communication control unit 3813: Storage control unit 3820: Storage unit 3830: Communication unit 3840: Input unit 3850: Output unit 3900: Network 4100: Playback window 4101: File button 4102: Display button 4103: Playback button 4104: Tool button 4105: Information button 4110: Video display area 4120: Acceleration graph display area 4130: Map image display area 4131: Current location icon 4140: Fog correction button 4141: HDR button 4142: Sharpening button 4143: Other processing button 4144: Partial processing button 4145: Stream display button 4146: Operation button 4150: Date and time display area 4160: Vehicle state display area 4170: Speed graph display Area 4180: playlist display area 4200: playback window 4210: video display area 4240: fog correction button 4300: playback window 4310: video display area 4311: fog correction area 4312: processing content display section 4313: sharpening region 4314: processing content Display unit 4340: Fog correction button 4342: Sharpening button 4344: Partial processing button 4400: Reproduction window 4440: Sunny correction button 4441: Cloud correction button 4442: Rain correction button 4443: Fog correction button 4500: Reproduction window 4510: video display area 4540: fog correction button 4600: playback window 4610: video display area 4630: map image display area 4631: current location icon 4640: tunnel correction button 4700: playback window 4720: acceleration graph display area 4721: acceleration display section 4740 : Acceleration correction button 4770: Speed graph display area 4800: Playback window 4810: Video display area 4830: Map image display area 4831: Current location icon 4832: Processing area 4840: Fog correction button 4900: Playback window 4910: Video display area 4911: Weather Information 4941: Rain correction button 5000: Playback window 5010: Video display area 5030: Map image display area 5031: Current location icon 5040: Brake correction button 5070: Speed graph display area 5100: Playback window 5110: Video display area 5140: Night correction button 5150: date and time display area 5200: playback window 5210: video display area 5211: fog correction area 5240: fog correction button 5300: playback window 5310: video display area 5311: night correction area 5340: night correction button 5350: date and time display area 5400: Playback window 5410: video display area 5411: fog correction area 5430: map image display area 5431: current location icon 5432: processing area 5440: fog correction button 5500: playback window 5510: video display area 5511: fog correction area 5512: weather information 5540 : Cloud correction button 5541: Rain correction button 5600: Playback window 5610: Video display area 5611: Acceleration correction area 5620: Acceleration graph display area 5621: Acceleration display section 5640: Acceleration correction button 5700: Playback window 5710: Video display area 5711: Automatic fog correction area 5712: Processing content display section 5713: Manual fog correction area 5714: Processing content display section 5740: Fog correction button 5745: Stream display button 5800: Playback window 5810: Video display area 5811: Original image display section 5812: Original Display section 5813: Process 1 result display section 5814: Process 1 content display section 5815: Process 2 result display section 5816: Process 2 content display section 5817: Process 3 result display section 5818: Process 3 content display section 5846: Operation button 5900: Playback window 5910: Video display area 5911: Processing content display section 5940: Fog correction button 6000: Playback window 6080: Playlist table Display area 6081: playlist 6100: playback window 6110: video display area 6111: detection information display part 6140: fog correction button 6200: playback window 6290: image processing adjustment area 6291: hue adjustment object 6292: brightness adjustment object 6293: contrast Adjustment object 6294: Saturation adjustment object 6295: Gamma value adjustment object 6296: Sigma value adjustment object 6297: Radius adjustment object 6298: Variation amount adjustment object 6300: Playback window 6390: Image processing adjustment area 6391: Image adjustment fixed object 6392: Sharpness Adjustment fixed object 6393: Banding noise removal adjustment fixed object 6394: Film graining adjustment fixed object 6400: Playback window 6401: File button 6410: Video display area 6500: Playback window 6510: Video display area 6511: Other vehicle 6512: License plate information 6540: Personal information processing button 6541: HDR button 6600: Information processing system 6800: Information processing terminal 6811: Display control unit 6812: Communication control unit 6813: Storage control unit 6814: Image processing unit

Claims (23)

a system,
Equipped with a control unit,
The control unit is configured to perform the following steps,
The obtaining step obtains specific information about the state or situation of the mobile body,
In a first correction step, based on the specific information, performing a first correction process on image data generated by an imaging device of the moving body at a time corresponding to the specific information,
In the display control step, the image data on which the first correction process has been executed is displayed in a reproducible manner. The system of claim 1, wherein
In the first correction step, the first correction process for the image data is performed on the basis of the specific information so that a correction range, which is a range to be corrected, is variable. 3. The system of claim 2, wherein
The correction range is a region of interest set in the frame of the image data. In the system according to any one of claims 1 to 3,
The control unit is configured to further perform the following steps,
In a first receiving step, receiving a first input by the user,
The canceling step cancels the first correction process on the image data based on the first input. In the system according to any one of claims 1 to 4,
The control unit is configured to further perform the following steps,
A second receiving step receives a second input from the user,
A second correction step executes a second correction process on the image data based on the second input. 6. The system of claim 5, wherein
In the display control step, the image data is displayed reproducibly in such a manner that the first correction process and the second correction process can be distinguished from each other. In the system according to any one of claims 1 to 6,
The specific information is at least one of the position, orientation, speed, acceleration, time, time zone, weather, running state, and pedal work of the mobile body. In the system according to any one of claims 1 to 7,
The control unit is configured to further perform an obtaining step,
The specific information is the position of the mobile body,
The obtaining step obtains road network information,
In the first correction step, a first correction process is performed on the image data based on the position and the road network information. In the system according to any one of claims 1 to 8,
The control unit is configured to further perform an obtaining step,
The specific information is the position of the mobile object,
The obtaining step obtains terrain information,
In the first correction step, a first correction process is performed on the image data based on the position and the terrain information. In the system according to any one of claims 1 to 9,
the specific information is the position and orientation of the moving body;
In the first correction step, a first correction process is performed on the image data based on the position of the sun estimated from the position and the orientation. In the system according to any one of claims 1 to 10,
The first correction step comprises a preparation step and a processing step,
The control unit is configured to be able to call a first program stored in advance and a second program different from the first program,
The first program causes the preparatory step to be executed,
In the preparatory step,
extracting parameters included in the image data;
passing the extracted parameter values to the second program;
The second program causes the processing step and the display control step to be executed,
In the processing step, the value of the parameter is changed by executing the first correction processing on the parameter;
In the display control step, the image data based on the value of the parameter changed in the processing step is reproducibly displayed. 12. The system of claim 11, wherein
The parameters include at least one of hue, brightness, saturation, contrast, gamma, sharpness, playback start time, playback end time, and presence/absence of loop. A system according to claim 11 or claim 12, wherein
In the preparation step, passing operation information about the operation of the second program to the second program;
In the display control step, based on the operation information, at least part of the functions of the second program are restricted to display the image data in a reproducible manner. In the system according to any one of claims 11 to 13,
The first program is configured to execute the display control step,
In the display control step, if the second program is not installed, a screen recommending installation of the second program is displayed. In the system according to any one of claims 11 to 13,
The second program is a program stored in an external device,
The control unit is configured to call the second program via a network. A system comprising a control unit,
The control unit is configured to perform the following steps,
In the recording step, the video captured by the camera is recorded as image data,
In the display control step, the image captured by the camera is displayed as a live view,
The parameters of the recorded image data and the displayed video are set differently. 17. The system of claim 16, wherein
In the recording step, the image is recorded as the image data without correction,
In the display control step, the image is corrected and displayed as the live view. 18. The system of claim 17, wherein
In the recording step, parameters of the displayed image are recorded as a parameter log in chronological order together with the image data. 19. The system of claim 18, wherein
The control unit is configured to further execute a third receiving step,
In the third receiving step, receiving a third input by the user;
In the display control step, based on the third input, the image data and the parameter log are read, and the corrected image data is reproducibly displayed. In the system according to any one of claims 16 to 19,
The control unit is configured to be able to call a third program stored in advance and a fourth program different from the third program,
In the display control step, when the image data is reproduced by the fourth program, information prompting reproduction by the third program is displayed. 21. The system of claim 20, wherein
The information prompting reproduction is displayed together with a reproduction screen of the image data. An information processing method,
A method comprising the steps of a system according to any one of claims 1-21. a program,
A computer that functions as a controller of the system according to any one of claims 1 to 21.

Images