JP7210394B2 - INFORMATION PROVIDING DEVICE, INFORMATION PROVIDING METHOD AND PROGRAM - Google Patents

Description

The present invention relates to an information providing device, an information providing method, and a program.

Images taken by cameras mounted on vehicles can be used to recognize the surroundings of the vehicle, fixed cameras installed at predetermined locations can capture images of the surroundings, and the captured images can be used to monitor the surroundings and detect suspicious persons. is known (see Patent Documents 1 to 3, for example).

Japanese Patent No. 3991314 JP 2004-254087 A JP-A-2000-032436

However, in the prior art, no consideration has been given to providing the user with images captured for various purposes.

One aspect of the present invention has been made in consideration of such circumstances, and aims to provide an information providing apparatus, an information providing method, and a program capable of providing a user with more appropriate image information. one of the purposes.

An information providing device, an information providing method, and a program according to the present invention employ the following configurations.
(1): An information providing device according to an aspect of the present invention is a first image capturing unit mounted on a vehicle and a first image capturing unit mounted on a vehicle based on position information of an image capturing target according to image capturing content received from a terminal device used by a user. an imaging control unit that gives predetermined instructions to a plurality of imaging units including a second imaging unit installed at a predetermined position and a third imaging unit that is used by someone other than the user; and a predetermined instruction by the imaging control unit. an image acquisition unit that acquires an image including an imaging target captured by the first imaging unit, the second imaging unit, and the third imaging unit by and an image that satisfies the imaging content acquired by the image acquisition unit and an information providing unit that provides the terminal device with information containing:

(2): In the above aspect (1), the imaging content includes control content not to upload the images captured by the plurality of imaging units to the information providing device.

(3): In the aspect of (1) above, the imaging content includes control content for uploading an image captured outside the predetermined area to the information providing device.

(4): In the aspect of (1) above, the captured image is an image captured by at least a part of the first imaging section, the second imaging section, and the third imaging section. It contains control content that permits

(5): In the aspect of (1) above, the imaging content includes attribute information of the owner of the third imaging unit, and the content of control permitting upload from the third imaging unit corresponding to the attribute. includes.

(6): In an information providing method according to an aspect of the present invention, an information providing device installed in a vehicle based on position information of an imaging target according to imaged content received from a terminal device used by a user. giving a predetermined instruction to a plurality of imaging units including a first imaging unit, a second imaging unit installed at a predetermined position, and a third imaging unit used by someone other than the user; 1 image capturing unit, the second image capturing unit, and the third image capturing unit acquire an image including an image capturing target, and transmit information including an image satisfying the image capturing content acquired by the image acquiring unit to the terminal device; It is a method of providing information to

(7): A program according to an aspect of the present invention provides an information providing device with a first image, which is mounted on a vehicle, based on position information of an imaged target according to imaged content received from a terminal device used by a user. A plurality of image capturing units including an image capturing unit, a second image capturing unit installed at a predetermined position, and a third image capturing unit used by someone other than the user are caused to issue a predetermined instruction, and the first image capturing unit is operated according to the predetermined instruction. Acquiring an image including an imaging target imaged by an imaging unit, the second imaging unit, and the third imaging unit, and transmitting information including the image satisfying the imaging content acquired by the image acquisition unit to the terminal device It is a program that provides

According to the above (1) to (7), more appropriate image information can be provided to the user.

1 is a configuration diagram of an information providing system 1 including an information providing device of an embodiment; FIG. 1 is a configuration diagram of a vehicle system 3 according to an embodiment; FIG. 3 is a functional configuration diagram of a first controller 120 and a second controller 160. FIG. 2 is a diagram showing an example of a functional configuration of a running driving force output device 200; FIG. 1 is a configuration diagram of an information providing server 300 of an embodiment; FIG. It is a figure which shows an example of the content of camera DB362. It is a figure which shows an example of the content of image DB364. It is a figure which shows an example of the content of user DB366. FIG. 11 is a diagram showing an example of the content of a provided image DB 368; FIG. It is a figure which shows an example of the content of provider DB370. 1 is a configuration diagram of a terminal device 400 according to an embodiment; FIG. 2 is a diagram showing a configuration diagram of a fixed camera device 500. FIG. 6 is a configuration diagram of a drone 600. FIG. BRIEF DESCRIPTION OF THE DRAWINGS It is a figure for demonstrating the specific example of the information provision system 1 of embodiment. Fig. 2 shows an example of an image IM1 containing an image request; FIG. 4 is a diagram for explaining the contents of shooting level information; FIG. It is a figure for demonstrating a mode that the own vehicle M is imaged. FIG. 10 is a diagram showing an example of an image IM2 for inquiring whether or not to shoot a terminal device 400-2. FIG. 4 is a diagram showing an example of an image IM3 including feature information; 4A and 4B are diagrams showing an example of an image extracted by an extraction unit 346; FIG. It is a figure which shows an example of image IM5 containing the user U2 by which the own vehicle M was imaged. FIG. 10 is a diagram showing an example of an image IM6 including a user U1 seated in a driver's seat DS of own vehicle M; FIG. 11 is a diagram showing an example of an image album AL1 edited by an editing unit 348; FIG. 11 is a diagram showing an example of an image album AL2 edited by an editing unit 348; It is a sequence diagram which shows an example of the flow of processing of the information provision system 1 of embodiment. FIG. 10 is a diagram for explaining guiding an imaging target; FIG. It is a figure which shows an example of the hardware constitutions of the information provision server 300 of embodiment.

Hereinafter, embodiments of an information providing apparatus, an information providing method, and a program according to the present invention will be described with reference to the drawings.
[overall structure]
FIG. 1 is a configuration diagram of an information providing system 1 including an information providing device of an embodiment. The information providing system 1 includes, for example, one or more vehicles 2, an information providing server 300, one or more terminal devices 400, one or more fixed camera devices 500, and one or more drones 600. In the example of FIG. 1, it is assumed that the vehicle system 3 is installed in the vehicle 2 . The vehicle system 3, the information providing server 300, the terminal device 400, the fixed camera device 500, and the drone 600 can communicate with each other via the network NW, for example. The network NW is, for example, a cellular network, a Wi-Fi network, Bluetooth (registered trademark), the Internet, a WAN (Wide Area Network), a LAN (Local Area Network), a public line, a provider device, a dedicated line, a wireless base station, etc. including. The information providing server 300 is an example of an "information providing device". Drone 600 is an example of a "flying object." Flying objects include, for example, remote-controlled unmanned flying objects and manned flying objects.

Each of vehicle system 3, terminal device 400, fixed camera device 500, and drone 600 includes a camera (an example of an imaging unit). Images captured by each camera are transmitted to the information providing server 300 via the network NW. The information providing server 300 receives images captured by each camera, selects or edits images to be provided to the user from the received captured images, and provides information including the selected or edited images to the user. Note that the image provided to the user may include a video (moving image).

Each function of the vehicle system 3, the information providing server 300, the terminal device 400, the fixed camera device 500, and the drone 600 will be described below. In the following description, terminal devices used by users U1 and U2 are referred to as terminal devices 400-1 and 400-2, respectively. Further, the vehicle owned or driven by the user U1 is referred to as own vehicle M, and the vehicles other than own vehicle M are referred to as other vehicles m. The vehicle system 3 mounted on the own vehicle M and the other vehicle m is applied to, for example, an automatically driving vehicle. Automatic driving means, for example, controlling one or both of steering and acceleration/deceleration of a vehicle to execute driving control. The above-described driving control includes, for example, ACC (Adaptive Cruise Control System), TJP (Traffic Jam Pilot), ALC (Auto Lane Changing), CMBS (Collision Mitigation Brake System), LKAS (Lane Keeping Assistance System), etc. is included. Further, the automatic driving vehicle may be controlled by manual driving by the passenger (driver). In addition, an automatic driving vehicle is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electric power generated by a generator connected to the internal combustion engine, or electric power discharged from a secondary battery or a fuel cell. In the following, the case where the left-hand traffic regulation applies will be explained, but when the right-hand traffic regulation applies, the right and left should be reversed. In the following description, X is one horizontal direction, Y is the other horizontal direction, and Z is a vertical direction perpendicular to the XY horizontal direction.

[Vehicle system]
FIG. 2 is a configuration diagram of the vehicle system 3 according to the embodiment. The vehicle system 3 mounted on the own vehicle M will be described below. The vehicle system 3 includes, for example, a camera 10, a radar device 12, a finder 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, An MPU (Map Positioning Unit) 60 , a driving operator 80 , a vehicle interior camera 90 , an automatic driving control device 100 , a driving force output device 200 , a braking device 210 and a steering device 220 are provided. These apparatuses and devices are connected to each other by multiplex communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, and the like. Note that the configuration shown in FIG. 2 is merely an example, and a part of the configuration may be omitted, or another configuration may be added.

The camera 10 is, for example, a digital camera using a solid-state imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached to the arbitrary location of the own vehicle M. As shown in FIG. For example, when capturing an image in front of the own vehicle M, the camera 10 is attached to the upper part of the front windshield, the rear surface of the rearview mirror, or the like. Moreover, when imaging the back of the own vehicle M, the camera 10 is attached to the upper part of the rear windshield or the like. Moreover, when imaging the right side or the left side of the own vehicle M, the camera 10 is attached to the vehicle body, the right side or left side of a door mirror, or the like. The camera 10, for example, repeatedly images the surroundings of the own vehicle M periodically. Camera 10 may be a stereo camera.

The radar device 12 radiates radio waves such as millimeter waves around the vehicle M, and detects radio waves (reflected waves) reflected by an object to detect at least the position (distance and direction) of the object. The radar device 12 is attached to any location of the own vehicle M. As shown in FIG. The radar device 12 may detect the position and velocity of an object by the FM-CW (Frequency Modulated Continuous Wave) method.

The finder 14 is a LIDAR (Light Detection and Ranging). The finder 14 irradiates light around the vehicle M and measures scattered light. The finder 14 detects the distance to the object based on the time from light emission to light reception. The irradiated light is, for example, pulsed laser light. The viewfinder 14 is attached to any location on the host vehicle M. As shown in FIG.

The object recognition device 16 performs sensor fusion processing on the detection results of some or all of the camera 10, the radar device 12, and the finder 14, and recognizes the position, type, speed, etc. of the object. The object recognition device 16 outputs recognition results to the automatic driving control device 100 . Object recognition device 16 may output the detection result of camera 10, radar device 12, and finder 14 to automatic operation control device 100 as it is. In this case, the object recognition device 16 may be omitted from the vehicle system 3 .

The communication device 20 uses, for example, a cellular network, a Wi-Fi network, Bluetooth, DSRC (Dedicated Short Range Communication), or the like, to communicate with other vehicles existing in the vicinity of the own vehicle M, or via a radio base station. communication with various information providing servers.

The HMI 30 presents various types of information to the occupants (including the driver) of the host vehicle M and accepts input operations by the occupants. The HMI 30 includes, for example, various display devices, speakers, buzzers, touch panels, switches, keys, and the like. The HMI 30 may also include a microphone that picks up sounds inside or outside the vehicle.

The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the vehicle M, an acceleration sensor that detects acceleration, a yaw rate sensor that detects angular velocity about a vertical axis, a direction sensor that detects the direction of the vehicle M, and the like. The vehicle sensors 40 include a position sensor for detecting the position (latitude and longitude) of the vehicle M, a vibration sensor for detecting shaking and vibration of the vehicle M, a rainfall sensor for measuring rainfall outside the vehicle, and a wind speed outside the vehicle. A wind speed sensor or the like that measures such as may also be included.

The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51 , a navigation HMI 52 and a route determining section 53 . The navigation device 50 holds first map information 54 in a storage device such as an HDD (Hard Disk Drive) or flash memory. The GNSS receiver 51 identifies the position of the own vehicle M based on the signals received from the GNSS satellites. The position of the own vehicle M may be specified or complemented by an INS (Inertial Navigation System) using the output of the vehicle sensor 40 . The navigation HMI 52 includes a display device, speaker, touch panel, keys, and the like. The navigation HMI 52 may be partially or entirely shared with the HMI 30 described above. For example, the route determination unit 53 determines a route from the position of the own vehicle M specified by the GNSS receiver 51 (or any input position) to the destination input by the occupant using the navigation HMI 52 (hereinafter referred to as route on the map) is determined with reference to the first map information 54 . The first map information 54 is, for example, information in which road shapes are represented by links indicating roads and nodes connected by the links. The first map information 54 may include road curvature, POI (Point Of Interest) information, and the like. A route on the map is output to the MPU 60 . The navigation device 50 may provide route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be implemented by, for example, the functions of the terminal device 400-1 owned by the passenger. The navigation device 50 may transmit the current position and the destination to the navigation server via the communication device 20 and acquire a route equivalent to the route on the map from the navigation server.

The MPU 60 includes, for example, a recommended lane determination unit 61, and holds second map information 62 in a storage device such as an HDD or flash memory. The recommended lane determining unit 61 divides the route on the map provided from the navigation device 50 into a plurality of blocks (for example, by dividing each block by 100 [m] in the vehicle traveling direction), and refers to the second map information 62. Determine recommended lanes for each block. The recommended lane decision unit 61 decides which lane to drive from the left. The recommended lane determination unit 61 determines a recommended lane so that the vehicle M can travel a rational route to the branch when there is a branch on the route on the map.

The second map information 62 is map information with higher precision than the first map information 54 . A combination of the first map information 54 and the second map information 62 is an example of "map information". The second map information 62 includes, for example, lane center information or lane boundary information. Further, the second map information 62 may include road information, traffic regulation information, address information (address/zip code), facility information, telephone number information, and the like. Further, one or both of the first map information 54 and the second map information 62 may include more detailed road information than the road information described above. For details, road information includes, for example, types of traffic objects installed on roads, information on installation time, information on features around roads, power supply sources for traffic objects, etc., durability of roads, altitude from sea level is information such as the height of Traffic objects include, for example, traffic lights, traffic signs, stationary cameras that image the road, and other objects placed around the road. In addition, features include, for example, structures such as buildings, bridges, and steel towers, and various objects including trees and plants such as forests and windbreaks. Further, the information about the thing includes information about the type and size of the constructed or planted object, and the time when it was constructed or planted. Also, one or both of the first map information 54 and the second map information 62 may be updated at any time by the communication device 20 communicating with an external device such as the information providing server 300 .

The driving operator 80 includes, for example, an accelerator pedal, a brake pedal, a shift lever, a steering wheel, a modified steering wheel, a joystick, and other operators. A sensor that detects the amount of operation or the presence or absence of operation is attached to the driving operation element 80, and the detection result is applied to the automatic driving control device 100, or the driving force output device 200, the brake device 210, and the steering device. 220 to some or all of them.

The vehicle interior camera 90 is, for example, a digital camera using a solid-state imaging device such as CCD or CMOS. The vehicle interior camera 90 may be a stereo camera. The vehicle interior camera 90 is attached to any location in the vehicle interior. The vehicle interior camera 90 captures an image of an area including each seat present in the vehicle interior. As a result, images of the occupants seated on the respective seats can be acquired from the images captured by the vehicle interior camera 90 . The in-vehicle camera 90 may repeatedly take images of the above region periodically, or may take images at predetermined timings. The predetermined timing is the timing at which the own vehicle M reaches a predetermined point, or the timing at which the photographing instruction is received from the information providing server 300 . An image captured by the vehicle interior camera 90 is output to the automatic driving control device 100 .

The automatic operation control device 100 includes, for example, a first control unit 120, a second control unit 160, and an image acquisition control unit 180. The first control unit 120, the second control unit 160, and the image acquisition control unit 180 are each implemented by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components are hardware (circuits) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit) (including circuitry), or by cooperation of software and hardware. The program may be stored in advance in a storage device such as the HDD or flash memory of the automatic operation control device 100 (a storage device having a non-transitory storage medium), or may be detachable such as a DVD or CD-ROM. It is stored in a storage medium, and may be installed in the HDD or flash memory of the automatic operation control device 100 by attaching the storage medium (non-transitory storage medium) to the drive device.

FIG. 3 is a functional configuration diagram of the first control unit 120 and the second control unit 160. As shown in FIG. The 1st control part 120 is provided with the recognition part 130 and the action plan production|generation part 140, for example. A combination of the action plan generation unit 140 and the second control unit 160 is an example of the "operation control unit". The first control unit 120, for example, realizes in parallel a function based on AI (Artificial Intelligence) and a function based on a model given in advance. For example, the "recognize intersections" function performs in parallel recognition of intersections by deep learning, etc., and recognition based on predetermined conditions (signals that can be pattern-matched, road markings, etc.). It may be realized by scoring and evaluating comprehensively. This ensures the reliability of automated driving.

The recognition unit 130 recognizes the surrounding environment of the own vehicle M. FIG. For example, the recognition unit 130 recognizes an object (for example, a surrounding vehicle or a target) around the own vehicle M based on information input from the camera 10, the radar device 12, and the finder 14 via the object recognition device 16. position, speed, acceleration, direction of travel, etc. The position of the object is recognized, for example, as a position on absolute coordinates with a representative point (the center of gravity, the center of the drive shaft, etc.) of the own vehicle M as the origin, and used for control. The position of an object may be represented by a representative point such as the center of gravity, center, or corner of the object, or may be represented by a represented area. If the object is a vehicle, the "state" of the object may include acceleration or jerk of the object, or "behavioral state" (eg, whether it is making or attempting to turn).

Further, the recognition unit 130 recognizes, for example, the lane in which the host vehicle M is traveling (running lane). For example, the recognition unit 130 recognizes a pattern of road markings obtained from the second map information 62 (for example, an array of solid lines and broken lines) and road markings around the vehicle M recognized from the image captured by the camera 10. The driving lane is recognized by comparing with the pattern of Note that the recognition unit 130 may recognize the driving lane by recognizing lane boundaries (road boundaries) including road division lines, road shoulders, curbs, medians, guardrails, etc., in addition to road division lines. . In this recognition, the position of the own vehicle M acquired from the navigation device 50 and the processing result by the INS may be taken into consideration. The recognition unit 130 also recognizes stop lines, obstacles, red lights, toll gates, and other road events.

The recognition unit 130 recognizes the position and posture of the own vehicle M with respect to the driving lane when recognizing the driving lane. For example, the recognizing unit 130 calculates the deviation of the reference point of the own vehicle M from the lane center and the angle formed with the line connecting the lane center in the direction of travel (Y direction) of the own vehicle M to the traveling lane. It may be recognized as the relative position and orientation of M. Instead, the recognition unit 130 recognizes the position of the reference point of the own vehicle M with respect to one of the side edges (road division line or road boundary) of the driving lane as the relative position of the own vehicle M with respect to the driving lane. You may

The recognition unit 130 recognizes surrounding vehicles of the own vehicle M recognized from the image captured by the camera 10, the image captured by the camera 10, traffic information around the own vehicle M acquired by the navigation device 50, or the 2) Based on the positional information obtained from the map information 62, information regarding the position of surrounding vehicles is recognized.

Note that the recognition unit 130 acquires various types of information received from vehicles traveling around the own vehicle M through inter-vehicle communication via the communication device 20, and recognizes the surroundings of the own vehicle M based on the information. may Further, the recognition unit 130 may recognize an object that is highly similar to the feature information based on the feature information of the imaging target included in the image acquisition instruction information acquired from the information providing server 300 . Objects to be imaged include, for example, objects such as people, vehicles, and features, and landscapes. Also, the person may include the user himself/herself, the user's family, acquaintances, celebrities, and the like.

The action plan generation unit 140 and the second control unit 160 control one or both of the speed and steering of the vehicle M based on the recognition result of the recognition unit 130 to perform driving control. In principle, the action plan generation unit 140 drives the recommended lane determined by the recommended lane determination unit 61 when automatic driving is executed, A target trajectory on which the vehicle M will travel in the future is automatically generated (without relying on the operation of the driver). The target trajectory includes, for example, velocity elements. For example, the target trajectory is represented by arranging points (trajectory points) that the host vehicle M should reach in order. A trajectory point is a point to be reached by the own vehicle M for each predetermined travel distance (for example, about several [m]) along the road. ) are generated as part of the target trajectory. Also, the trajectory point may be a position that the vehicle M should reach at each predetermined sampling time. In this case, the information on the target velocity and target acceleration is represented by the intervals between the trajectory points.

The action plan generator 140 may set an automatic driving event when generating the target trajectory. Automatic driving events include constant-speed driving events, low-speed following driving events, lane change events, branching events, merging events, and takeover events for ending automatic driving and switching to manual driving. The action plan generator 140 generates a target trajectory according to the activated event.

Further, the action plan generation unit 140 controls the speed of the own vehicle M based on the control information from the image acquisition control unit 180 when the imaging target is photographed by the camera 10 or when the own vehicle M is photographed. A target trajectory is generated for performing steering control (for example, decelerating at the time of photographing), traveling in a position or route that facilitates photographing, or directing the camera to the front. The object to be imaged includes, for example, a person to be imaged and an object to be imaged (such as a vehicle). In addition, the action plan generating unit 140 outputs the driving force from the driving force output device 200 based on the instruction operation by the passenger using the HMI 30, the driving condition of the own vehicle M, and the control information from the image acquisition control unit 180. (For example, a drive mode to be described later) may be controlled to switch.

The second control unit 160 controls the driving force output device 200, the braking device 210, and the steering device 220 so that the vehicle M passes the target trajectory generated by the action plan generating unit 140 at the scheduled time. Control.

The second control unit 160 includes a target trajectory acquisition unit 162, a speed control unit 164, and a steering control unit 166, for example. The target trajectory acquisition unit 162 acquires information on the target trajectory (trajectory point) generated by the action plan generation unit 140 and stores it in a memory (not shown). Speed control unit 164 controls running driving force output device 200 or brake device 210 based on the speed element associated with the target trajectory stored in the memory. The steering control unit 166 controls the steering device 220 according to the curve of the target trajectory stored in the memory. The processing of the speed control unit 164 and the steering control unit 166 is realized by, for example, a combination of feedforward control and feedback control. As an example, the steering control unit 166 performs a combination of feedforward control according to the curvature of the road ahead of the host vehicle M and feedback control based on deviation from the target trajectory.

Returning to FIG. 2 , the image acquisition control unit 180 causes the recognition unit 130 to recognize the imaging target based on the image acquisition instruction information from the information providing server 300 , and instructs the action plan generation unit 140 to determine the imaging target with the camera 10 . For photographing, a target trajectory is generated by adjusting the speed and steering of the own vehicle M. In addition, when the own vehicle M is photographed by an external camera, the image acquisition control unit 180 adjusts the speed and steering to the action plan generation unit 140 in order to make it easier for the external camera to photograph the own vehicle M. A target trajectory may be generated. Further, the image acquisition control section 180 may cause the driving force output device 200 to switch the driving force.

Further, based on the image acquisition instruction information, the image acquisition control unit 180 extracts an image that satisfies a predetermined condition from the images captured by the camera 10 or the vehicle interior camera 90 at predetermined intervals. The predetermined condition is, for example, when the distance between the own vehicle M and the imaging target is within a predetermined distance, when the own vehicle M reaches a predetermined position, or when the characteristic information included in the image acquisition instruction information in the captured image. This is the case where an object with a high degree of similarity is included. Predetermined positions are, for example, roads, intersections, parking lots, homes, companies, stations, shops, public facilities, leisure facilities, sightseeing spots, and the like. Also, the predetermined position may be, for example, one or more positions preset by the user. The image acquisition control unit 180 may acquire voices outside or inside the vehicle from a microphone included in the HMI 30 in addition to acquiring the images described above. The acquired image or sound is transmitted (uploaded) to the information providing server 300 via the communication device 20 along with the camera ID based on the image acquisition instruction information.

The running driving force output device 200 outputs running driving force (torque) for running the vehicle to the drive wheels. FIG. 4 is a diagram showing an example of the functional configuration of the running driving force output device 200. As shown in FIG. The running driving force output device 200 includes, for example, a driving force switching unit 202, a first driving force output unit 204, and a second driving force output unit 206. The driving force switching unit 202 switches between the output of the running driving force by the first driving force output unit 204 and the output of the running driving force by the second driving force output unit 206 based on the control information from the action plan generating unit 140. . Hereinafter, the output of the running driving force by the first driving force output section 204 will be referred to as the "first driving mode", and the output of the running driving force by the second driving force output section 206 will be referred to as the "second driving mode".

The first driving force output unit 204 causes the vehicle M to travel using the internal combustion engine as a power source. The first driving force output section 204 includes, for example, an engine ECU (Electronic Control Unit) 204A and an engine 204B. Engine ECU 204A controls engine 204B when driving force switching unit 202 receives a switching instruction to execute the first driving mode. The engine ECU 204A adjusts the throttle opening, shift stage, etc. of the engine 204B according to information input from the second control unit 160 or information input from the operation operation element 80, and drives the host vehicle M to run. output force.

The second driving force output unit 206 includes, for example, a motor ECU 206A, a vehicle battery 206B, and a travel motor 206C. The motor ECU 206A controls driving of the traveling motor 206C using electric power supplied from the vehicle battery 206B. The motor ECU 206A adjusts the duty ratio of the PWM signal given to the driving motor 206C according to the information input from the second control unit 160 or the information input from the driving operation element 80, and the driving motor 206C controls the own vehicle. It outputs a running driving force (torque) for M to run. Further, the motor ECU 206A may charge the vehicle battery 206B by returning the electricity generated by forcibly turning the driving motor 206C due to the rotation of the wheels after the accelerator is released, for example. . Vehicle battery 206B is, for example, a secondary battery such as a lithium ion battery. Vehicle battery 206B is charged and discharged under the control of motor ECU 206A. The above-described second drive mode produces less drive noise due to vibrations or the like during running than the first drive mode in which the engine is driven.

The brake device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motors according to information input from the second control unit 160 or information input from the driving operator 80 so that brake torque corresponding to the braking operation is output to each wheel. The brake device 210 may include, as a backup, a mechanism that transmits hydraulic pressure generated by operating a brake pedal included in the operation operator 80 to the cylinders via a master cylinder. The brake device 210 is not limited to the configuration described above, and is an electronically controlled hydraulic brake device that controls the actuator according to information input from the second control unit 160 to transmit the hydraulic pressure of the master cylinder to the cylinder. good too.

The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the orientation of the steered wheels. The steering ECU drives the electric motor according to information input from the second control unit 160 or information input from the driving operator 80 to change the direction of the steered wheels.

[Information providing server]
FIG. 5 is a configuration diagram of the information providing server 300 of the embodiment. The information providing server 300 includes a server-side communication section 310 , an input section 320 , an output section 330 , a server-side control section 340 and a server-side storage section 360 . The information providing server 300 may function as a cloud server that communicates with, for example, the vehicle 2 (subject vehicle M, other vehicle m) on which the vehicle system 3 is mounted, via the network NW, and transmits and receives various data. .

The server-side communication unit 310 includes a communication interface such as a NIC (Network Interface Card). The server-side communication unit 310 uses, for example, a cellular network, a Wi-Fi network, Bluetooth, etc., to communicate with vehicles and other external devices via the network NW.

The input unit 320 is, for example, a user interface such as buttons, a keyboard, and a mouse. The input unit 320 receives an operation by a server administrator or the like. The input unit 320 may be a touch panel integrated with the display unit of the output unit 330 .

The output unit 330 outputs information to a server administrator or the like. The output unit 330 includes, for example, a display unit that displays images and an audio output unit that outputs audio. The display unit includes, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL display. The display unit displays an image of information output by the server-side control unit 340 . The audio output unit is, for example, a speaker. The audio output unit outputs audio of information output by the server-side control unit 340 .

The server-side control unit 340 includes, for example, an imaging control unit 342, an image acquisition unit 344, an extraction unit 346, an editing unit 348, an information providing unit 350, and a settlement unit 352. Each component of the server-side control unit 340 is implemented by, for example, executing a program stored in the server-side storage unit 360 by a processor such as a CPU. Some or all of the components of the server-side control unit 340 may be realized by hardware (circuitry) such as LSI, ASIC, FPGA, or GPU, or cooperation between software and hardware. may be realized by The above-described program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as the HDD or flash memory of the information providing server 300, or may be stored in a DVD, CD-ROM, memory card, or the like. , and may be installed in the storage device of the information providing server 300 by inserting the storage medium (non-transitory storage medium) into a drive device, card slot, or the like.

The server-side storage unit 360 is realized by, for example, a storage device such as an HDD, flash memory, EEPROM, ROM, and RAM. The server-side storage unit 360 stores, for example, a camera DB 362, an image DB 364, a user DB 366, a provided image DB 368, a provider DB 370, programs read and executed by the processor, and various other information.

The imaging control unit 342 issues imaging instructions, requests for acquiring captured images, and the like to each camera such as the camera 10 mounted on the vehicle 2, the cameras provided in the terminal device 400 and the drone 600, and the fixed camera device 500. conduct. For example, the imaging control unit 342 refers to the camera DB 362 stored by the server-side storage unit 360 and acquires position information of each camera.

FIG. 6 is a diagram showing an example of the contents of the camera DB 362. As shown in FIG. In the camera DB 362, a camera ID as identification information for identifying a camera is associated with camera position information and owner attribute information. The location information is, for example, latitude and longitude information. Also, the position information may include angle-of-view information of the camera and the like. The imaging control unit 342 acquires position information from each camera other than the fixed camera device 500 at a predetermined cycle, and registers the acquired position information in the camera DB 362 in association with the camera ID. As a result, the current position information of the cameras mounted on moving objects such as the vehicle 2, the terminal device 400, and the drone 600 can be more accurately grasped. The owner attribute information includes, for example, information such as the sex, age, frequently used applications (for example, photography applications) of the owner of the camera identified by the camera ID.

For example, when there is an image request from the user based on the position information of each camera acquired from the camera DB 362, the imaging control unit 342 refers to the camera DB 362 using the position information of the shooting target acquired together with the image request. , the camera IDs existing in the vicinity of the position of the object to be photographed (for example, within a predetermined distance from the position of the object to be photographed) are extracted. In addition to the position information, the imaging control unit 342 may refer to the owner attribute information of the camera DB 362 based on the attribute information included in the image request, and extract the camera ID that satisfies the attribute condition. Then, the imaging control unit 342 outputs a predetermined instruction to the extracted camera ID. The predetermined instruction is, for example, image acquisition instruction information. The imaging instruction information includes, for example, positional information and characteristic information of the imaging target. The image instruction information may also include, for example, an instruction to upload images captured by a plurality of cameras to the information providing server 300 . Further, the image instruction information may include an instruction to extract a captured image and upload it to the information providing apparatus when the distance from the imaging target is within a predetermined distance.

The image acquisition unit 344 acquires an image captured by each of the vehicle system 3, the terminal device 400, the fixed camera device 500, and the drone 600 that have transmitted the image acquisition instruction information, which are connected via the network NW. . The image acquisition unit 344 stores the acquired captured image in the image DB 364 . FIG. 7 is a diagram showing an example of the contents of the image DB 364. As shown in FIG. In the image DB 364, a camera ID is associated with a shooting date and time indicating the date and time when an image was taken, a captured image ID as identification information for identifying a captured image, and captured image data.

The extractor 346 extracts an image suitable for the user from the image DB 364 . For example, the extracting unit 346 acquires the feature information of the target person or object registered in the user DB 366 based on the identification information of the target person or object included in the image request from the user, and stores it in the image DB 364. Among the obtained images, an image containing an object having a high similarity to the feature information (an object having a degree of similarity equal to or higher than a threshold) is extracted as a provisional image. A provisional image is, for example, an image extracted by the extraction unit 346 and not edited by the editing unit 348 .

FIG. 8 is a diagram showing an example of the contents of the user DB 366. As shown in FIG. For example, the user DB 366 associates a user ID as identification information for identifying a user, an imaging target ID as identification information for identifying an imaging target, and feature information. For example, when the imaging target is a person, the feature information includes the outline of the face, the shape and arrangement of each part such as the eyes, nose, mouth, and ears, color information, and the like. When the imaging target is an object such as the vehicle 2, it is the shape, size, color, and other identification information of the object. For example, when the object to be photographed is the vehicle 2, the other identification information may include license plate information, vehicle type information, and the like. The extraction unit 346 may generate or update feature information by performing machine learning based on image data (correct data) obtained by imaging an imaging target, for example.

The editing unit 348 edits the temporary image or the like extracted by the extracting unit 346 to generate an edited image. An edited image is an image obtained by superimposing information on the date, shooting location, and the camera that took the image on the captured image, or by decorating the captured image. Also, the edited image may include an image group in which a plurality of images are arranged in a predetermined order such as time series. Also, the edited image may include an image obtained by masking a non-target person, a non-target object, or scenery included in the image.

The provisional image extracted by the extraction unit 346 and the edited image edited by the editing unit 348 are stored in the provided image DB 368 . FIG. 9 is a diagram showing an example of the contents of the provided image DB 368. As shown in FIG. In the provided image DB 368, the user ID is associated with a provisional image ID as identification information for identifying a provisional image, an edited image ID as identification information for identifying an edited image, a provision date and time, and a usage fee. The usage fee is a usage fee for each user calculated by the settlement unit 352 .

The information providing unit 350 provides the image extracted by the extracting unit 346 to the user who requested the image. For example, when an image request is received from the terminal device 400 of the user, the information providing unit 350 transmits the provisional image extracted by the extracting unit 346 to the terminal device 400 within a predetermined time after receiving the image request. Further, the information providing unit 350 transmits the edited image edited by the editing unit 348 to the terminal device 400 of the user after a predetermined time has elapsed. For example, if the vehicle 2 does not have a function for immediately transmitting an image captured by the camera 10, or if the image is captured in a parking lot or other location with poor communication environment, the captured image is immediately sent to the information providing server 300. Unable to send. Therefore, the information providing unit 350 provides the user with the provisional image extracted by the extracting unit 346 from among the images acquired by the image acquiring unit 344 within a predetermined period of time after receiving the image request. After the elapse, the edited image edited by the editing unit 348 is provided to the user. As a result, provisional images can be provided to users who cannot wait for a long time to acquire images. High image quality can also be provided. Note that the predetermined time may be a fixed time such as half a day or a whole day, or may be a time designated by the user. For example, when the user goes on a trip, the predetermined time may be the travel period.

Further, the information providing unit 350 may transmit only the image requested by the user, out of the provisional image or the edited image. Further, when the information providing unit 350 acquires the sound together with the image, the image and the sound may be associated with each other and provided to the user.

The settlement unit 352 calculates a usage fee for the user based on the image provided to the user, and stores the calculated usage fee in the provided image DB 368 in association with the user ID of the user who provided the image. The billing unit 352 may be billed according to the number of images or the like, or may be billed according to the type of image provided (for example, provisional image, edited image). Also, the settlement unit 352 derives a reward for the image provider who provided the captured image, and stores the derived reward information in the provider DB 370 . The image provider is, for example, the owner or manufacturer of the vehicle 2 that captured the captured image, the user of the terminal device 400, the administrator who manages the fixed camera device 500, the operating company which manages the drone 600, or the like. Remuneration information is information such as money, points, products, and services, for example.

FIG. 10 is a diagram showing an example of the contents of the provider DB 370. As shown in FIG. In the provider DB 370, the provider ID as identification information for identifying the provider is associated with the captured image ID, the adoption flag, and the remuneration information. The adoption flag is information indicating whether or not the image provided by the provider has been adopted as the provided image (provisional image or edited image). In the example of FIG. 10, the adoption flag is set to "1" when the image is adopted as the provided image, and the adoption flag is set to "0" when the image is not adopted as the provided image. Not limited. The settlement unit 352 makes the reward A larger when the captured image is adopted as the provided image than the reward B when it is not adopted. As a result, it is possible to increase the awareness of the image provider to take an image with a high reward. Further, even if the captured image is not adopted as the provided image, the settlement unit 352 gives some kind of reward for providing the image. As a result, the image provider can be more motivated to provide the captured image.

[Terminal device]
FIG. 11 is a configuration diagram of the terminal device 400 of the embodiment. The terminal device 400 is, for example, a terminal device that can be carried by a user, such as a smart phone, a tablet terminal, or a personal computer. The terminal device 400 includes, for example, a terminal-side communication unit 410, an input unit 420, a display 430, a speaker 440, a position acquisition unit 450, a terminal-side camera 460, an application execution unit 470, and an output control unit 480. , and a terminal-side storage unit 490 . The position acquisition unit 450, the application execution unit 470, and the output control unit 480 are implemented by, for example, a hardware processor such as a CPU executing a program (software). Also, some or all of these components may be realized by hardware (circuitry) such as LSI, ASIC, FPGA, GPU, etc., or by cooperation of software and hardware may be The above-described program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as an HDD or flash memory of the terminal device 400, or may be stored in a storage device such as a DVD, CD-ROM, or memory card. It is stored in a detachable storage medium, and may be installed in the storage device of the terminal device 400 by inserting the storage medium (non-transitory storage medium) into a drive device, card slot, or the like.

The terminal-side storage unit 490 may be realized by the above-described various storage devices, EEPROM, ROM, RAM, or the like. Information such as an image providing application 492, a provisional image 494, an edited image 496, and the like is stored in the terminal-side storage unit 490, for example.

The terminal-side communication unit 410 communicates with external devices such as the vehicle 2 and the information providing server 300 using networks such as cellular networks, Wi-Fi networks, Bluetooth, DSRC, LAN, WAN, and the Internet.

The input unit 420 receives input from the user U by operating various keys and buttons, for example. The display 430 is, for example, an LCD, an organic EL display, or the like. Input unit 420 may be configured integrally with display 430 as a touch panel. The display 430 displays various information in the image providing process in the embodiment under the control of the output control section 480 . The speaker 440 outputs a predetermined sound under the control of the output control section 480, for example.

The position acquisition unit 450 acquires the position information of the terminal device 400 using a GNSS receiver (not shown) built in the terminal device 400 and transmits the acquired position information to the information providing server 300 .

The terminal-side camera 460 is, for example, a digital camera using a solid-state imaging device such as CCD or CMOS. Terminal-side camera 460 captures an image of the direction in which terminal device 400 is directed by the user's operation of terminal device 400 . An image captured by the terminal-side camera 460 is transmitted (uploaded) to the information providing server 300 according to an instruction from the application execution unit 470 .

Application execution unit 470 is realized by executing image providing application 492 stored in terminal-side storage unit 490 . The image providing application 492 is, for example, an application program for communicating with the information providing server 300 via the network NW and acquiring image data stored in the information providing server 300 . Image providing application 492 causes display 430 to display an image request screen for setting image information to be provided by the user, and accepts setting information (image request) from the user. The image providing application 492 also transmits the received image request to the information providing server 300 and acquires an image (which may include sound) corresponding to the image request.

The application executing unit 470 also switches whether to execute the image providing application 492, downloads images (provisional images 494 and edited images 496) from the information providing server 300, and stores them in the terminal-side storage unit 490. , each image may be read out. The application execution unit 470 may also perform control to transmit the captured image of the terminal-side camera 460 to the information providing server 300 .

Output control unit 480 controls the content and display mode of an image to be displayed on display 430 and the content and output mode of audio to be output to speaker 440 according to instructions from application execution unit 470 . Further, the output control unit 480 may generate an inquiry image for asking the user whether or not to image the object to be imaged based on the control information from the information providing server 300 and display it on the display 430 .

[Fixed camera device]
FIG. 12 is a diagram showing a configuration diagram of the fixed camera device 500. As shown in FIG. The fixed camera device 500 is, for example, a camera such as a surveillance camera or a security camera that captures an image of the surroundings at a predetermined period or when an object is detected. The fixed camera device 500 includes, for example, a camera-side communication section 510, a camera 520, a microphone 530, an imaging control section 540, and a storage section 550.

The camera-side communication unit 510 communicates with external devices such as the vehicle 2 and the information providing server 300 using networks such as cellular networks, Wi-Fi networks, Bluetooth, DSRC, LAN, WAN, and the Internet.

The camera 520 is, for example, a digital camera using a solid-state imaging device such as CCD or CMOS. The camera 520 captures an image of the scenery in a fixed direction under the control of the imaging control section 540 . The microphone 530 collects sounds around the fixed camera device 500 . The microphone 530 may collect surrounding sounds at the timing when the imaging control section 540 captures an image of the surroundings with the camera 520 .

The imaging control unit 540 controls the timing of starting and ending imaging with the camera 520 . The imaging control unit 540 may, for example, start imaging with the camera 520 at a preset timing (for example, a preset time period or the timing at which the vehicle passes). Based on this, the camera 520 may start imaging. In addition, the imaging control unit 540 causes the storage unit 550 to store a captured image 552 captured by the camera 520, or causes the camera-side communication unit 510 to transmit the captured image 552 together with the camera ID to the information providing server 300. FIG. In addition, the imaging control unit 540 may store the sound collected by the microphone 530 together with the captured image 552 in the storage unit 550 or transmit it to the information providing server 300 .

The storage unit 550 is implemented by, for example, an HDD, flash memory, EEPROM, ROM, or RAM. Information such as a captured image 552 captured by the camera 520 is stored in the storage unit 550, for example.

[Drone]
FIG. 13 is a configuration diagram of the drone 600. As shown in FIG. The drone 600 includes, for example, a drone-side communication unit 610, a drone-side camera 620, a position acquisition unit 630, a sensor 640, a motor 650, a charging unit 660, a control unit 670, and a drone-side storage unit 680. Prepare.

The drone-side communication unit 610 is, for example, a communication interface for connecting to the network NW and directly communicating with the information providing server 300 . The drone-side communication unit 610 may include, for example, a NIC, and may perform wireless communication based on Wi-Fi, DSRC, Bluetooth, or other communication standards. A plurality of units may be prepared as the drone-side communication unit 610 according to the application.

The drone-side camera 620 is, for example, a digital camera using a solid-state imaging device such as CCD or CMOS. One or a plurality of drone-side cameras 620 are attached to arbitrary locations of the drone 600 . The drone-side camera 620 captures an image of the space in the direction in which the camera is installed at a predetermined timing under the control of the control unit 670 .

A position acquisition unit 630 acquires the position of the drone 600 using a GNSS receiver (not shown). In addition, the position acquisition unit 630 outputs the acquired position information to the control unit 670 or causes the drone-side communication unit 610 to transmit the information to the information providing server 300 . The sensor 640 is, for example, a magnetic sensor, a triaxial gyro sensor, a triaxial acceleration sensor, or the like. The content detected by sensor 640 is output to control unit 670 and the like.

Motor 650 includes a plurality of motors, and each motor is attached with a propulsion mechanism such as a propeller, for example. The charging unit 660 includes a rechargeable battery and supplies power to various parts of the drone 600 . Charging section 660 may also include a connection section for charging a rechargeable battery.

The control unit 670 is implemented, for example, by a processor such as a CPU executing a program (software) stored in the storage unit 290 . Also, one or both of these functional units may be implemented by hardware such as LSI, ASIC, FPGA, or GPU, or may be implemented by cooperation of software and hardware. The control unit 670 controls the motor so that the drone 600 flies according to the flight plan obtained from the information providing server 300 or the like based on the position information obtained by the position obtaining unit 630 and the contents detected by the sensor 640, for example. 650 is controlled. A flight plan includes, for example, flight path, altitude, speed, and flight mode. Flight modes include climb, straight flight, descent, hovering, side-by-side, and the like. Parallel flight is to follow and fly in accordance with the movement of the object to be imaged.

Also, based on the image acquisition instruction information received from the information providing server 300, the control unit 670 controls the drone-side camera 620 so as to capture an image of an object within a predetermined distance or within the angle of view of the drone-side camera 620. do. For example, the control unit 670 analyzes the image captured by the drone-side camera 620 to obtain feature information of an object, etc., and the feature information of the imaging target included in the imaging instruction information from the information providing server 300. If the degree of similarity with is high, the image captured by the drone-side camera 620 is acquired. In addition, the control unit 670 causes the drone-side storage unit 680 to store an image captured by the drone-side camera 620, or causes the drone-side communication unit 610 to transmit (upload) the captured image to the information providing server 300 together with the camera ID. Further, the control unit 670 may identify an imaging target by communicating with the imaging target (for example, the vehicle 2 or the terminal device) using the drone-side communication unit 610, and may capture an image of the specified imaging target.

The drone-side storage unit 680 is implemented by, for example, an HDD, flash memory, EEPROM, ROM, RAM, or the like. Information such as a captured image 682 captured by the drone-side camera 620 is stored in the drone-side storage unit 680, for example.

[Specific example of information provision system]
A specific example of the information providing system 1 of the embodiment will be described below. FIG. 14 is a diagram for explaining a specific example of the information providing system 1 of the embodiment. In the example of FIG. 14, an intersection CR1 and roads R1 to R4 connected to the intersection CR1 are shown. The host vehicle M travels on the road R1 at a speed VM in the direction of the intersection CR1 (Y direction in the figure), and the other vehicle m1 travels on the road R1 at a speed Vm1 in the direction away from the intersection CR1 (the -Y direction in the figure). shall be Also, in the example of FIG. 14, it is assumed that fixed camera devices 500-1 and 500-2 are installed near the road. It is also assumed that a fixed camera device 500-3 is installed in the shop near the intersection CR1. Also, in the example of FIG. 14, it is assumed that the drone 600 is flying near the intersection CR1. The information providing server 300 acquires images captured by the cameras of the own vehicle M, the other vehicle m1, the fixed camera devices 500-1 to 500-3, and the drone 600. FIG. The camera 10 mounted on the own vehicle M and the vehicle interior camera 90 are an example of the "first imaging section". In addition, cameras 520 of fixed camera devices 500-1 to 500-3 are an example of a "second imaging section". Also, the “second imaging unit” may include the camera 10 of the other vehicle m1, the vehicle interior camera, and the drone-side camera 620 of the drone 600 . Also, in the following description, the terminal-side camera 460 of the terminal device 400 used by someone other than the user U1 (for example, the user U2) is an example of the "third imaging unit." The image captured by the first imaging unit is an example of the "first image", and the image captured by the second imaging unit is an example of the "second image", and is captured by the third imaging unit. The image displayed is an example of a "third image".

An example of a situation where user U1 using terminal device 400-1 uses the information providing service according to the present embodiment will be described below. First, user U1 activates image providing application 492 installed in terminal device 400-1. Image providing application 492 causes display 430 to display image IM1 including the image request after being activated. FIG. 15 is a diagram showing an example of an image IM1 including an image request. The image IM1 includes, for example, an index display area A11, an image setting area A12, and a GUI (Graphical User Interface) switch display area A13. The headline display area A11 displays information for prompting the user to input settings related to the image request. In the example of FIG. 15, text information such as "<Image request screen> Please enter the image information to be provided." is displayed in the headline display area A11.

The image setting area A12 displays combo boxes CB11 to CB16 for setting items such as information on the imaging level, information on the object to be imaged, presence/absence of image, presence/absence of sound, attributes of the camera owner, and the like. The imaging level is level information set according to, for example, whether or not transmission (uploading) of a captured image to the information providing server 300 is required, whether or not image provision is required, and imaging conditions.

FIG. 16 is a diagram for explaining the contents of the shooting level information. In the shooting level information, shooting levels are associated with shooting contents. The imaging content includes control content for not uploading images captured by a plurality of cameras to the information providing server 300 and control content for uploading an image captured outside a predetermined area to the information providing server 300 . In addition, the imaging content includes control content that permits uploading of images captured by at least some of the “first imaging unit”, “second imaging unit”, and “third imaging unit”. may be included. In addition, the imaging content may include attribute information of the owner of the “third imaging unit” and control content for permitting upload from the third imaging unit corresponding to the attribute.

In the example of FIG. 16, "level 1" is level information indicating that the captured image is neither uploaded nor provided to the user. “Level 2” is level information indicating that uploading of captured images and provision of information outside the predetermined area are permitted (in other words, uploading of captured images and provision of information within the predetermined area are not permitted). “Level 3” is level information indicating permission to upload captured images and provide images. “Level 4” is level information that permits uploading of images captured by cameras other than terminal devices (first imaging unit, second imaging unit) and provision of images of people other than the user. "Level 5" is level information indicating permission to upload and provide an image captured by a terminal device of another person who satisfies the owner attribute of the camera. The shooting level information shown in FIG. 16 may be stored in the terminal-side storage unit 490 or may be stored in the server-side storage unit 360, for example. By selecting "level 1" from the combo box CB11, the user U1 can, for example, suppress photographing and image provision in a situation or environment not intended by the user U1. In addition, by selecting “level 2”, the user U1 can, for example, suppress photography and provision of images within a specific range such as inside the home, or accept provision of images only at tourist spots. In addition, by selecting “Level 3”, the user U1 can acquire, for example, his or her actions and events during the day, the situation during a trip, etc., from images captured by one or more external cameras or the like. . In addition, by selecting "level 4", the user U1 can acquire an image captured by a vehicle camera or a fixed camera set in the city as a standard setting, for example. In addition, by selecting “level 5”, the user U1 can obtain images desired by the user according to the sex and age of the camera owner, for example.

When "level 2" to "level 5" are set as the shooting level, the combo box CB12 for setting the imaging target and whether or not provisional images and edited images are desired are selected. Combo boxes CB13 and CB14, a combo box CB15 for selecting whether or not voice is desired, a combo box CB16 for selecting an attribute, and the like are displayed. The imaging target selectable by the combo box CB12 is, for example, a target registered in the user DB 366 in advance. Sound is sound before and after capturing an image. The sound is, for example, the sound inside or outside the vehicle collected by a microphone or the like included in the HMI 30 or the ambient sound collected by the fixed camera device 500 . By inputting setting conditions in each box, it is possible to execute an image providing service that meets the intention of the user U1. In the example of FIG. 15, "level 3" is selected as the imaging level, "vehicle (own vehicle M)" is selected as the object, and "desired" is set for the provisional image, edited image, and sound. , and the attribute "20's woman" is selected. For example, women in their twenties are expected to be accustomed to photographing using their own terminal device 400 compared to other age groups and genders, so "women in their twenties" is set as the attribute of the camera owner. By doing so, a higher quality image can be obtained.

The GUI switch display area A13 displays, for example, an icon for receiving an instruction from the user U1. Icons include, for example, an icon IC11 drawn with the characters "OK" and an icon IC12 with the characters "CANCEL" drawn. When the icon IC11 is selected by the user U1, the image providing application 492 transmits the information (image request) input in the image setting area A12 to the information providing server 300 via the terminal-side communication unit 410, and The display of the setting screen image IM1 is terminated. Further, when the icon IC12 is selected by the user U1, the content set in the image setting area A12 is canceled and the display of the image IM1 ends.

The information providing server 300 acquires information set in the image setting area A12 by the terminal device 400, and performs various controls related to image provision based on the acquired information. For example, when the imaging level is “level 1”, the server-side control unit 340 stops acquisition of images from each camera or suppresses provision of images to the terminal device 400 . In addition, when the imaging level is "level 2" to "level 5", the camera ID is obtained by referring to the camera DB 362 under conditions corresponding to each level, and image obtaining instruction information is transmitted to the obtained camera ID. . The image acquisition instruction information includes, in addition to the position and characteristic information of the object to be imaged, the estimated time until the distance to the object to be imaged is within a predetermined distance (within the distance that is predicted to be photographable). you can Each device that has received the imaging instruction captures an image of the imaging target when the imaging target approaches within a predetermined distance.

FIG. 17 is a diagram for explaining how the own vehicle M is imaged. The example of FIG. 17 shows a scene in which own vehicle M set as an imaging target is imaged by terminal-side camera 460 of terminal device 400-2, camera 520 of fixed camera device 500-1, and drone-side camera 620. . Terminal device 400-2, fixed camera device 500-1, and drone 600 acquire position information of own vehicle M based on image acquisition instruction information from information providing server 300, and based on the acquired position information, It acquires whether or not the own vehicle M is approaching, and images the own vehicle M when approaching.

It should be noted that output control unit 480 of terminal device 400-2 may cause display 430 to display an image for asking user U2 whether or not to perform imaging when a photographing instruction is received from information providing server 300. good. FIG. 18 is a diagram showing an example of an image IM2 for inquiring whether or not to shoot terminal device 400-2. The image IM2 includes, for example, an index display area A21, an inquiry content display area A22, and a GUI switch display area A23. Information (for example, “inquiry”) indicating an inquiry about an imaging request from the information providing server 300 is displayed in the headline display area A21.

Inquiry content display area A22 displays information for asking user U2 of terminal device 400-2 whether or not to photograph the imaging target. In the example of FIG. 18, the shooting instruction display area A22 displays text information such as "The imaging request has been received. There are three minutes left until the imaging target approaches. Do you want to shoot?"

For example, an icon for receiving an instruction from the user U2 is displayed in the GUI switch display area A23. Icons include, for example, an icon IC21 in which characters "YES" are drawn and an icon IC22 in which characters "NO" are drawn. When the icon IC21 is selected by the user U2, the output control unit 480 generates an image IM3 notifying the feature information of the imaging target acquired from the information providing server 300, and causes the display 430 to display the generated image. Further, when the icon IC22 is selected by the user U2, the output control unit 480 terminates the display of the image IM2.

FIG. 19 is a diagram showing an example of an image IM3 including feature information. The image IM3 includes, for example, an index display area A31 and an imaging target display area A32. Information (for example, “Notification”) indicating notification of the imaging target is displayed in the heading display area A31. Information about the imaging target is displayed in the imaging target display area A32. Image providing application 492 displays image IM3 when the distance between terminal device 400-2 and the object to be imaged (self-vehicle M) is within a predetermined distance, or when it is predicted that they will pass within a predetermined time. 430 may be displayed. In the example of FIG. 19, in the imaging target display area A32, ``The imaging target will pass soon. : ○○○” and “Color: Red” are displayed. Further, the image providing application 492 may activate the terminal-side camera 460 to prepare for photographing while displaying the image IM3 or after displaying the image IM3 for a predetermined period of time. This makes it easier for the user U2 to photograph the imaging target.

Terminal-side camera 460 of terminal device 400-2 used by user U2, camera 520 of fixed camera device 500-1, and drone-side camera 620 each communicate with own vehicle M, which is an object to be imaged. A distance to the vehicle M may be obtained.

Here, the image acquisition control unit 180 of the own vehicle M may perform control to decelerate the speed of the own vehicle M when the own vehicle M approaches the camera that captures the image. Information about the position of the camera that captures the vehicle M may be obtained from the information providing server 300, or may be obtained through direct communication with each device. In the example of FIG. 17, the automatic driving control device 100 of the own vehicle M is controlled by the image acquisition control unit 180, the terminal device 400-2, the fixed camera device 500-1, and the drone 600 that capture the image of the own vehicle M is equal to or less than a predetermined distance, operation control is executed to reduce the speed from VM to VM# (VM>VM#). The amount of deceleration may be a fixed value, or may be set based on the current speed of surrounding vehicles and the road environment. In addition, the automatic driving control device 100, for example, under the control of the image acquisition control unit 180, the fixed camera device 500-1 and the terminal device 400-2 to travel in a position and route that is easy to capture the steering (steering angle). may be controlled.

Images captured by the terminal-side camera 460 of the terminal device 400-2, the camera 520 of the fixed camera device 500-1, and the drone-side camera 620 are transmitted to the information providing server 300. FIG. The image acquisition unit 344 of the information providing server 300 acquires images captured by the terminal-side camera 460 of the terminal device 400-2, the camera 520 of the fixed camera device 500-1, and the drone-side camera 620, and stores the acquired images in the image DB 364. Memorize. The extraction unit 346 extracts an image including the own vehicle M from the images registered in the image DB 364 . FIG. 20 is a diagram showing an example of an image extracted by the extraction unit 346. As shown in FIG. In the example of FIG. 20, images IM41 and IM42 including own vehicle M are captured by terminal-side camera 460 of terminal device 400-2, camera 520 of fixed camera device 500-1, and drone-side camera 620, respectively. , IM43 . . .

When information requesting a provisional image is included in the image request content from terminal device 400-1, information providing unit 350 transmits the image extracted by extraction unit 346 to terminal device 400-1 as a provisional image. . Thereby, the captured image can be provided to the user U1 more quickly.

In the above-described example, the case where the subject vehicle M is imaged by the surrounding cameras has been described, but the subject to be imaged may be imaged by the camera 10 of the subject vehicle M. FIG. 21 is a diagram showing an example of an image IM5 including a user U2 whose own vehicle M is captured. In the example of FIG. 21, it is assumed that the imaging target is the user U2. When capturing an image of the user U2 based on the image acquisition instruction information from the information providing server 300, the image acquisition control unit 180 of the own vehicle M communicates with the terminal device 400-2 of the user U2, and the image acquisition control unit 180 of the own vehicle M An image IM5 captured when the distance from the position of the vehicle M to the position of the terminal device 400 reaches a predetermined distance is acquired from the images constantly captured by the camera 10 for recognizing the surroundings. The image IM5 is transmitted and uploaded to the information providing server 300 .

Further, when the user U2 desires voice, the image acquisition control unit 180 collects the voice when the distance from the terminal device 400-2 reaches a predetermined distance using the microphone included in the HMI 30, and collects the voice. The voice information and the captured image are transmitted (uploaded) to the information providing server 300 .

In addition, when the voice around the user U2 is collected by the microphone mounted on the own vehicle M, the sound around the user U2 may not be collected accurately due to the engine sound of the own vehicle M. Therefore, when the driving force output device 200 is operating in the first drive mode, the image acquisition control unit 180 performs control so as to switch to the operation in the second drive mode. As a result, when the microphone mounted on the own vehicle M collects the voice near the position where the user U2 is present, the noise of the traveling drive system of the own vehicle M is suppressed, and the sound around the user U2 ( In the example of FIG. 21, "Please take a picture!") can be collected.

In addition, the image acquisition control unit 180 may cause the vehicle interior camera 90 to capture an image of the user U1, and transmit (upload) the captured image to the information providing server 300 together with the camera ID. FIG. 22 is a diagram showing an example of an image IM6 including the user U1 seated in the driver's seat DS of the vehicle M. As shown in FIG. For example, the image acquisition control unit 180 analyzes an image including the user U1 captured by the in-vehicle camera 90, extracts an image of the user U1 having a specific facial expression (for example, a smile or a surprised face), and extracts the extracted image. The image is transmitted to the information providing server 300 . In addition, the image acquisition control unit 180 acquires the position information of the own vehicle M, and the image of the user U1 captured by the vehicle interior camera 90 when the own vehicle M arrives at a specific position (for example, a tourist spot or a view spot). image may be transmitted to the information providing server 300 . Accordingly, the information providing server 300 can provide the user U1 with images captured by various cameras.

The information providing server 300 transmits the provisional image to the terminal device 400-1 of the user U1 if it is within a predetermined time from the image capturing request from the user, among the images captured by each camera described above, and edits the image after a predetermined time. The edited image edited by the unit 348 is transmitted to the terminal device 400-1. For example, when the image request from the terminal device 400-1 includes information requesting an edited image, the information providing unit 350 edits and edits at least one image that satisfies a predetermined condition from the provisional image. The image is transmitted to the terminal device 400-1.

For example, the editing unit 348 selects predetermined conditions from among the plurality of temporary images IM41, IM42, IM43, . . . shown in the example of FIG. Extract the image that satisfies. Predetermined conditions include, for example, "the subject is photographed from the front", "the image includes a landscape, famous place, or feature that identifies the location", and "the facial expression of the subject is smiling." There is. Editing unit 348 also generates edited image IM41# by adding comment information CM1 to extracted image IM41, and transmits edited image IM41# to terminal device 400-1. The comment information CM1 includes, for example, the date and time of photography, the location of photography, and information about the camera that took the photograph. As a result, the user U1 can obtain not only images captured by himself/herself, but also images captured by various cameras around the user U1.

Also, the editing unit 348 may edit the image album by arranging the images captured by a plurality of cameras in a predetermined order. FIG. 23 is a diagram showing an example of the image album AL1 edited by the editing unit 348. As shown in FIG. In the example of FIG. 23, an image album AL1 showing a day of user U1 is edited using images captured by a plurality of cameras. Images IM71 and IM75 are images captured by cameras installed in the home. Image IM72 is a captured image of fixed camera device 500 installed on the way to work or a vehicle. Images IM73 and IM74 are images captured by a camera provided in user U1's company. The image album AL1 may include audio information acquired together with the images. As shown in FIG. 23, the editing unit 348 generates an image album AL1 by arranging images captured by a plurality of cameras in chronological order, so that the daily behavior of the user U1 can be viewed by the user U1 himself or by the user U1. It is possible to make it easier for the administrator of the In addition, the editing unit 348 generates an image album by arranging, for example, in chronological order images taken at various points along a route to a destination set for a family trip, drive, etc., so that the user can view the album. It is possible to make it easier to recall the memories of the user.

Also, the editing unit 348 may edit the image album, for example, based on the user's past captured image history stored over a long period of time. FIG. 24 is a diagram showing an example of the image album AL2 edited by the editing unit 348. As shown in FIG. The example of FIG. 24 shows an image album AL2 in which captured images IM81 to IM85 of user U2 aged 10, 20, 40, 50, and 70 are arranged in chronological order. The predetermined order is not limited to chronological order, and may be an order arranged based on a predetermined reference value such as size order, color order, height order, or the like. By providing the user U2 with the image album AL2 as shown in FIGS. 23 and 24, the editing unit 348 can provide image services with higher added value.

[Processing sequence]
FIG. 25 is a sequence diagram showing an example of the processing flow of the information providing system 1 of the embodiment. In the example of FIG. 25, an example will be described in which the information providing server 300 provides an image in response to an image providing request from the terminal device 400-1 used by the user U1. Also, in the example of FIG. 25, images are acquired from the fixed camera device 500, the other vehicle m, the terminal device 400-2, and the drone 600. FIG.

The terminal device 400-1 receives shooting setting information from the shooting setting screen IM1 by the operation of the user U1 (step S100), and transmits the accepted shooting setting information to the information providing server 300 (step S300). Based on the shooting setting information received from the terminal device 400-1, the information providing server 300 extracts a camera for shooting from a plurality of cameras registered in advance in the camera DB 362 (step S104). In the example of FIG. 25, the fixed camera device 500, the other vehicle m, the terminal device 400-2, and the drone 600 are extracted. The information providing server 300 transmits an image acquisition instruction to each of the fixed camera device 500, the other vehicle m, the terminal device 400-2, and the drone 600 (steps S106 to S112).

Based on the image acquisition instruction from the information providing server 300, the fixed camera device 500 acquires a captured image when the imaging target set by the imaging setting information approaches within a predetermined distance (step S114). The other vehicle m acquires a captured image when the imaging target approaches within a predetermined distance based on the image acquisition instruction from the information providing server 300 (step S116). Based on the image acquisition instruction from the information providing server 300, the terminal device 400-2 captures the image captured by the user U2 when the image capturing target approaches within a predetermined distance when the user U2 permits capturing. An image is acquired (step S118). The drone 600 takes an image when the imaging target approaches within a predetermined distance based on the image acquisition instruction from the information providing server 300 (step S120). Each of the fixed camera device 500, the other vehicle m, the terminal device 400-2, and the drone 600 transmits (uploads) captured images to the information providing server (steps S122 to S128). Note that the fixed camera device 500, the other vehicle m, the terminal device 400-2, and the drone 600 capture images at their respective image capturing timings, and after the images are captured, they are ready to communicate with the information providing server 300, or at a predetermined timing. , the captured image is transmitted (uploaded) to the information providing server 300 .

The information providing server 300 stores captured images obtained from the fixed camera device 500, the other vehicle m, the terminal device 400-2, and the drone 600 in the server-side storage unit 360 (step S130). Next, the information providing server 300 extracts the image of the imaging target from the server-side storage section 360 as a provisional image (step S132), and transmits the extracted image to the terminal device 400-1 (step S134).

Terminal device 400-1 displays the provisional image on display 430 (step S136). Further, information providing server 300 edits the image to be provided to user U1 after a predetermined period of time has passed since the image request was received from terminal device 400-1 (step S138), and sends the edited image to terminal device 400-1. (step S140). Terminal device 400-1 displays the edited image on display 430 (step S142). This completes the processing of this sequence.

According to the embodiments described above, the information providing server 300 can provide the user with more appropriate image information. Specifically, according to the embodiment, in addition to providing an image captured by the user, the user cannot capture the image by using various cameras such as a camera mounted on the vehicle and a camera installed at a predetermined position. Images captured from various angles and positions can be acquired. Further, according to the embodiment, by providing the provisional version of the image to the user when the image requested by the user is extracted, the image can be provided without making the user wait too long. In addition, by presenting the edited image as a complete version to the user again after the predetermined time has elapsed, even if part of the captured image has not been acquired in the provisional version extraction stage, the An edited image that meets the user's intention can be provided.

Further, according to the embodiment, when an object to be imaged is captured by a vehicle, a better captured image can be captured by adjusting the speed of the vehicle or adjusting the steering of the vehicle. Also, in the case of collecting sound near the object to be imaged, the sound near the object to be imaged can be collected more clearly by switching to a driving mode in which the noise due to the running driving force is reduced.

Further, according to the embodiment, when an imaging request is made in the information providing system, by setting the imaging level, an image can be provided at the timing intended by the imaging target. Therefore, for example, it is possible to suppress the provision of images in daily life and receive the provision of images only when there is a special event such as a family trip. In addition, by changing the imaging level, it is possible to perform control such that an image is captured when an action history or the like is desired.

[Modification]
In the information providing system 1 described above, when an object to be imaged approaches, the image is taken by a camera existing in the vicinity of the object. may be guided to FIG. 26 is a diagram for explaining guiding of an imaging target. FIG. 26 shows a scene in which the own vehicle M is parked in the parking lot PA.

In this scene, the other vehicle m2 performs inter-vehicle communication with the own vehicle M, and transmits a request signal to the own vehicle M so that the own vehicle M parks at the position P2 in front of the parking position P1 of the other vehicle m2. . The automatic driving control device 100 of the own vehicle M executes driving control for parking the own vehicle M at the position P2 when it is determined that the parking is possible at the position P2. As a result, the other vehicle m2 can take a close-up image of the vehicle M being parked by the mounted camera 10 .

Further, in the above-described embodiment, the case where the own vehicle M is being driven automatically has been described as an example, but other information may be provided to the vehicle when it is being driven manually. For example, in the above-described example, when the own vehicle M passes near the camera photographed by the own vehicle M, the speed control and the steering control of the own vehicle M are performed automatically. If it is in the middle, a message prompting speed control (for example, deceleration) or steering control may be displayed on the display device of the HMI 30, or an instruction to perform voice output control may be transmitted to the vehicle. In addition, when manual driving is being executed and voice is collected by the microphone of the HMI 30, the information providing server 300 notifies the vehicle driving mode to be switched from the first driving mode to the second driving mode. may be transmitted to the vehicle to display on the display device of the HMI 30 . Further, as shown in FIG. 26 described above, when the vehicle is parked at position P2, control may be performed to display information about the parking position on the display device. As a result, when the vehicle 2 is manually driven, it is possible to request the driver to drive so that images can be captured easily. As a result, a more suitable image can be obtained.

[Hardware configuration]
FIG. 27 is a diagram showing an example of the hardware configuration of the information providing server 300 of the embodiment. As shown, the information providing server 300 includes a communication controller 300-1, a CPU 300-2, a RAM 300-3 used as working memory, a ROM 300-4 for storing boot programs and the like, and a storage device 300 such as a flash memory or HDD. -5, drive device 300-6, etc. are interconnected by an internal bus or a dedicated communication line. The communication controller 300-1 communicates with components other than the information providing server 300. FIG. A program 300-5a executed by the CPU 300-2 is stored in the storage device 300-5. This program is expanded in RAM 300-3 by a DMA (Direct Memory Access) controller (not shown) or the like and executed by CPU 300-2. Thereby, part or all of each functional configuration of the information providing server 300 is realized.

The embodiment described above can be expressed as follows.
a storage device storing a program;
a hardware processor;
By the hardware processor executing the program stored in the storage device,
According to the content of the image received from the terminal device used by the user, based on the positional information of the object to be imaged, a first imaging unit mounted on the vehicle, a second imaging unit installed at a predetermined position, and other persons other than the user giving a predetermined instruction to a plurality of imaging units including a third imaging unit used by a person;
Acquiring an image including an imaging target captured by the first imaging unit, the second imaging unit, and the third imaging unit according to the predetermined instruction,
providing the terminal device with information including an image that satisfies the imaging content acquired by the image acquisition unit;
An information providing device configured to:

As described above, the mode for carrying out the present invention has been described using the embodiments, but the present invention is not limited to such embodiments at all, and various modifications and replacements can be made without departing from the scope of the present invention. can be added.

DESCRIPTION OF SYMBOLS 1... Information provision system 2... Vehicle 3... Vehicle system 10... Camera 12... Radar apparatus 14... Finder 16... Object recognition apparatus 20... Communication apparatus 30... HMI 40... Vehicle sensor 50... Navigation device 60 MPU 80 Driving operator 100 Automatic driving control device 120 First control unit 130 Recognition unit 140 Action plan generation unit 160 Second control unit 200 Running drive Force output device 210 Brake device 220 Steering device 300 Information providing server 310 Server communication unit 320 Input unit 330 Output unit 340 Server control unit 342 Imaging control unit 344... Image acquisition unit, 346... Extraction unit, 348... Editing unit, 350... Information providing unit, 352... Payment unit, 360... Server side storage unit, 400... Terminal device, 500... Fixed camera device, 600... Drone

Claims (6)

According to the content of the image received from the terminal device used by the user, based on the positional information of the object to be imaged, a first imaging unit mounted on the vehicle, a second imaging unit installed at a predetermined position, and other persons other than the user an imaging control unit that gives a predetermined instruction to a plurality of imaging units including a third imaging unit used by a person;
an image acquisition unit configured to acquire an image including an imaging target captured by the first imaging unit, the second imaging unit, and the third imaging unit according to a predetermined instruction from the imaging control unit;
an information providing unit that provides information including an image that satisfies the captured image acquired by the image acquisition unit to the terminal device ;
The content of the imaging includes a permission level for uploading images captured by the first imaging unit, the second imaging unit, and the third imaging unit and providing the images to the user, and the third imaging unit. and information about the attributes of the other person who uses the department,
The predetermined instruction includes an instruction to upload an image captured by an imaging unit extracted from the plurality of imaging units according to the permission level and the attribute to an information providing device.
Information provider. The permission level includes a level that permits uploading of an image taken outside a predetermined area to the information providing device and provision of the image to the user .
The information providing device according to claim 1. Among the first imaging unit, the second imaging unit, and the third imaging unit, the permission level is defined as uploading an image captured by the first imaging unit and the second imaging unit and uploading the image to the user. including levels that allow the provision of
3. The information providing device according to claim 1 or 2 . The permission level is defined as uploading to the information providing apparatus an image captured by the third imaging unit used by the other person who satisfies the attribute condition set by the user, and uploading the image to the user. including levels that allow the provision of
The information providing device according to any one of claims 1 to 3. The information providing device
According to the content of the image received from the terminal device used by the user, based on the positional information of the object to be imaged, a first imaging unit mounted on the vehicle, a second imaging unit installed at a predetermined position, and other persons other than the user giving a predetermined instruction to a plurality of imaging units including a third imaging unit used by a person;
Acquiring an image including an imaging target captured by the first imaging unit, the second imaging unit, and the third imaging unit according to the predetermined instruction,
providing information including the acquired image to the terminal device ;
The content of the imaging includes a permission level for uploading images captured by the first imaging unit, the second imaging unit, and the third imaging unit and providing the images to the user, and the third imaging unit. and information about the attributes of the other person who uses the department,
The predetermined instruction includes an instruction to upload an image captured by an imaging unit extracted from the plurality of imaging units according to the permission level and the attribute to an information providing device.
How to provide information. to the information providing device,
According to the content of the image received from the terminal device used by the user, based on the positional information of the object to be imaged, a first imaging unit mounted on the vehicle, a second imaging unit installed at a predetermined position, and other persons other than the user giving a predetermined instruction to a plurality of imaging units including a third imaging unit used by a person;
Acquiring an image including an imaging target imaged by the first imaging unit, the second imaging unit, and the third imaging unit according to the predetermined instruction,
causing the terminal device to provide information including the acquired image ;
The content of the imaging includes a permission level for uploading images captured by the first imaging unit, the second imaging unit, and the third imaging unit and providing the images to the user, and the third imaging unit. and information about the attributes of the other person who uses the department,
The predetermined instruction includes an instruction to upload an image captured by an imaging unit extracted from the plurality of imaging units according to the permission level and the attribute to an information providing device.
program.

Images