JP2020127175A - Image acquisition system, image acquisition method, and program - Google Patents

Abstract

To acquire information of higher value for a user.SOLUTION: An image acquisition system (1) comprises a first acquisition unit (124) for acquiring positional information of a mobile unit, a second acquisition unit (126) for acquiring the emotion of the user of the mobile unit, and a third acquisition unit (128) for acquiring a specific image captured when the positional information is included in an imaging area set in advance and the emotion of the user acquired by the second acquisition unit coincides with designated emotion set in advance.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image acquisition system, an image acquisition method, and a program.

Conventionally, a sensor that detects an electrical signal generated in the body of an individual, an extraction circuit that extracts an electrical signal that is greater than a threshold value among the electrical signals as an emotional undulation signal, and a body that is integrated with the human body of the individual An apparatus including a camera mounted on a moving body is disclosed (for example, refer to Patent Document 1). This camera captures a lifetime cross section corresponding to the external world of the human body at a shooting time corresponding to the time of generation of the emotional undulation signal.

JP, 2005-260892, A

However, the above-described device may not be able to acquire information that is valuable to the user.

The present invention has been made in consideration of such circumstances, and one of the objects is to provide an image acquisition system, an image acquisition method, and a program that can acquire information of high value to a user. To do.

The image acquisition system, the image acquisition method, and the program according to the present invention have the following configurations.
(1): An image acquisition system according to an aspect of the present invention includes a first acquisition unit that acquires position information of a moving body, a second acquisition unit that acquires emotion of a user of the moving body, and the position information. And a third acquisition unit that acquires a specific image captured when the emotion of the user acquired by the second acquisition unit matches the preset specified emotion, Equipped with.

(2): In the aspect of (1) above, the second acquisition unit acquires the emotion of the user when the position information acquired by the first acquisition unit is included in the imaging area. Is what starts.

(3): In the aspect of (1) or (2), the third acquisition unit acquires the specific image captured by the first imaging unit that captures the user.

(4): In the aspect of (3), the first imaging unit is provided around a moving body on which the user gets on or a road on which the moving body passes.

(5): In any one of the above aspects (1) to (4), the third acquisition unit is imaged by a second imaging unit that is provided in the moving body and that images the surrounding environment of the user. A specific image is acquired.

(6): In any one of the above aspects (1) to (5), an information providing unit for providing the specific image to the user is further provided.

(7): In the aspect of (6), the third acquisition unit is provided on the first specific image captured by the first image capturing unit that captures the user and the moving body, and the user's The second specific image captured by the second image capturing unit that captures the surrounding environment is acquired, and the information providing unit provides the user with information in which the first specific image and the second specific image are associated with each other. It is a thing.

(8): In the aspect of (7), the third acquisition unit further acquires a third specific image captured by an imaging unit that images the moving body in the imaging area, and the information providing unit, Information in which the first specific image, the second specific image, and the third specific image are associated with each other is provided to the user.

(9): In any one of the above aspects (1) to (8), information provision for providing the user with information in which the recommended imaging area or the imaging area designated by the user and the map information are associated with each other. It further comprises a section.

(10): In the above aspects (1) to (9), information in which the area in which the specific image is acquired and map information are associated with each other, the area in which the specific image is acquired, the specific image, and the map It further comprises an information providing unit for providing the user with information associated with information or information associated with the area where the specific image is acquired and the movement trajectory of the moving body.

(11): In an image acquisition method according to an aspect of the present invention, a computer acquires position information of a moving body, acquires emotions of a user of the moving body, and the position information is set in advance in an imaging area. And a specific image captured when the acquired emotion of the user matches a preset designated emotion.

(12): A program according to an aspect of the present invention causes a computer to acquire position information of a moving body, acquire emotions of a user of the moving body, and the position information is included in a preset imaging area. In addition, the program is a program for acquiring a specific image captured when the acquired emotion of the user matches a preset designated emotion.

According to (1), (6), (7), (8), (11), and (12), use in which the position information is included in the preset imaging area and which is acquired by the second acquisition unit By obtaining the specific image captured when the emotion of the person matches the preset designated emotion, it is possible to obtain information of high value to the user.

According to (2), since the estimation of emotion is acquired when it is included in the imaging area, unnecessary processing is suppressed.

According to (3) or (4), since an image including the user is acquired, it is possible to acquire information of higher value to the user.

According to (5), since an image of the surrounding landscape is obtained, it is possible to obtain more valuable information for the user.

According to (9), since the information in which the imaging area and the map information are associated with each other is provided to the user, the user can easily recognize the imaging area.

According to (10), the information in which the area where the specific image is acquired and the map information are associated with each other, and the information in which the area where the specific image is acquired and the movement trajectory of the moving body are associated with each other are provided to the user. By providing, the user can easily recognize the area where the specific image is acquired.

It is a figure which shows an example of a structure of the image acquisition system 1. 3 is a diagram showing an example of a functional configuration of the center server device 10. FIG. It is a figure which shows an example of a part of functional structure contained in a vehicle. FIG. 7 is a sequence diagram showing processing executed by the image acquisition system 1. It is a figure which shows an example of the content of the imaging process information 34. It is a figure which shows an example of the image IM1 displayed on the display part (touch panel) of the terminal device 2. It is a figure which shows an example of the scene where an automatic imaging process is performed. It is a figure for demonstrating the process when an image is acquired after the fact. It is a figure which shows an example of the image IM2 displayed on the terminal device 2. It is a figure which shows an example of the image IM3 displayed on a display part, when the icon which shows a predetermined desired imaging area is operated. It is a figure which shows an example of the image IM4 displayed on the terminal device 2. It is a flow chart which shows an example of a flow of processing performed by control device 120 of a 2nd embodiment. It is a figure showing an example of composition of image acquisition system 1A of a 3rd embodiment. It is a sequence diagram which shows an example of the flow of the process performed by the image acquisition system 1A of 3rd Embodiment. It is a figure for demonstrating the process which the imaged image is extracted. It is a figure which shows an example of the scene where an automatic imaging process is performed. It is a figure which shows an example of the image IM5 displayed on the display part of the terminal device 2 of 3rd Embodiment. It is a figure which shows an example of the hardware constitutions of the control apparatus 120 of embodiment.

Hereinafter, embodiments of an image acquisition system, an image acquisition method, and a program of the present invention will be described with reference to the drawings. In the following description, in the image acquisition system, an example of capturing an image of a user who rides on a four-wheel vehicle will be described. However, the present invention is not limited to this, and when capturing an image of a user who rides on a moving body other than a four-wheel vehicle. An image acquisition system may be applied to. A moving body other than a vehicle is a vehicle such as a two-wheel vehicle or a three-wheel vehicle, or a bicycle. Further, the moving body may be a terminal device (for example, a smartphone) that moves together with the occupant.

<First Embodiment>
FIG. 1 is a diagram illustrating an example of the configuration of the image acquisition system 1. The image acquisition system 1 includes, for example, one or more terminal devices 2, one or more center server devices 10, and one or more vehicles 100.

The terminal device 2, the center server device 10, and the vehicle 100 communicate with each other via the network NW. The network NW includes, for example, the Internet, WAN (Wide Area Network), LAN (Local Area Network), provider device, wireless base station, and the like. Note that part or all of the functional configuration included in the terminal device 2, the center server device 10, or the vehicle 100 described below may be included in another device.

[Terminal device]
The terminal device 2 is, for example, a smartphone, a tablet terminal, a personal computer, or the like. In the terminal device 2, an application program for using the image acquisition system 1, a browser, or the like is activated to support the services described below. In the following description, it is assumed that the terminal device 2 is a smartphone and the application program (imaging application 4) is running. The imaging application 4 transmits a predetermined request to the center server device 10 according to a user's operation, or performs a push notification based on the information received from the center server device 10. The imaging application 4 is an application program provided by the center server device 10 or another server device (not shown).

[Center server device]
FIG. 2 is a diagram showing an example of a functional configuration of the center server device 10. The center server device 10 includes, for example, a communication unit 12, an information processing unit 14, an information providing unit 16, and a storage unit 30. The information processing unit 14 and the information providing unit 16 are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components are hardware (circuit part; LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), etc. It may be realized by (including circuitry) or by the cooperation of software and hardware. The program may be stored in advance in a storage device (non-transitory storage medium) such as an HDD (Hard Disk Drive) or a flash memory, or a removable storage medium (non-transitory storage medium) such as a DVD or a CD-ROM. The storage medium may be installed by being installed in the drive device.

The storage unit 30 is realized by, for example, a HDD, a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), a ROM (Read Only Memory), a RAM (Random Access Memory), or the like.

The communication unit 12 is a communication unit that communicates with the vehicle 100 or the terminal device 2 via the network NW.

The information processing unit 14 manages information acquired by the center server device 10 and information processed by the center server device 10. The information processing unit 14 processes the information transmitted by the terminal device 2 or the information transmitted by the vehicle 100 (details will be described later). The information providing unit 16 provides the terminal device 2 with the image transmitted by the vehicle 100.

The storage unit 30 stores, for example, application information 32, imaging processing information 34 (described later), captured image information 36 (described later), map information (not shown), and the like. The application information 32 is information regarding the imaging application 4 provided to the terminal device 2.

[vehicle]
FIG. 3 is a diagram showing an example of a part of the functional configuration included in the vehicle. The vehicle 100 includes, for example, a communication device 102, an HMI (Human Machine Interface) 104, a microphone 106, an occupant imaging camera (an example of “first imaging unit”) 108, and a landscape imaging camera (“second imaging unit”). An example) 110, a control device 120, and a navigation device 170.

The communication device 102 uses, for example, a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), or the like to communicate with another vehicle existing around the vehicle 100, or a wireless base station. It communicates with the center server device 10 via a station. The communication device 102 performs wireless communication to communicate with the terminal device 2.

The HMI 104 presents various information to an occupant of the vehicle 100 and receives an input operation by the occupant. The HMI 104 includes various display devices, speakers, buzzers, touch panels, switches, keys and the like. The microphone 106 acquires the input voice.

The occupant imaging camera 108 is, for example, a digital camera that uses a solid-state imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The occupant imaging camera 108 is attached at a position where an image of a user (occupant) in the vehicle interior can be captured.

The landscape image pickup camera 110 is, for example, a digital camera using a solid-state image pickup device such as CCD or CMOS. The landscape image pickup camera 110 is attached at a position where an image of a landscape around the vehicle can be picked up. Hereinafter, when the passenger image capturing camera 108 and the landscape image capturing camera 110 are not distinguished, they are simply referred to as “vehicle camera”.

The control device 120 includes, for example, an information processing unit 122, a position acquisition unit (an example of a “first acquisition unit”) 124, an emotion estimation unit (an example of a “second acquisition unit”) 126, and an imaging processing unit (“ An example of a “third acquisition unit” 128 and a storage unit 150. The information processing unit 122, the position acquisition unit 124, the emotion estimation unit 126, and the imaging processing unit 128 are realized, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these constituent elements may be realized by hardware (including a circuit unit; circuitry) such as an LSI, an ASIC, an FPGA, or a GPU, or by cooperation of software and hardware. Good. The program may be stored in advance in a storage device (non-transitory storage medium) such as HDD or flash memory, or in a removable storage medium (non-transitory storage medium) such as DVD or CD-ROM. It may be installed and installed by mounting the storage medium on the drive device.

The storage unit 150 is realized by, for example, a HDD, a flash memory, an EEPROM, a ROM, a RAM, or the like. The storage unit 150 stores, for example, imaging processing information 152 and captured image information 154. Details of these pieces of information will be described later.

The information processing unit 122 manages the information acquired by the vehicle 100 and performs a predetermined process on the information. The position acquisition unit 124 acquires the position information specified by the GNSS receiver 172 described later.

The emotion estimation unit 126 acquires the emotion of the user. The emotion estimation unit 126 estimates the emotion of the user, for example, based on the image of the occupant acquired in advance. The emotion estimation unit 126 compares, for example, an image of each emotion of the user stored in the storage unit 150 in advance (for example, an image when smiling or an image when having fun) with the captured image, and based on the comparison result. And estimate the emotion of the user. For example, the emotion estimation unit 126 extracts the feature amount of the image when the user smiles, which is stored in advance in the storage unit, and the extracted feature amount and the feature amount extracted from the captured image have a predetermined degree or more. If they match, the user is estimated to be smiling. The feature amount is information based on the brightness of each pixel or pixel group of the image. In this way, the emotion estimation unit 126 acquires the emotion of the user.

The emotion estimation unit 126 may also acquire the emotion of the user based on model information (not shown). The model information is information generated by performing machine learning such as deep learning on the learning target information. The learning target information is information in which an image of a person and a type of emotion corresponding to the image are associated with each other. This model information outputs the emotion of the user included in the image when the image is input. The emotion estimation unit 126 inputs, for example, the image of the user into the model information, and acquires the emotion of the user based on the output result of the model information.

The emotion estimation unit 126 may also perform spectrum analysis of the voice of the user input to the microphone 106 and estimate the emotion of the user based on the analysis result. In this case, the emotion estimation unit 126 estimates the emotion of the user based on, for example, the previously obtained voice analysis result for each emotion of the user and the voice analysis result input in real time. Further, the emotion estimation unit 126 may estimate the emotion of the user by combining the methods described above. In this case, emotions are estimated based on the result of statistically processing the score of the estimation result derived by each method.

The image capturing processing unit 128 captures an image when the position information of the vehicle 100 is included in the set image capturing area and the emotion of the user acquired by the emotion estimating unit 126 matches the emotion associated with the image capturing area. To obtain the specified image. Details of this processing will be described later. The set imaging area may be an imaging area desired by the user (desired imaging area) or may be an imaging area determined by a person other than the user or the system. For example, the imaging area is an imaging area recommended by a person other than the user (for example, an acquaintance or an acquaintance who has given authority), an area recommended by a predetermined website, or a predetermined website is evaluated. Areas that are more than a degree. The area recommended by the acquaintance is an area included in the information posted by the acquaintance in social work (SNS) or the like, or an area directly provided by the acquaintance to the user or another device such as the center server device 10. Hereinafter, an example in which the set imaging area is the “desired imaging area” will be described. The “emotion associated with the imaging area” may be the emotion set by the user or the emotion set by a person other than the user or the system.

The navigation device 170 includes, for example, a GNSS (Global Navigation Satellite System) receiver 172, a navigation HMI 174, and a route determination unit 176. The navigation device 170 holds the map information 180 in a storage device such as an HDD or a flash memory. The GNSS receiver 172 identifies the position of the vehicle 100 based on the signal received from the GNSS satellite. The navigation HMI 174 includes a display device, a speaker, a touch panel, keys, and the like. The navigation HMI 174 may be partly or wholly shared with the above-mentioned HMI 104. The route determination unit 176, for example, the route from the position of the vehicle 100 specified by the GNSS receiver 172 (or any input position) to the destination input by the occupant using the navigation HMI 174 (hereinafter, referred to as a map). The upper route) is determined by referring to the map information 180. The map information 180 is, for example, information in which a road shape is expressed by a link indicating a road and nodes connected by the link. The map information 180 may include curvature of the road, POI (Point Of Interest) information, and the like.

The navigation device 170 may perform route guidance using the navigation HMI 174 based on the route on the map. The navigation device 170 may be realized by, for example, the function of the terminal device 2 such as a smartphone or a tablet terminal owned by an occupant. The navigation device 170 may transmit the current position and the destination to the navigation server via the communication device 20 and acquire the route equivalent to the route on the map from the navigation server.

[sequence]
FIG. 4 is a sequence diagram showing processing executed by the image acquisition system 1. First, the terminal device 2 specifies a desired imaging area and emotion based on a user operation (step S100), and transmits the specified information (desired imaging area and emotion) to the center server device 10 (step S102). .. Next, the information processing unit 14 of the center server device 10 acquires the information transmitted to the terminal device 2, and stores the acquired information in the storage unit 30 as the imaging processing information 34 (step S104).

FIG. 5 is a diagram showing an example of the content of the imaging processing information 34. The imaging processing information 34 is, for example, information in which a desired imaging area and emotion are associated with each other. The imaging processing information 34 is transmitted to the vehicle 200. The vehicle 200 stores the imaging processing information 34 (imaging processing information 152) in the storage unit 150.

FIG. 6 is a diagram illustrating an example of the image IM1 displayed on the display unit (touch panel) of the terminal device 2. The image IM1 is an example of “information in which a recommended desired imaging area or a desired imaging area designated by the user and map information are associated with each other”. For example, a button B1 for designating a desired imaging area, a pull-down menu M1 for designating an emotion, and a map Map are displayed on the display unit. The map Map is associated with preset recommended desired imaging areas and emotions that define the timing at which images are taken in the desired imaging areas.

The recommended desired image pickup area is a high image pickup area designated by another user so far, an image pickup area designated by another user (for example, an acquaintance of the user of the terminal device 2), a web page, etc. It is an imaging area and the like which is said to have a high reputation in. For example, the center server device 10 may provide the recommended desired imaging area to the terminal device 2 based on the information transmitted by the terminal device having a predetermined authority (for example, an acquaintance of the user).

For example, if the user performs a predetermined operation such as a tap operation on the icon of recommendation 1 in the figure, when the user (for example, a vehicle occupant) smiles when passing through the hill of XX. An image is taken. Hereinafter, in such a desired imaging area, when the user's emotion becomes a prescribed emotion, the process of automatically acquiring the image at that time is referred to as “automatic imaging process”.

Further, when the user operates the button B1 for designating the desired imaging area, designates the desired imaging area on the map Map, and further operates the pull-down menu M1 to designate the emotion, the designated desired imaging area and Automatic image capturing processing is performed based on emotions.

Returning to FIG. Next, the position acquisition unit 124 of the vehicle 200 acquires position information (step S106), and transmits the acquired position information to the center server device 10 (step S108). The position acquisition unit 124 repeats the process of transmitting the position information at predetermined intervals (step S110).

Next, the information processing unit 14 of the center server device 10 determines whether or not the vehicle 100 is included in the desired imaging area (step S112), and when it is determined that the vehicle 100 is included in the desired imaging area, preparation for imaging is performed. To the vehicle 200 (step S114). At this time, the center server device 10 transmits the emotion corresponding to the desired imaging area to the vehicle 200.

Next, when the emotion estimation unit 126 of the vehicle 200 acquires the instruction to prepare for imaging, the emotion estimation unit 126 starts the process of estimating the emotion of the occupant (step S116). The image capturing processing unit 128 of the vehicle 200 determines whether or not the estimated emotion matches the emotion corresponding to the desired image capturing area of the image capturing processing information 152, and when it matches, the vehicle including the user and the user see it. An image of the scenery outside the vehicle is captured (step S118).

Next, the image capturing processing unit 128 of the vehicle 200 stores the captured image information 154 in which the captured image, the time when the image was captured, and the emotion are associated with each other in the storage unit 150 (step S120). Next, the vehicle 200 imaging processing unit 128 transmits the captured image information 154 to the center server device 10 (step S122). The information processing unit 14 of the center server device 10 stores the captured image information 154 (captured image information 36) transmitted by the vehicle 200 in the storage unit 30 (step S124).

Then, when there is a request from the terminal device 2, the information providing unit 16 of the center server device 10 displays the image captured by the user of the terminal device 2 stored in the storage unit 30 by the automatic image capturing process. (Step S126). As a result, the image captured by the automatic image capturing process is provided to the user.

In the configuration of the image acquisition system 1 of the present embodiment described above, the center server device 10 may be omitted. In this case, for example, in the process of FIG. 4, the terminal device 2 and the vehicle 100 directly communicate with each other to transmit and receive the desired imaging area and emotion. When the vehicle 100 is included in the desired imaging area, the vehicle 100 prepares for imaging, estimates the emotion of the occupant, and includes the user when the estimated emotion matches the emotion corresponding to the desired imaging area. An image of the scenery outside the vehicle seen from the user is captured.

[Applicable scene]
FIG. 7 is a diagram illustrating an example of a scene where the automatic image capturing process is performed. In the following scenes, an example will be described in which an image is taken when the user smiles in the desired imaging area AR.

At time T, when vehicle 200 enters desired imaging area AR, vehicle 200 estimates the emotion of the user. At time T+1, when the user smiles, the occupant imaging camera 108 images the user and the landscape imaging camera 110 images the landscape outside the vehicle. For example, the landscape image capturing camera 110 may capture a landscape in a direction in which the face of a smiling user faces. For example, the image capturing processing unit 128 may analyze the image captured by the user, derive the face orientation, and control the image capturing direction of the landscape image capturing camera 110 based on the derived result.

When the user no longer smiles at time T+2, the image capturing by the occupant image capturing camera 108 and the landscape image capturing camera 110 ends. Further, when the vehicle 100 passes through the desired imaging area AR, the process of estimating the emotion of the user is also terminated. As a result, an image of the user's smile and an image of the landscape expressing the smile are acquired.

In the above example, the image is described as being acquired when the emotion of the user becomes a designated emotion (for example, a smile), but instead of this, for example, when the user smiles after the fact. May be acquired (or retained).

FIG. 8 is a diagram for explaining a process when an image is acquired posteriorly. For example, the control device 120 of the vehicle 100 acquires images and estimates emotions during the period from entering the desired imaging area AR until leaving. Then, the control device 120 generates the captured image information 154 in which the emotion of the user and the image are associated with each other at predetermined time intervals. For example, the emotion estimation unit 126 classifies the user's emotion (eg, smile) into a predetermined pattern. For example, the emotion estimation unit 126 classifies the user into smiling, slightly smiling, and not smiling, and associates the classification result with the captured image information 154.

The control device 120 refers to the captured image information 154, extracts an image when the user is smiling (and has a slight smile), and sends the extracted images (IM15A, IM15B, IM16A, IM16B) to the center server device 10. To do. The process of extracting the image of a smiling face may be performed by the center server device 10.

[Image displayed on the display unit of the terminal device (1)]
FIG. 9 is a diagram showing an example of the image IM2 displayed on the terminal device 2. The image IM2 is an example of “information in which the area where the specific image is acquired and the map information are associated with each other” or “information in which the area where the specific image is acquired and the movement trajectory of the moving body are associated with each other”. .. For example, in the image IM2, the route along which the vehicle 100 has traveled, the desired image capturing area for which the automatic image capturing processing has been performed, and the emotion associated with the desired image capturing area are associated with the map information. For example, when the user operates an icon indicating a predetermined desired imaging area, the image captured in the desired imaging area is displayed on the display unit.

FIG. 10 is a diagram illustrating an example of the image IM3 displayed on the display unit when the icon indicating the predetermined desired imaging area is operated. For example, the image IM3 includes an image of a smiling user and an image of a landscape around the vehicle when the user is smiling.

As described above, the user can easily acquire the captured image by referring to the information of the route and the desired image capturing area associated with the map information.

[Image displayed on the display unit of the terminal device (Part 2)]
An image as shown below may be displayed on the display unit of the terminal device 2. FIG. 11 is a diagram showing an example of the image IM4 displayed on the terminal device 2. The image IM4 is an example of “information in which a region in which a specific image is acquired, a specific image and map information are associated with each other” or “information in which the first specific image and the second specific image are associated with each other”. .. For example, in the image IM4, the route along which the vehicle 100 has traveled, the desired image capturing area for which the automatic image capturing processing has been performed, and the image captured in the desired image capturing area are associated with the map information. The image captured in the desired imaging area is, for example, an image of the interior of the vehicle including the user and an image of the scenery around the vehicle when the user is smiling. Note that one of these images may be omitted. Further, in addition to the information included in the image IM2 in FIG. 9 described above, the image in the vehicle interior and the image around the vehicle may be associated with the map information.

As described above, by referring to the map information in which the image inside the vehicle interior and the image around the vehicle are associated with each other, the user can look back on the experience such as pleasure when driving.

In the above embodiment, the captured image is described as being provided to the user. However, instead of (or in addition to) this, a moving image may be provided to the user, or the moving image may be provided continuously in time. The captured image may be provided to the user. Further, the image (or moving image) provided to the user is not limited to the image when a specified emotion such as a smile is expressed, and may include images before and after the emotion is expressed.

Further, in the above embodiment, the image is described as being captured when the user's emotion becomes a predetermined emotion, but instead of this, when the emotion is different from the predetermined emotion, the image is displayed. It may be imaged. For example, an image may be captured when the emotion is different from the reference emotion (e.g., emotion in normal times) (pleasant emotion, happy emotion, etc.).

Further, in the above-described embodiment, the automatic image capturing process may be performed when the emotion of the designated user becomes the designated emotion. For example, when the emotions of a user (such as a child) sitting in the back seat or a user who matches the user included in the pre-registered image become the designated emotion, the automatic image capturing process is performed. May be done. Further, the designated user may be a plurality of users, and the automatic image capturing process may be performed when all or some of the plurality of users have a designated emotion.

According to the first embodiment described above, the image capturing processing unit 128 is imaged when the position information is included in the desired image capturing area and the emotion of the user matches the emotion associated with the image capturing processing information 152. By acquiring the specific image, it is possible to acquire highly valuable information for the user.

<Second Embodiment>
The second embodiment will be described below. In the first embodiment, the vehicle 100 has been described as performing the automatic image capturing process based on the instruction from the center server device 10. In the second embodiment, the vehicle 100 performs the automatic image capturing process without depending on the instruction from the center server device 10. Further, in the second embodiment, the desired image pickup area and emotion are set based on the operation of the HMI 104 of the vehicle 100. Hereinafter, differences from the first embodiment will be mainly described.

FIG. 12 is a flowchart showing an example of the flow of processing executed by the control device 120 of the second embodiment. First, the navigation device 170 sets the route of the vehicle based on the user's instruction (step S200). For example, the user operates the navigation HMI 174 to set a route from the departure point to the destination point.

Next, the information processing section 122 sets a desired imaging area and emotion based on the user's operation (step S202). For example, the information processing unit 122 causes the HMI 104 to display an image such as the image IM1 shown in FIG. 6 described above. The user performs an operation of setting a desired imaging area and emotion on the image IM1. The information processing unit 122 sets a desired imaging area and emotion based on the user's operation.

Next, the position acquisition unit 124 acquires position information (step S204). Next, the imaging processing unit 128 determines whether or not the acquired position information is included in the desired imaging area (step S206). If the acquired position information is not included in the desired imaging area, the process proceeds to step S214. In step S214, control device 120 determines whether vehicle 100 has arrived at the destination set by the user (step S214). When the vehicle 100 arrives at the destination set by the user, the process of one routine in this flowchart ends. If the vehicle 100 has not arrived at the destination set by the user, the process returns to step S204.

When the acquired position information is included in the desired imaging area, the emotion estimation unit 126 estimates the emotion of the user (step S208). Next, the imaging processing unit 128 determines whether or not the estimated emotion of the user matches the emotion set for the desired imaging area (step S210). Matching is not limited to complete matching, and may be that the matching degree is equal to or higher than a predetermined degree. For example, when the emotion set for the desired imaging area is a smile, it may be determined that the estimated emotion is a smile or a match. If the estimated emotion of the user does not match the emotion set for the desired imaging area, the process returns to step S204.

When the estimated emotion of the user matches the emotion set for the desired imaging area, the imaging processing unit 128 causes the vehicle camera to capture an image (step S212). Then, the process returns to step S204.

As described above, the user can operate the navigation device 170 to set the route and the automatic imaging process, which further improves the convenience for the user.

In the first embodiment, the image captured by the automatic image capturing process is described as being displayed on the display unit of the terminal device 2. However, instead of (or in addition to) this, the image captured by the automatic image capturing process is captured. The image (for example, the image IM2 of FIG. 9, the image IM3 of FIG. 10, or the image IM4 of FIG. 11) may be displayed on the display unit of the navigation HMI of the navigation device 170 or the display unit of the HMI 104. In this case, when the user selects an image that he/she likes, the communication device 102 of the vehicle 100 transmits the selected image to the terminal device 2 by performing wireless communication via the network NW or directly. In this way, the image captured by the automatic image capturing process is provided to the user. “Providing” means transmitting information to the user's terminal device 2 (or a terminal device specified by the user) or displaying the information on a predetermined display unit (for example, the display unit of the vehicle 100) so that the user can visually recognize it. Including being done.

According to the second embodiment described above, the vehicle 100 performs the automatic image capturing process and presents the image captured by the automatic image capturing process to the user, thereby improving the convenience of the user or simply The image captured by the automatic image capturing process may be shared by users in the vehicle. In this way, the user can acquire valuable information.

The processing of the second embodiment may be realized by the terminal device 2 provided with a camera corresponding to the vehicle camera. Further, the vehicle camera may be a specific camera provided in a place (vehicle or the like) different from the terminal device 2. In this case, the terminal device 2 communicates with the specific camera to control the specific camera.

<Third Embodiment>
The third embodiment will be described below. In the third embodiment, a camera set in a place different from the vehicle 100 (for example, a road) captures the image of the vehicle 100 in which the user who expresses the specified emotion is in the vehicle. Hereinafter, differences from the first embodiment will be mainly described.

FIG. 13: is a figure which shows an example of a structure of 1 A of image acquisition systems of 3rd Embodiment. The image acquisition system 1A further includes one or more roadside cameras 300 (300-1 to 300-3) in addition to the configuration of the image acquisition system 1 of the first embodiment. The roadside camera 300 is an example of a “first imaging unit”, a “second imaging unit”, or a “third imaging unit”. Although the roadside camera 300 is described as being installed on the road in the present embodiment, it is set at a place different from the vehicle 100 and is arranged at a position where an image including the vehicle 100 can be taken in the desired imaging area. If The roadside camera 300 images the vehicle 100 on which the user has boarded when the vehicle 100 enters the desired imaging area. The roadside camera 300 serves as the center server device 10 via the network NW. The road-side camera 300 may communicate with the vehicle 100 by performing wireless communication.

FIG. 14 is a sequence diagram showing an example of the flow of processing executed by the image acquisition system 1A of the third embodiment. The processing of steps S100 to S122 of FIG. 14 is the same as the processing of steps S100 to S122 of FIG. 4, and thus description thereof will be omitted (however, the processing of step S114 is omitted in FIG. 14).

After the processing of step S112, the center server device 10 instructs the roadside camera 300 to start imaging (step S113A). Next, the roadside camera 300 starts imaging according to the instruction of step S113A (step S113B). Next, the road-side camera 300 stores the captured image in the storage unit of the device itself together with the capturing time (step S113C). Next, the roadside camera 300 transmits information in which the captured image and the captured time are associated with each other to the center server device 10 (step S113D).

Next, the center server device 10 stores the information transmitted by the vehicle 100 and the roadside camera 300 in the storage unit 30 (step S123). Next, the center server device 10 refers to the information stored in the storage unit 30, and extracts the image captured when the user has the designated emotion using the time as a key (step S125).

FIG. 15 is a diagram for explaining a process of extracting a captured image. First, the center server device 10 refers to the information transmitted by the vehicle 100, identifies the time when the user has a designated emotion (for example, a smile), and further confirms the information transmitted by the roadside camera 300. The image picked up at the specified time is extracted with reference. Then, the center server device 10 associates the extracted image with the image of the user's designated smile in the information transmitted by the vehicle 100. As a result, the center server device 10 extracts the image captured by the camera of the vehicle 100 and the roadside camera 300 when the user is smiling.

Then, in the center server device 10, when there is a request from the terminal device 2, the user of the terminal device 2 stored in the storage unit 30 performs an automatic image capturing process (the passenger image capturing camera 108, the landscape image capturing camera 110, and the road side). The image captured by the camera 300) is transmitted to the terminal device 2 (step S127).

In the above example, the center server device 10 has been described as instructing the roadside camera 300 to take an image. However, when the vehicle 100 determines that the vehicle is included in the desired image pickup area, the roadside camera 300 is described. The image capturing instruction may be given to the user.

In the configuration of the image acquisition system 1 of the present embodiment described above, the center server device 200 may be omitted. In this case, for example, in the process of FIG. 14, when the terminal device 2 and the vehicle 100 directly communicate with each other to transmit and receive the designated desired image capturing area and emotion, and the vehicle 100 is included in the desired image capturing area. The vehicle 100 may prepare for imaging and start estimating the emotion of the occupant. In addition, the terminal device 2 or the vehicle 100 may instruct the road-side camera 300 to start imaging when the vehicle 100 is included in the desired imaging area.

[Applicable scene]
FIG. 16 is a diagram illustrating an example of a scene in which the automatic image capturing process is performed. Descriptions similar to those of FIG. 7 described above are omitted, and the processing of the roadside camera 300 will be mainly described here.

At time T, when the vehicle 200 enters the desired imaging area AR, the roadside camera 300 starts imaging. At time T+1 and time T+2 as well, the roadside camera 300 continues imaging. At time T+3, when the vehicle 200 exits from the desired imaging area AR, the roadside camera 300 stops imaging, associates the images captured between time T and time T+3 with time, and associates the associated information with the center. It is transmitted to the server device 10. Accordingly, the center server device 10 can acquire the image captured by the roadside camera 300 when the vehicle 100 is in the desired imaging area AR.

The timing at which the roadside cameras 300-1 to 300-3 start (or stop) imaging may be different. For example, when the vehicle 100 enters the desired imaging area AR, the roadside camera 300-1 starts imaging after the first predetermined time has elapsed, the imaging ends after the second predetermined time has elapsed, and the roadside has passed after the third predetermined time has elapsed. The camera 300-2 may start the image capturing and may end the image capturing after the elapse of the fourth predetermined time. In this case, each of the road-side cameras 300 starts the imaging at the timing when the vehicle 100 reaches the imaging area of the road-side camera 300, and ends the imaging at the timing when the vehicle 100 deviates from the imaging area. Timing is set. As a result, useless image capturing is suppressed.

FIG. 17 is a diagram illustrating an example of the image IM5 displayed on the display unit of the terminal device 2 according to the third embodiment. For example, in the image IM5, an image of a smiling user imaged by the occupant imaging camera 108, an image of the vehicle surroundings when the user is smiling imaged by the landscape imaging camera 110, and the roadside camera 300. An image including the vehicle 100 in which the user is riding when the imaged user is smiling is included. The image IM5 is an example of “information in which the first specific image, the second specific image, and the third specific image are associated with each other”. Note that, for example, the image IM5 may include an image of a smiling user imaged by the roadside camera 300 instead of (or in addition to) the occupant imaging camera 108. Further, for example, the image IM5 may include an image around the vehicle when the smiling face is captured by the road-side camera 300 instead of (or in addition to) the landscape image capturing camera 110.

In this way, the image including the vehicle in which the passenger is riding, which is captured by the roadside camera 300, is displayed on the display unit of the terminal device 2. As a result, the user can acquire images of the scenery, the vehicle, and the user when the user has a predetermined emotion (for example, a smile) while driving, and can immerse himself in more memories.

According to the third embodiment described above, the image captured by the roadside camera 300 can be acquired, so that the user can acquire more valuable information.

Further, some or all of the first to third embodiments described above may be combined and implemented. In addition, some or all of the functional configurations included in the terminal device 2, the center server device 10, or the vehicle 100 according to the first to third embodiments may be included in another device. For example, the functional configuration of the control device 120 or the center server device 10 may be included in the terminal device 2. Further, a part of the processes of the above sequence diagram or the flowchart may be omitted, or a part of the processes may be performed by another device.

For example, the terminal device 2 may include the functional configurations of the occupant imaging camera 108, the scenery imaging camera 110, the control device 120, and the navigation device 170 included in the vehicle 200 described above. In this case, of the two cameras of the terminal device 2, the terminal device 2 is installed in the vehicle such that the first camera images the occupant in the vehicle and the second camera images the landscape. Further, the terminal device 2 or the vehicle 200 may include the functional configuration of the center server device 10.

According to the embodiment described above, the image acquisition system 1 includes the position acquisition unit 124 that acquires the position information of the moving body (vehicle 100), the emotion estimation unit 126 that acquires the emotion of the user of the moving body, and the position. The image capturing processing unit 128 (which acquires the specific image captured when the information contained in the preset image capturing area and the emotion of the user acquired by the emotion estimating unit 126 matches the preset designated emotion) Alternatively, by providing the center server device 10), it is possible to obtain information of high value to the user.

[Hardware configuration]
FIG. 18 is a diagram illustrating an example of a hardware configuration of the control device 120 of the embodiment. As illustrated, the control device 120 includes a communication controller 120-1, a CPU 120-2, a RAM (Random Access Memory) 120-3 used as a working memory, and a ROM (Read Only Memory) 120- for storing a boot program and the like. 4, a storage device 120-5 such as a flash memory or an HDD (Hard Disk Drive), a drive device 120-6, etc. are connected to each other by an internal bus or a dedicated communication line. The communication controller 120-1 communicates with components other than the control device 120. A program 120-5a executed by the CPU 120-2 is stored in the storage device 120-5. This program is expanded in the RAM 120-3 by a DMA (Direct Memory Access) controller (not shown) or the like and executed by the CPU 120-2. As a result, part or all of the information processing unit 122, the emotion estimation unit 126, the position acquisition unit 124, and the imaging processing unit 128 are realized.

The embodiment described above can be expressed as follows.
A storage device that stores a program,
A hardware processor,
The hardware processor executes a program stored in the storage device,
Acquire the position information of the moving body,
Acquiring the emotions of the mobile user,
Acquiring a specific image captured when the position information is included in a preset imaging area and the acquired emotion of the user matches a preset designated emotion,
The imaging device configured as described above.

As described above, the embodiments for carrying out the present invention have been described using the embodiments, but the present invention is not limited to such embodiments, and various modifications and substitutions are made without departing from the gist of the present invention. Can be added.

1, 1A... Image acquisition system, 2... Terminal device, 4... Imaging application, 10... Center server device, 14... Information processing unit, 16... Information providing unit, 30... Storage unit, 34... Imaging processing information, 36... Imaging Image information, 100, 100A... Vehicle, 102... Communication device, 104... HMI, 106... Microphone, 108... Occupant imaging camera, 110... Landscape imaging camera, 120... Control device, 122... Information processing unit, 124... Position acquisition unit 126, emotion estimation unit, 128, imaging processing unit, 150, storage unit, 152, imaging processing information, 154, captured image information, 170, navigation device, 300, roadside camera

Claims (12)

A first acquisition unit for acquiring position information of the moving body;
A second acquisition unit for acquiring the emotion of the user of the mobile body;
Third, a specific image captured when the position information is included in a preset imaging area and the emotion of the user acquired by the second acquisition unit matches a preset specified emotion The acquisition part,
An image acquisition system including. The second acquisition unit starts a process of acquiring the emotion of the user when the position information acquired by the first acquisition unit is included in the imaging area.
The image acquisition system according to claim 1. The third acquisition unit acquires the specific image captured by the first imaging unit that captures the user,
The image acquisition system according to claim 1 or 2. The first imaging unit is provided around a moving body on which the user boarded or a road on which the moving body passes,
The image acquisition system according to claim 3. The third acquisition unit acquires a specific image captured by a second image capturing unit, which is provided in the moving body and captures the surrounding environment of the user,
The image acquisition system according to any one of claims 1 to 4. Further comprising an information providing unit for providing the specific image to the user,
The image acquisition system according to any one of claims 1 to 5. The third acquisition unit is provided on the first specific image captured by the first imaging unit that captures the user and the moving body, and is captured by the second imaging unit that captures the surrounding environment of the user. Acquired the second specific image,
The information providing unit provides a user with information in which the first specific image and the second specific image are associated with each other,
The image acquisition system according to claim 6. The third acquisition unit further acquires a third specific image imaged by the third imaging unit that images the moving body in the imaging area,
The information providing unit provides a user with information in which the first specific image, the second specific image, and the third specific image are associated with each other,
The image acquisition system according to claim 7. Further, an information providing unit that provides the user with information in which the recommended imaging area or the imaging area designated by the user and the map information are associated with each other,
The image acquisition system according to any one of claims 1 to 8. Information in which the area in which the specific image is acquired and map information are associated, information in which the area in which the specific image is acquired, the specific image, and the map information, or the specific image is acquired Further provided with an information providing unit for providing the user with information in which the region and the movement trajectory of the moving body are associated with each other,
The image acquisition system according to any one of claims 1 to 9. Computer
Acquire the position information of the moving body,
Acquiring the emotions of the mobile user,
Acquiring a specific image captured when the position information is included in a preset imaging area and the acquired emotion of the user matches a preset designated emotion,
Image acquisition method. On the computer,
Get the location information of the moving body,
The emotion of the user of the moving body is acquired,
The position information is included in a preset imaging area, and a specific image captured when the acquired emotion of the user matches a preset designated emotion is acquired,
program.

Images