JP7140692B2 - Image Acquisition System, Image Acquisition Method, and Program - Google Patents

Description

The present invention relates to an image acquisition system, an image acquisition method, and a program.

Conventionally, a sensor that detects an electrical signal generated in the body of an individual, an extraction circuit that extracts an electrical signal greater than a threshold value among the electrical signals as an emotional undulating signal, and an extraction circuit that is integrated with the human body of the individual. A device is disclosed that includes a camera mounted on a moving body (see, for example, Patent Document 1). This camera captures a cross-section of a life span corresponding to the external world of the human body during the imaging period corresponding to the generation period of the emotional undulating signal.

JP 2005-260892 A

However, there are cases in which the above devices cannot acquire valuable information for users.

SUMMARY OF THE INVENTION The present invention has been made in consideration of such circumstances, and one object thereof is to provide an image acquisition system, an image acquisition method, and a program capable of acquiring information of high value to the user. do.

An image acquisition system, an image acquisition method, and a program according to the present invention employ the following configurations.
(1): An image acquisition system according to an aspect of the present invention includes a first acquisition unit that acquires position information of a mobile object, a second acquisition unit that acquires emotion of a user of the mobile object, and the position information is included in a preset imaging area, and acquires a specific image captured when the user's emotion acquired by the second acquisition unit matches a preset specified emotion; Prepare.

(2): In the aspect of (1) above, the second acquisition unit acquires the emotion of the user when the position information acquired by the first acquisition unit is included in the imaging area. is the one that starts the

(3): In the aspect (1) or (2) above, the third acquiring section acquires the specific image captured by the first imaging section that captures the image of the user.

(4): In the aspect of (3) above, the first imaging section is provided around a moving object on which the user has boarded or a road through which the moving object passes.

(5): In any one of the above aspects (1) to (4), the third acquisition unit is provided in the moving object and is captured by a second imaging unit that captures an image of the surrounding environment of the user. It acquires the specified image.

(6): Aspects of any one of (1) to (5) above, further comprising an information providing unit for providing the specific image to the user.

(7): In the aspect of (6) above, the third acquisition unit is provided in the moving object and the first specific image captured by the first imaging unit that captures the image of the user, and A second specific image captured by a second imaging unit that captures an image of the surrounding environment is obtained, and the information providing unit provides the user with information in which the first specific image and the second specific image are associated with each other. It is.

(8): In the aspect of (7) above, the third obtaining unit further obtains a third specific image captured by an imaging unit that captures an image of the moving object in the imaging area, and the information providing unit Information that associates the first specific image, the second specific image, and the third specific image is provided to the user.

(9): In any one of the aspects (1) to (8) above, provision of information for providing the user with information in which the recommended imaging area or the imaging area designated by the user is associated with the map information. It further comprises a part.

(10): In the above aspects (1) to (9), information in which the area from which the specific image was obtained and map information are associated, the area from which the specific image was obtained, the specific image, and the map and an information providing unit for providing the user with information associated with the information or information associated with the area where the specific image was obtained and the moving locus of the moving body.

(11): An image acquisition method according to an aspect of the present invention is such that a computer acquires position information of a mobile object, acquires the emotion of a user of the mobile object, and captures an imaging area in which the position information is preset. and acquiring a specific image captured when the acquired emotion of the user matches a preset designated emotion.

(12): A program according to an aspect of the present invention causes a computer to acquire position information of a mobile body, acquire emotions of a user of the mobile body, and include the position information in a preset imaging area. and acquires a specific image captured when the acquired emotion of the user matches a preset specified emotion.

According to (1), (6), (7), (8), (11), and (12), the position information is included in the preset imaging area, and the use acquired by the second acquisition unit By acquiring a specific image captured when a person's emotion matches a preset specified emotion, it is possible to acquire valuable information for the user.

According to (2), unnecessary processing is suppressed because emotion estimation is acquired when it is included in the imaging area.

According to (3) or (4), since an image including the user is acquired, information of higher value to the user can be acquired.

According to (5), since an image in which the surrounding scenery is captured is acquired, it is possible to acquire more valuable information for the user.

According to (9), information in which the imaging area and the map information are associated is provided to the user, so that the user can easily recognize the imaging area.

According to (10), information in which the area from which the specific image is obtained is associated with map information, or information in which the area from which the specific image is obtained is associated with the movement trajectory of the moving object is sent to the user. By providing it, the user can easily recognize the area where the specific image was acquired.

1 is a diagram showing an example of the configuration of an image acquisition system 1; FIG. 2 is a diagram showing an example of a functional configuration of a center server device 10; FIG. It is a figure which shows an example of a part of functional structure contained in a vehicle. 4 is a sequence diagram showing processing executed by the image acquisition system 1; FIG. 4 is a diagram showing an example of the contents of imaging processing information 34. FIG. 3 is a diagram showing an example of an image IM1 displayed on the display unit (touch panel) of the terminal device 2; FIG. FIG. 4 is a diagram showing an example of a scene where automatic imaging processing is performed; FIG. 10 is a diagram for explaining processing when an image is acquired after the fact; 3 is a diagram showing an example of an image IM2 displayed on the terminal device 2; FIG. FIG. 10 is a diagram showing an example of an image IM3 displayed on the display unit when an icon indicating a predetermined desired imaging area is operated; 4 is a diagram showing an example of an image IM4 displayed on the terminal device 2. FIG. 9 is a flow chart showing an example of the flow of processing executed by the control device 120 of the second embodiment; FIG. 11 is a diagram showing an example of the configuration of an image acquisition system 1A according to a third embodiment; FIG. FIG. 12 is a sequence diagram showing an example of the flow of processing executed by the image acquisition system 1A of the third embodiment; It is a figure for demonstrating the process by which the imaged image is extracted. FIG. 4 is a diagram showing an example of a scene where automatic imaging processing is performed; FIG. 12 is a diagram showing an example of an image IM5 displayed on the display unit of the terminal device 2 of the third embodiment; FIG. It is a figure which shows an example of the hardware constitutions of the control apparatus 120 of embodiment.

Embodiments of an image acquisition system, an image acquisition method, and a program according to the present invention will be described below with reference to the drawings. In the following description, an example in which an image acquisition system captures an image of a user riding in a four-wheeled vehicle will be described. The image acquisition system may be applied to Mobile objects other than vehicles include vehicles such as two-wheeled vehicles and three-wheeled vehicles, bicycles, and the like. Also, the mobile object may be a terminal device (for example, a smart phone) that moves with the passenger.

<First Embodiment>
FIG. 1 is a diagram showing an example of the configuration of an image acquisition system 1. As shown in FIG. The image acquisition system 1 includes, for example, one or more terminal devices 2 , one or more center server devices 10 , and one or more vehicles 100 .

Terminal device 2, center server device 10, and vehicle 100 communicate with each other via network NW. The network NW includes, for example, the Internet, a WAN (Wide Area Network), a LAN (Local Area Network), a provider device, a radio base station, and the like. A part or all of the functional configuration included in the terminal device 2, the center server device 10, or the vehicle 100, which will be described later, may be included in another device.

[Terminal device]
The terminal device 2 is, for example, a smart phone, a tablet terminal, a personal computer, or the like. In the terminal device 2, an application program, a browser, or the like for using the image acquisition system 1 is activated and supports the services described below. In the following description, it is assumed that the terminal device 2 is a smart phone and the application program (imaging application 4) is running. The imaging application 4 transmits a predetermined request to the center server device 10 according to a user's operation, and performs push notification based on information received from the center server device 10 . The imaging application 4 is an application program provided by the center server device 10 or another server device (not shown).

[Center server device]
FIG. 2 is a diagram showing an example of the functional configuration of the center server device 10. As shown in FIG. The center server device 10 includes, for example, a communication unit 12, an information processing unit 14, an information providing unit 16, and a storage unit 30. The information processing unit 14 and the information providing unit 16 are implemented by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components are hardware (circuit part; circuitry) or by cooperation of software and hardware. The program may be stored in advance in a storage device (non-transitory storage medium) such as a HDD (Hard Disk Drive) or flash memory, or may be stored in a removable storage medium (non-transitory storage medium) such as a DVD or CD-ROM. storage medium), and may be installed by loading the storage medium into a drive device.

The storage unit 30 is implemented by, for example, an HDD, flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory).

The communication unit 12 is a communication unit that communicates with the vehicle 100 or the terminal device 2 via the network NW.

The information processing unit 14 manages information acquired by the center server device 10 and information processed by the center server device 10 . The information processing unit 14 processes the information transmitted by the terminal device 2 or the information transmitted by the vehicle 100 (details will be described later). The information providing unit 16 provides the terminal device 2 with the image transmitted by the vehicle 100 .

The storage unit 30 stores, for example, application information 32, imaging processing information 34 (described later), captured image information 36 (described later), map information (not shown), and the like. The application information 32 is information about the imaging application 4 provided to the terminal device 2 .

[vehicle]
FIG. 3 is a diagram showing an example of part of the functional configuration included in the vehicle. The vehicle 100 includes, for example, a communication device 102, an HMI (Human Machine Interface) 104, a microphone 106, an occupant imaging camera (an example of a "first imaging unit") 108, and a scenery imaging camera (a "second imaging unit"). example) 110 , a control device 120 , and a navigation device 170 .

The communication device 102 uses, for example, a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), or the like, to communicate with other vehicles existing around the vehicle 100, or to communicate with a wireless base. It communicates with the center server device 10 via the station. The communication device 102 communicates with the terminal device 2 by wireless communication.

The HMI 104 presents various types of information to the occupants of the vehicle 100 and accepts input operations by the occupants. The HMI 104 includes various display devices, speakers, buzzers, touch panels, switches, keys, and the like. A microphone 106 acquires the input voice.

The occupant imaging camera 108 is, for example, a digital camera using a solid-state imaging device such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). The occupant imaging camera 108 is attached at a position capable of imaging a user (occupant) inside the vehicle.

The landscape imaging camera 110 is, for example, a digital camera using a solid-state imaging device such as CCD or CMOS. The landscape imaging camera 110 is attached at a position capable of imaging the surrounding landscape of the vehicle. Hereinafter, when the occupant imaging camera 108 and the scenery imaging camera 110 are not distinguished, they are simply referred to as "vehicle camera".

The control device 120 includes, for example, an information processing unit 122, a position acquisition unit (an example of a “first acquisition unit”) 124, an emotion estimation unit (an example of a “second acquisition unit”) 126, and an imaging processing unit (a “ An example of a “third acquisition unit”) 128 and a storage unit 150 . The information processing unit 122, the position acquiring unit 124, the emotion estimating unit 126, and the imaging processing unit 128 are implemented by, for example, a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as LSI, ASIC, FPGA, GPU, etc., or by cooperation of software and hardware. good too. The program may be stored in advance in a storage device (non-transitory storage medium) such as an HDD or flash memory, or may be stored in a removable storage medium (non-transitory storage medium) such as a DVD or CD-ROM. It may be installed by loading the storage medium into the drive device.

The storage unit 150 is implemented by, for example, an HDD, flash memory, EEPROM, ROM, or RAM. The storage unit 150 stores, for example, imaging processing information 152 and captured image information 154 . Details of these pieces of information will be described later.

The information processing unit 122 manages information acquired by the vehicle 100 and performs predetermined processing on the information. The position acquisition unit 124 acquires position information specified by a GNSS receiver 172, which will be described later.

Emotion estimation unit 126 acquires the user's emotion. The emotion estimator 126, for example, estimates the user's emotion based on an image of the passenger acquired in advance. The emotion estimating unit 126 compares, for example, an image for each emotion of the user stored in advance in the storage unit 150 (for example, an image when smiling or an image when having fun) with the captured image, and based on the comparison result. to estimate the user's emotion. For example, the emotion estimating unit 126 extracts the feature amount of an image when the user is smiling, which is stored in advance in the storage unit, and the extracted feature amount and the feature amount extracted from the captured image are equal to or greater than a predetermined degree. If there is a match, the user is assumed to be smiling. A feature amount is information based on the luminance of each pixel or pixel group of an image. Thus, the emotion estimation unit 126 acquires the user's emotion.

Also, the emotion estimation unit 126 may acquire the user's emotion based on model information (not shown). Model information is information generated by performing machine learning such as deep learning on learning target information. The learning target information is information in which an image of a person and a type of emotion corresponding to the image are associated with each other. When an image is input, this model information outputs the user's emotion contained in the image. The emotion estimation unit 126, for example, inputs a user's image into the model information and acquires the user's emotion based on the output result of the model information.

The emotion estimation unit 126 may perform spectral analysis on the user's voice input to the microphone 106 and estimate the user's emotion based on the analysis results. In this case, the emotion estimating unit 126 estimates the user's emotion based on, for example, the analysis result of the voice obtained in advance for each emotion of the user and the analysis result of the voice input in real time. Moreover, the emotion estimation unit 126 may estimate the user's emotion by combining the techniques described above. In this case, emotions are estimated based on results obtained by statistically processing scores of estimation results derived by each method.

The imaging processing unit 128 captures an image when the position information of the vehicle 100 is included in the set imaging area and the user's emotion acquired by the emotion estimation unit 126 matches the emotion associated with the imaging area. get the specified image. The details of this processing will be described later. The set imaging area may be an imaging area desired by the user (desired imaging area), or may be an imaging area determined by a person other than the user or by the system. For example, the imaging area is an imaging area recommended by a person other than the user (for example, an acquaintance or an acquaintance who has given authority), an area recommended by a predetermined website, etc., and a predetermined evaluation on a predetermined website. For example, an area that is greater than or equal to the degree. The area recommended by an acquaintance is an area included in information posted by the acquaintance on a social work (SNS) or the like, or an area directly provided by the acquaintance to a user or another device such as the center server device 10. - 特許庁An example in which the set imaging area is the "desired imaging area" will be described below. Also, the "emotion associated with the imaging area" may be an emotion set by the user, or may be an emotion set by a person other than the user or by the system.

The navigation device 170 includes, for example, a GNSS (Global Navigation Satellite System) receiver 172 , a navigation HMI 174 and a route determination section 176 . The navigation device 170 holds map information 180 in a storage device such as an HDD or flash memory. GNSS receiver 172 locates vehicle 100 based on signals received from GNSS satellites. The navigation HMI 174 includes a display device, speaker, touch panel, keys, and the like. The navigation HMI 174 may be partially or entirely shared with the HMI 104 described above. For example, the route determination unit 176 determines a route (hereinafter referred to as a map upper route) is determined with reference to the map information 180 . The map information 180 is, for example, information in which road shapes are represented by links indicating roads and nodes connected by the links. The map information 180 may include road curvature, POI (Point Of Interest) information, and the like.

The navigation device 170 may provide route guidance using the navigation HMI 174 based on the route on the map. The navigation device 170 may be realized, for example, by the function of the terminal device 2 such as a smart phone or a tablet terminal owned by the passenger. The navigation device 170 may transmit the current position and the destination to the navigation server via the communication device 20 and acquire a route equivalent to the route on the map from the navigation server.

[sequence]
FIG. 4 is a sequence diagram showing processing executed by the image acquisition system 1. As shown in FIG. First, the terminal device 2 designates the desired imaging area and emotion based on the user's operation (step S100), and transmits the designated information (desired imaging area and emotion) to the center server device 10 (step S102). . Next, the information processing section 14 of the center server device 10 acquires the information transmitted to the terminal device 2, and stores the acquired information in the storage section 30 as the imaging processing information 34 (step S104).

FIG. 5 is a diagram showing an example of the contents of the imaging processing information 34. As shown in FIG. The imaging processing information 34 is, for example, information in which a desired imaging area and an emotion are associated with each other. This imaging processing information 34 is transmitted to the vehicle 200 . The vehicle 200 causes the storage unit 150 to store the imaging processing information 34 (imaging processing information 152).

FIG. 6 is a diagram showing an example of an image IM1 displayed on the display unit (touch panel) of the terminal device 2. As shown in FIG. The image IM1 is an example of "information in which a recommended desired imaging area or a desired imaging area designated by a user is associated with map information". For example, the display unit displays a button B1 for specifying a desired imaging area, a pull-down menu M1 for specifying an emotion, and a map Map. In the map Map, a preset recommended desired imaging area is associated with an emotion that defines the timing at which an image is captured in the desired imaging area.

The recommended desired imaging area is an imaging area that has been designated by other users to a high degree, an imaging area designated by another user (for example, an acquaintance of the user of the terminal device 2), a web page, or the like. It is an imaging area, etc., which is said to have a high reputation in the field. For example, the center server device 10 may provide a recommended desired imaging area to the terminal device 2 based on information transmitted from a terminal device of a person with predetermined authority (for example, an acquaintance of the user).

For example, when the user performs a predetermined operation such as a tap operation on the icon of Recommendation 1 in the figure, the user (for example, the passenger of the vehicle) smiles when passing through the hill of ○○. An image is captured. Hereinafter, such a process of automatically acquiring an image when the user's emotion becomes a specified emotion in the desired imaging area will be referred to as "automatic imaging process".

Further, when the user operates the button B1 for specifying the desired imaging area, specifies the desired imaging area on the map Map, and further operates the pull-down menu M1 to specify the emotion, the specified desired imaging area and Automatic imaging processing is performed based on emotions.

Return to FIG. Next, the position acquisition unit 124 of the vehicle 200 acquires position information (step S106), and transmits the acquired position information to the center server device 10 (step S108). The position acquisition unit 124 repeats the process of transmitting position information at predetermined intervals (step S110).

Next, the information processing unit 14 of the center server device 10 determines whether or not the vehicle 100 is included in the desired imaging area (step S112). to the vehicle 200 (step S114). At this time, the center server device 10 transmits the emotion corresponding to the desired imaging area to the vehicle 200 .

Next, when the emotion estimating unit 126 of the vehicle 200 acquires an instruction to prepare for imaging, the process of estimating the emotion of the occupant is started (step S116). The imaging processing unit 128 of the vehicle 200 determines whether or not the estimated emotion matches the emotion corresponding to the desired imaging area of the imaging processing information 152. The scenery outside the vehicle is imaged (step S118).

Next, the imaging processing unit 128 of the vehicle 200 causes the storage unit 150 to store the captured image information 154 in which the captured image, the time when the image was captured, and the emotion are associated with each other (step S120). Next, the vehicle 200 imaging processing unit 128 transmits the captured image information 154 to the center server device 10 (step S122). The information processing unit 14 of the center server device 10 stores the captured image information 154 (captured image information 36) transmitted from the vehicle 200 in the storage unit 30 (step S124).

In response to a request from the terminal device 2, the information providing unit 16 of the center server device 10 sends an image captured by the user of the terminal device 2 stored in the storage unit 30 by the automatic imaging process to the terminal device 2. (step S126). As a result, the user is provided with an image captured by the automatic image capturing process.

In the configuration of the image acquisition system 1 of this embodiment described above, the center server device 10 may be omitted. In this case, for example, in the process of FIG. 4, the terminal device 2 and the vehicle 100 directly communicate to transmit and receive the desired imaging area and emotion. Then, when the vehicle 100 is included in the desired imaging area, the vehicle 100 prepares for imaging, estimates the emotion of the occupant, and if the estimated emotion matches the emotion corresponding to the desired imaging area, the vehicle including the user and To take an image of the scenery outside the vehicle seen from the user.

[Applicable scenes]
FIG. 7 is a diagram showing an example of a scene where automatic imaging processing is performed. In the following scene, an example in which an image is captured when the user smiles in the desired imaging area AR will be described.

At time T, when the vehicle 200 enters the desired imaging area AR, the vehicle 200 estimates the emotion of the user. At time T+1, when the user smiles, the occupant imaging camera 108 images the user and the landscape imaging camera 110 images the scenery outside the vehicle. For example, the scenery imaging camera 110 may capture the scenery in the direction in which the smiling user's face is facing. For example, the imaging processing unit 128 may analyze an image of the user captured, derive the orientation of the face, and control the imaging direction of the landscape imaging camera 110 based on the derivation result.

At time T+2, when the user no longer smiles, the occupant imaging camera 108 and the landscape imaging camera 110 stop capturing images. Further, when the vehicle 100 passes through the desired imaging area AR, the process of estimating the user's emotion also ends. As a result, an image of the smiling face of the user and an image of the scenery expressing the smiling face are obtained.

In the above example, the image is acquired when the user's emotion is a specified emotion (for example, a smile). images may be acquired (or retained).

FIG. 8 is a diagram for explaining processing when an image is acquired after the fact. For example, the control device 120 of the vehicle 100 acquires images and estimates emotions during the period from entering the desired imaging area AR to leaving the desired imaging area AR. Then, the control device 120 generates captured image information 154 in which the user's emotion and the image are associated with each other at predetermined time intervals. For example, the emotion estimator 126 classifies the user's emotions (eg, smile) into predetermined patterns. For example, the emotion estimation unit 126 classifies the user into smiling, slightly smiling, not smiling, etc., and associates the classification result with the captured image information 154 .

The control device 120 refers to the captured image information 154 to extract an image when the user is smiling (or slightly smiling), and transmits the extracted images (IM15A, IM15B, IM16A, IM16B) to the center server device 10. do. Note that the process of extracting an image of a smiling face may be performed by the center server device 10 .

[Image displayed on the display unit of the terminal device (Part 1)]
FIG. 9 is a diagram showing an example of an image IM2 displayed on the terminal device 2. As shown in FIG. The image IM2 is an example of "information in which the area where the specific image is acquired and the map information are associated" or "information in which the area where the specific image is acquired and the movement trajectory of the moving object are associated". . For example, in the image IM2, the route traveled by the vehicle 100, the desired imaging area for which automatic imaging processing was performed, and the emotion associated with the desired imaging area are associated with the map information. For example, when the user operates an icon indicating a predetermined desired imaging area, an image captured in the desired imaging area is displayed on the display unit.

FIG. 10 is a diagram showing an example of an image IM3 displayed on the display section when an icon indicating a predetermined desired imaging area is operated. For example, the image IM3 includes an image of a smiling user and an image of scenery around the vehicle when the user is smiling.

As described above, the user can easily obtain the captured image by referring to the route and desired imaging area information associated with the map information.

[Image displayed on the display unit of the terminal device (part 2)]
An image as shown below may be displayed on the display unit of the terminal device 2 . FIG. 11 is a diagram showing an example of an image IM4 displayed on the terminal device 2. As shown in FIG. The image IM4 is an example of "information in which the area from which the specific image was acquired, the specific image, and map information are associated" or "information in which the first specific image and the second specific image are associated." . For example, in the image IM4, the route traveled by the vehicle 100, the desired imaging area for which automatic imaging processing was performed, and the image captured in the desired imaging area are associated with the map information. The image captured in the desired imaging area is, for example, an image of the inside of the vehicle including the user and an image of the scenery around the vehicle when the user is smiling. Note that one of these images may be omitted. Further, in addition to the information included in the image IM2 of FIG. 9 described above, the image inside the vehicle interior and the image around the vehicle may be associated with the map information.

As described above, the user can look back on the experience of driving, such as the enjoyment, by referring to the map information in which the image of the interior of the vehicle and the image of the surroundings of the vehicle are associated with each other.

In the above embodiment, the captured image is provided to the user, but instead (or in addition), a moving image may be provided to the user, or the image may be provided continuously in time. A captured image may be provided to the user. In addition, the image (or moving image) provided to the user is not limited to the image when the designated emotion such as a smile is expressed, and may include images before and after the expression of the emotion.

Also, in the above embodiment, an image is captured when the user's emotion is a predetermined emotion. It may be imaged. For example, an image may be captured when the emotion becomes an emotion (e.g., happy emotion, happy emotion, etc.) different from the reference emotion (eg, normal emotion).

Further, in the above-described embodiment, automatic imaging processing may be performed when the specified emotion of the user becomes the specified emotion. For example, when the emotion of a user (for example, a child) sitting in a backseat or a user who matches a user included in a pre-registered image has a specified emotion, automatic imaging processing is performed. may be Also, the designated user may be a plurality of users, and automatic imaging processing may be performed when all or some of the plurality of users exhibit a designated emotion.

According to the first embodiment described above, the imaging processing unit 128 performs imaging when the position information is included in the desired imaging area and the user's emotion matches the emotion associated with the imaging processing information 152. By acquiring the specific image, it is possible to acquire valuable information for the user.

<Second embodiment>
A second embodiment will be described below. In the first embodiment, it is assumed that the vehicle 100 performs automatic imaging processing based on instructions from the center server device 10 . In the second embodiment, the vehicle 100 performs automatic imaging processing without depending on instructions from the center server device 10 . Further, in the second embodiment, setting of the desired imaging area and emotion is performed based on the operation of the HMI 104 of the vehicle 100 . The following description will focus on differences from the first embodiment.

FIG. 12 is a flow chart showing an example of the flow of processing executed by the control device 120 of the second embodiment. First, the navigation device 170 sets the route of the vehicle based on the user's instruction (step S200). For example, the user operates the navigation HMI 174 to set a route from the departure point to the destination point.

Next, the information processing section 122 sets the desired imaging area and emotion based on the user's operation (step S202). For example, the information processing unit 122 causes the HMI 104 to display an image such as the image IM1 shown in FIG. 6 described above. The user performs an operation of setting a desired imaging area and emotion for the image IM1. The information processing section 122 sets the desired imaging area and emotion based on the user's operation.

Next, the position acquisition unit 124 acquires position information (step S204). Next, the imaging processing unit 128 determines whether or not the acquired position information is included in the desired imaging area (step S206). If the acquired position information is not included in the desired imaging area, the process proceeds to step S214. In step S214, control device 120 determines whether or not vehicle 100 has arrived at the destination set by the user (step S214). When the vehicle 100 reaches the destination set by the user, the processing of one routine of this flowchart ends. If the vehicle 100 has not arrived at the destination set by the user, the process returns to step S204.

If the acquired position information is included in the desired imaging area, the emotion estimation unit 126 estimates the user's emotion (step S208). Next, the imaging processing unit 128 determines whether or not the estimated user's emotion matches the emotion set for the desired imaging area (step S210). Matching is not limited to perfect matching, and may be a degree of matching equal to or higher than a predetermined degree. For example, when the emotion set for the desired imaging area is a smile, even if the estimated emotion is a slight smile, it may be determined as a match. If the estimated user's emotion does not match the emotion set for the desired imaging area, the process returns to step S204.

If the estimated user's emotion matches the emotion set for the desired imaging area, the imaging processing unit 128 causes the vehicle camera to capture an image (step S212). After that, the process returns to step S204.

As described above, the user can operate the navigation device 170 to set the route as well as the automatic imaging process, which further improves convenience for the user.

Further, in the first embodiment, the image captured by the automatic image capturing process is displayed on the display unit of the terminal device 2, but instead of (or in addition to) the image captured by the automatic image capturing process. 9, image IM3 in FIG. 10, or image IM4 in FIG. In this case, when the user selects a favorite image, the communication device 102 of the vehicle 100 transmits the selected image to the terminal device 2 via the network NW or through direct wireless communication. In this way, the user is provided with an image captured by the automatic image capturing process. "Providing" means transmitting information to the terminal device 2 of the user (or a terminal device designated by the user), or displaying the information on a predetermined display unit (for example, the display unit of the vehicle 100) so that the user can visually recognize it. including being

According to the second embodiment described above, the vehicle 100 performs automatic imaging processing and presents an image captured by the automatic imaging processing to the user. Users in the vehicle can share an image captured by the automatic image capturing process. Thus, the user can obtain valuable information.

In addition, the process of 2nd Embodiment may be implement|achieved by the terminal device 2 provided with the camera corresponded to a vehicle camera. Also, the vehicle camera may be a specific camera provided at a location (vehicle or the like) different from the terminal device 2 . In this case, the terminal device 2 communicates with the specific camera and controls the specific camera.

<Third Embodiment>
A third embodiment will be described below. In the third embodiment, a camera set at a location (for example, a road) different from the vehicle 100 captures an image of the vehicle 100 boarded by a user who expresses a specified emotion. The following description will focus on differences from the first embodiment.

FIG. 13 is a diagram showing an example of the configuration of an image acquisition system 1A according to the third embodiment. The image acquisition system 1A further includes one or more roadside cameras 300 (300-1 to 300-3) in addition to the configuration of the image acquisition system 1 of the first embodiment. The roadside camera 300 is an example of a "first imaging section", a "second imaging section", or a "third imaging section". In this embodiment, the roadside camera 300 is described as being installed on the road, but is set at a location different from the vehicle 100 and arranged at a position where an image including the vehicle 100 can be captured in the desired imaging area. If it is The road-side camera 300 captures an image of the vehicle 100 boarded by the user when the vehicle 100 enters the desired imaging area. The roadside camera 300 is connected to the center server device 10 via the network NW. Moreover, the roadside camera 300 may communicate with the vehicle 100 by wireless communication.

FIG. 14 is a sequence diagram showing an example of the flow of processing executed by the image acquisition system 1A of the third embodiment. The processing of steps S100 to S122 of FIG. 14 is the same as the processing of steps S100 to S122 of FIG. 4, so the description thereof is omitted (however, the processing of step S114 is omitted in FIG. 14).

After the process of step S112, the center server device 10 instructs the roadside camera 300 to start imaging (step S113A). Next, road-side camera 300 starts imaging in accordance with the instruction in step S113A (step S113B). Next, the road-side camera 300 stores the imaged image together with the imaged time in the storage unit of its own device (step S113C). Next, the road-side camera 300 transmits to the center server device 10 information in which the imaged image and the imaging time are associated with each other (step S113D).

Next, the center server device 10 stores the information transmitted by the vehicle 100 and the roadside camera 300 in the storage unit 30 (step S123). Next, the center server device 10 refers to the information stored in the storage unit 30, and extracts an image captured when the user has the designated emotion, using time as a key (step S125).

FIG. 15 is a diagram for explaining processing for extracting a captured image. First, the center server device 10 refers to the information transmitted by the vehicle 100, identifies the time when the user has a specified emotion (for example, smile), and furthermore, reads the information transmitted by the roadside camera 300. By referring to it, an image captured at the specified time is extracted. Then, the center server device 10 associates the above extracted image with the image of the smile specified by the user in the information transmitted by the vehicle 100 . Thereby, the center server device 10 extracts an image when the user is smiling, which is captured by the camera of the vehicle 100 and the roadside camera 300 .

In response to a request from the terminal device 2, the center server device 10 allows the user of the terminal device 2 stored in the storage unit 30 to perform automatic imaging processing (passenger imaging camera 108, scenery imaging camera 110, and road side imaging). The image captured by the camera 300) is transmitted to the terminal device 2 (step S127).

In the above example, the center server device 10 instructs the roadside camera 300 to take an image. You may give an instruction of imaging to .

In the configuration of the image acquisition system 1 of this embodiment described above, the center server device 200 may be omitted. In this case, for example, in the process of FIG. 14, the terminal device 2 and the vehicle 100 directly communicate to transmit and receive the specified desired imaging area and emotion, and the vehicle 100 is included in the desired imaging area. , the vehicle 100 may prepare for imaging and start estimating the emotions of the occupant. Further, when the vehicle 100 is included in the desired imaging area, the terminal device 2 or the vehicle 100 may instruct the roadside camera 300 to start imaging.

[Applicable scene]
FIG. 16 is a diagram showing an example of a scene where automatic imaging processing is performed. The description similar to that of FIG. 7 is omitted, and the processing of the roadside camera 300 will be mainly described here.

At time T, when the vehicle 200 enters the desired imaging area AR, the roadside camera 300 starts imaging. Also at time T+1 and time T+2, the roadside camera 300 continues imaging. At time T+3, when the vehicle 200 leaves the desired imaging area AR, the roadside camera 300 stops imaging, associates the images captured between time T and time T+3 with the time, and sends the associated information to the center. Send to the server device 10 . Thereby, the center server device 10 can acquire the image captured by the roadside camera 300 when the vehicle 100 is present in the desired imaging area AR.

Note that the timing at which each of the roadside cameras 300-1 to 300-3 starts (or stops) imaging may be different. For example, the vehicle 100 enters the desired imaging area AR, the road side camera 300-1 starts imaging after a first predetermined time has passed, finishes imaging after a second predetermined time has passed, and road side camera 300-1 after a third predetermined time has passed. Camera 300-2 may start imaging and end imaging after the fourth predetermined time has elapsed. In this case, each of the roadside cameras 300 starts imaging when the vehicle 100 reaches the imaging area of the roadside camera 300 and ends imaging when the vehicle 100 leaves the imaging area. Timing is set. As a result, useless imaging of images is suppressed.

FIG. 17 is a diagram showing an example of an image IM5 displayed on the display unit of the terminal device 2 of the third embodiment. For example, the image IM5 includes an image of a smiling user captured by the occupant imaging camera 108, an image of the surroundings of the vehicle when the user is smiling captured by the scenery imaging camera 110, and an image captured by the roadside camera 300. An image including the vehicle 100 in which the user is riding when the captured user is smiling is included. The image IM5 is an example of "information in which the first specific image, the second specific image, and the third specific image are associated with each other." For example, the image IM5 may include an image of a smiling user captured by the roadside camera 300 instead of (or in addition to) the occupant imaging camera 108 . Further, for example, the image IM5 may include an image of the surroundings of the vehicle when the vehicle is smiling and captured by the roadside camera 300 instead of (or in addition to) the scenery imaging camera 110 .

In this way, an image including the vehicle in which the driver has boarded is captured by the roadside camera 300 and displayed on the display unit of the terminal device 2 . As a result, the user can acquire an image of the scenery, the vehicle, and the user when he or she is feeling a predetermined emotion (for example, smiling) while driving, and can be more immersed in memories.

According to the third embodiment described above, the image captured by the roadside camera 300 can be obtained, so that the user can obtain more valuable information.

Also, part or all of the first to third embodiments described above may be combined and implemented. Further, part or all of the functional configurations included in the terminal device 2, the center server device 10, or the vehicle 100 of the first to third embodiments may be included in another device. For example, the functional configuration of the control device 120 or the center server device 10 may be included in the terminal device 2 . Furthermore, part of the processes in the sequence diagrams or flowcharts described above may be omitted, and part of the processes may be performed by another device.

For example, the terminal device 2 may have the functional configurations of the passenger imaging camera 108, the scenery imaging camera 110, the control device 120, and the navigation device 170 included in the vehicle 200 described above. In this case, the terminal device 2 is installed in the vehicle or the like so that the first camera of the two cameras of the terminal device 2 captures the occupant in the vehicle and the second camera captures the scenery. Also, the terminal device 2 or the vehicle 200 may have the functional configuration of the center server device 10 .

According to the embodiment described above, the image acquisition system 1 includes the position acquisition unit 124 that acquires the position information of the mobile object (vehicle 100), the emotion estimation unit 126 that acquires the emotion of the user of the mobile object, and the position information. The imaging processing unit 128 (the imaging processing unit 128 ( Alternatively, by providing the center server device 10), it is possible to acquire valuable information for the user.

[Hardware configuration]
FIG. 18 is a diagram showing an example of the hardware configuration of the control device 120 of the embodiment. As shown, the control device 120 includes a communication controller 120-1, a CPU 120-2, a RAM (Random Access Memory) 120-3 used as a working memory, a ROM (Read Only Memory) 120- for storing a boot program and the like. 4. A storage device 120-5 such as a flash memory or HDD (Hard Disk Drive), a drive device 120-6, etc. are interconnected by an internal bus or a dedicated communication line. Communication controller 120-1 communicates with components other than control device 120. FIG. The storage device 120-5 stores a program 120-5a executed by the CPU 120-2. This program is developed in RAM 120-3 by a DMA (Direct Memory Access) controller (not shown) or the like and executed by CPU 120-2. Thereby, part or all of the information processing section 122, the emotion estimation section 126, the position acquisition section 124, and the imaging processing section 128 are realized.

The embodiment described above can be expressed as follows.
a storage device storing a program;
a hardware processor;
By executing the program stored in the storage device, the hardware processor
Acquire the location information of the moving object,
Acquiring the emotion of the user of the mobile body,
acquiring a specific image captured when the position information is included in a preset imaging area and the acquired emotion of the user matches a preset specified emotion;
An imaging device configured to:

As described above, the mode for carrying out the present invention has been described using the embodiments, but the present invention is not limited to such embodiments at all, and various modifications and replacements can be made without departing from the scope of the present invention. can be added.

1, 1A image acquisition system 2 terminal device 4 imaging application 10 center server device 14 information processing unit 16 information providing unit 30 storage unit 34 imaging processing information 36 imaging Image information 100, 100A vehicle 102 communication device 104 HMI 106 microphone 108 occupant imaging camera 110 landscape imaging camera 120 control device 122 information processing unit 124 position acquisition unit , 126... Emotion estimation unit 128... Imaging processing unit 150... Storage unit 152... Imaging processing information 154... Imaged image information 170... Navigation device 300... Road side camera

Claims (8)

a first acquisition unit that acquires position information of a mobile object;
a second acquisition unit that acquires the emotion of the user of the mobile object;
a third acquiring a specific image captured when the position information is included in a preset imaging area and the emotion of the user acquired by the second acquisition unit matches a preset specified emotion; a third acquisition unit that acquires the specific image captured by a first imaging unit that captures an image of the user and is provided around a moving object on which the user has boarded or a road through which the moving object passes; Department and
An image acquisition system comprising: a first acquisition unit that acquires position information of a mobile object;
a second acquisition unit that acquires the emotion of the user of the mobile object;
a third acquiring a specific image captured when the position information is included in a preset imaging area and the emotion of the user acquired by the second acquisition unit matches a preset specified emotion; an acquisition unit;
an information providing unit that provides the specific image to the user ,
The third acquisition unit captures a first specific image captured by a first imaging unit that captures an image of the user, and a second image captured by a second imaging unit that is provided on the moving body and captures the surrounding environment of the user. Acquiring a second specific image and a third specific image captured by a third imaging unit that captures an image of the moving object in the imaging area;
The information providing unit provides the user with information in which the first specific image, the second specific image, and the third specific image are associated with each other.
Image acquisition system. The second acquisition unit starts a process of acquiring the user's emotion when the position information acquired by the first acquisition unit is included in the imaging area.
3. An image acquisition system according to claim 1 or 2 . The third acquisition unit acquires a specific image captured by a second imaging unit that is provided in the moving object and that captures an image of the surrounding environment of the user.
4. An image acquisition system according to any one of claims 1-3 . An information providing unit that provides the user with information in which the recommended imaging area or the imaging area designated by the user is associated with the map information,
5. An image acquisition system according to any one of claims 1-4 . information in which the area from which the specific image was obtained is associated with map information, information in which the area from which the specific image is obtained, the specific image, and the map information are associated with each other, or the specific image is obtained further comprising an information providing unit that provides the user with information in which the region and the movement trajectory of the moving body are associated with each other;
An image acquisition system according to any one of claims 1-5. the computer
Acquire the location information of the moving object,
Acquiring the emotion of the user of the mobile body,
A process for acquiring a specific image captured when the position information is included in a preset imaging area and when the acquired emotion of the user matches a preset specified emotion, Acquiring the specific image captured by a first imaging unit that is provided around a moving body on which a person has boarded or a road through which the moving body passes and captures an image of the user ;
Image acquisition method. to the computer,
Get the position information of the moving object,
Acquiring the emotion of the user of the mobile body,
A process for acquiring a specific image captured when the position information is included in a preset imaging area and the acquired emotion of the user matches a preset specified emotion, Acquiring the specific image captured by a first imaging unit that is provided around a moving body on which a person has boarded or a road through which the moving body passes and captures an image of the user ;
program.

Images