JP2013255168A - Imaging apparatus and imaging method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus which automatically takes a picture of an object on which an occupant of a vehicle shows an interest.SOLUTION: The imaging apparatus for photographing an object outside the vehicle to which an occupant of the vehicle pays attention comprises: sight detection means which detects the direction of the sight of the occupant; steady gaze area determination means which, on the basis of the direction of the sight of the occupant detected by the sight detection means, determines a steady gaze area being an area to which the occupant pays attention in the visual field area of the occupant and determines the direction of the steady gaze area with respect to the vehicle and the timing of photographing; and photographing means which photographs the direction of the steady gaze area with respect to the vehicle in accordance with the determined timing of photographing.

Description

  The present invention relates to an imaging apparatus and an imaging method.

  In recent years, SNS (social networking service) that can send messages to an unspecified number of users has become widespread as a service for realizing communication between users on the Internet. Since SNS can transmit information at any time through a portable information terminal, it is used by a wide range of users.

  There is a demand for using such a service to transmit information even when traveling by car. In particular, if a portable information terminal having a camera function is taken outside the vehicle and an image is uploaded, the topic can be shared with other users in real time. However, since the terminal cannot be operated during driving, when the user is driving a car, photographing cannot be performed unless the vehicle is stopped. Even when a passenger tries to take a picture, there is a case where the timing of taking a picture is missed while the camera is being prepared, so it is not easy to take a picture while traveling.

On the other hand, techniques for assisting photography while a vehicle is running include the following.
Japanese Patent Application Laid-Open No. 2004-26883 can issue a shooting instruction by voice input, and can keep track of the positional relationship with the shooting target even when the traveling direction of the vehicle changes during voice recognition. An imaging device is described. By using the photographing apparatus, the driver can photograph the object with a small physical burden.

  Patent Document 2 describes an information recording device that records the driver's line of sight at the time the photograph button is pressed, identifies the object being watched, and can photograph the object. ing. By using the information recording apparatus, the user can shoot a desired subject simply by giving a shooting instruction without performing an operation of pointing the camera toward the subject.

  Non-Patent Document 1 describes an in-vehicle robot having a photographing function. The robot has a camera that captures the inside and outside of the vehicle and a microphone that collects the sound inside the vehicle, and has a function of performing imaging by detecting a rise in the vehicle. That is, it is possible to automatically perform photographing by detecting that the vehicle occupant is interested in something without giving an explicit photographing instruction.

JP 2006-343829 A JP 2004-64409 A

Kohei Ito, Ryujiro Fujita, Naohiko Ichihara, Hiroaki Shibasaki, Nobuyuki Sato, Tomohiro Adachi, “In-vehicle Robot Prototype”, PIONEER R & D, Vol.17, No.1, pp.23-27, 2007

Taking a picture using a camera mounted on a vehicle and trying to collect the image is a problem, but many of the conventional technologies cannot take a picture unless an explicit instruction is given. Is a point. For example, in the inventions described in Patent Document 1 and Patent Document 2, a shooting instruction must be given by voice input, button press, or the like. That is, there is a problem that a cognitive load is imposed on the occupant because some action needs to be performed on the device.

  On the other hand, in the technique described in Non-Patent Document 1, since the trigger for shooting is determined from the conversation between passengers, the user can collect images without being aware of the shooting timing. However, since the occupant does not necessarily have a conversation during traveling, even if this technology is used, it is not always possible to take a picture at an accurate timing. Further, since there is no means for specifying what the occupant is interested in, the camera cannot be directed in a specific direction. That is, there has been a problem that it is not possible to completely satisfy the demand for reliably photographing an object that the user is interested in.

  Thus, in the prior art, there was a mechanism for detecting by voice that a vehicle occupant showed interest in something, and a mechanism for automatically determining a shooting target in response to a shooting instruction from the occupant, There has been no device that automatically captures and collects objects that the occupant has shown interest without placing a cognitive load on the occupant.

  The present invention has been made in view of the above-described problems, and an object thereof is to provide an imaging apparatus that automatically captures a photograph of an object in which a vehicle occupant has shown interest.

  In order to achieve the above object, the imaging apparatus according to the present invention takes an image of an object by the following means.

  An imaging apparatus according to the present invention is an imaging apparatus that captures an object outside a vehicle that is being watched by an occupant of a vehicle, the sight line detecting means for detecting the direction of the sight line of the occupant, and the occupant detected by the sight line detecting means. Gaze area determination that determines a gaze area that is an area that the occupant is paying attention from among the occupant's visual field area, and determines a direction of the gaze area with respect to the vehicle and a shooting timing. Means, and photographing means for photographing the direction of the gaze area with respect to the vehicle according to the determined timing.

  The line-of-sight detection means is a means for detecting the direction of the line of sight of the occupant. Specifically, an image for determining the direction of the line of sight of the occupant from the acquired image and an image pickup means such as a sensor or a camera installed inside the vehicle. It consists of processing means. Further, the gaze area determination unit is a unit that determines the shooting direction and the shooting timing based on the detected direction of the sight line of the occupant. Specifically, the gaze area that is the area that the occupant is paying attention to in the occupant's visual field area is determined from the acquired gaze direction, and in which direction the determined gaze area is relative to the vehicle. Simultaneously, the shooting timing is determined. With this configuration, it is possible to determine the direction of the object that the occupant is interested in based on the movement of the sight line of the occupant and automatically perform shooting.

  Further, the gaze area determination unit calculates a movement frequency or a residence time of the line of sight with respect to a specific region of the occupant's visual field area from the direction of the eye line detected by the eye detection unit, and the movement frequency or the residence time of the line of sight May be the gaze region when the value is larger than a predetermined value.

  When the occupant is gazing at a specific area within the field of view repeatedly for a certain period of time or at a certain frequency, there is an object in which the occupant is interested (hereinafter referred to as an object of interest). And the area is determined as a gaze area. This is because the object that the occupant is interested in is watched for a long time or is watched multiple times.

  The line-of-sight detection unit detects a plurality of occupant's line-of-sight directions, and the gaze region determination unit determines that the plurality of occupant's lines of sight are facing the same or similar directions. The direction of the gaze area with respect to the vehicle and the timing of shooting may be determined using the direction of the line of sight of at least one of the occupants.

  The presence / absence and direction of the target object may be determined by detecting the direction of the line of sight of a plurality of passengers riding on the vehicle. Specifically, when the directions of the lines of sight of a plurality of occupants are substantially the same, it is estimated that there is a target object in that direction. As the number of people looking in a certain direction increases, it means that there is an object that attracts more attention. By using the number of people as a criterion, it is possible to photograph the object of interest more accurately. it can.

  The imaging apparatus according to the present invention further includes a vehicle speed sensor that acquires a vehicle speed, and the gaze area determination unit is configured to determine a line-of-sight per unit time from a direction of the line of sight of the occupant detected by the line-of-sight detection unit. A direction change amount is calculated, and a relative distance between the vehicle and the object may be further determined from the change amount and a vehicle speed acquired from the vehicle speed sensor.

  The relative distance between the object of interest and the vehicle can be estimated by acquiring the amount of change per unit time in the direction of the detected line of sight and performing the calculation by adding the speed of the vehicle. The relative distance is not necessarily the shortest distance between the vehicle and the target object, and may be a distance for calculating the shortest distance by trigonometry. By acquiring the distance between the vehicle and the object before shooting, it is possible to perform moving object prediction and focusing in shooting.

  In addition, the photographing unit is configured to correct a direction of the gaze area with respect to the vehicle, which is corrected based on a relative distance between the vehicle and the object, a vehicle speed acquired from the vehicle speed sensor, and a time lag required for the photographing unit to perform photographing. It may be characterized by photographing.

  When there is a time lag between the time when shooting is determined and the time when an image is acquired, there is a problem that the object moves during that time and moves out of the frame. Therefore, the direction of the target object at the shooting timing is predicted and the shooting direction is corrected using the acquired relative distance to the target object and the speed of the vehicle. With this configuration, the target object can be surely contained in the frame.

  In addition, the imaging apparatus according to the present invention stores position information acquisition means for acquiring position information, an image taken by the photographing means, presents the stored image to a user, and selects an image to be processed. Image storage means, and the image storage means captures the position information acquired by the position information acquisition means and the relative distance between the vehicle and the object calculated by the gaze area determination means. The position information of the object may be generated and stored simultaneously with the captured image.

  The captured image may be stored in a storage unit and presented to the user later. By temporarily storing the photographed image, it is possible to select a necessary image and a non-required image after stopping. Further, when storing the image, position information of the object may be generated and stored in association with the image. With this configuration, it is possible to present to the user where the photographed object was, and to remind the user of the situation at the time of photographing.

  In addition, the imaging apparatus according to the present invention may further include a communication unit that transmits the stored image to a network. By transmitting the photographed image to the SNS or blog as “tweet” or “diary”, communication with other users can be promoted.

  ADVANTAGE OF THE INVENTION According to this invention, the imaging device which image | photographs automatically the photograph of the target object which the passenger | crew of the vehicle showed interest can be provided.

1 is a system configuration diagram of an imaging apparatus according to a first embodiment. It is a figure for demonstrating the arrangement position of the camera for eye-gaze detection. It is a processing flowchart of the imaging device concerning a first embodiment. It is a 1st figure explaining the determination method of the gaze area in a first embodiment. It is a 2nd figure explaining the determination method of the gaze area | region in 1st embodiment. It is an example of the screen which selects a transmission image in 1st embodiment. It is a figure explaining a mode that movement of a look is detected in a first embodiment. It is a figure for demonstrating the time lag in imaging | photography. It is a system block diagram of the imaging device which concerns on 2nd embodiment. It is a process flowchart of the imaging device which concerns on 2nd embodiment. It is a figure explaining the distance calculation with the target object in 2nd embodiment. It is a figure explaining the determination method of the gaze area | region in 3rd embodiment.

(First embodiment)
<System configuration>
The imaging device according to the first embodiment is an in-vehicle device that is mounted on an automobile and automatically captures an image outside the vehicle with an in-vehicle camera. FIG. 1 is a system configuration diagram of an in-vehicle device according to the first embodiment.

  The in-vehicle device 10 includes a line-of-sight information acquisition unit 12, a gaze area determination unit 13, a photographing unit 14, a storage unit 15, an input / output unit 16, and a communication unit 17. The line-of-sight camera unit 11 is connected to the line-of-sight information acquisition unit 12, the in-vehicle camera unit 18 is connected to the imaging unit 14, and the GPS device 22 is connected to the storage unit 15.

  The line-of-sight camera unit 11 includes a plurality of cameras for capturing images of the occupant's face. For example, multiple cameras corresponding to each occupant are installed, such as for the driver's seat, passenger's seat, left rear seat, and right rear seat, and each passenger's face image is acquired by visible light or infrared rays Can do.

FIG. 2 is a diagram for explaining the installation position of the camera in the vehicle. The camera may be provided in front of the vehicle such as a dashboard, a rearview mirror, a steering wheel, and a front pillar (A-pillar) neighborhood, for example, as indicated by reference numeral 201. Moreover, like the code | symbol 202, you may be equipped in the vehicle center parts, such as an intermediate pillar (B pillar), a side window, and the rear part vicinity of a front seat.
Moreover, it is preferable that the camera is provided at a position corresponding to each seat, as indicated by reference numerals 203 and 204. In the case of the example of FIG. 2, the left and right cameras denoted by reference numeral 203 correspond to the driver seat and the passenger seat, and the middle camera corresponds to the center of the rear seat. The left and right cameras denoted by reference numeral 204 correspond to the left and right rear seats, respectively. The camera arrangement in FIG. 2 is an example, but if the occupant's face can be captured, the camera can be arranged at an arbitrary location.

The line-of-sight information acquisition unit 12 is a unit that acquires an occupant's face image from the above-described camera and extracts the direction of the occupant's line of sight from the acquired face image, and is a line-of-sight detection unit in the present invention. For example, “Yasuhiro Ono, Takahiro Okabe, Yoichi Sato”, “Estimation of Gaze Direction from Low-Resolution Images”, IEICE Transactions, Vol.J90-D, No.8 , P.2212-2222, 2007 ”. In addition to this, it is possible to use existing techniques such as extracting facial feature points and specifying the position of the eyes and the position of the glow.

  The gaze area determination unit 13 is a means for determining the direction of the target object and the timing of shooting based on the gaze direction of one or more occupants extracted by the gaze information acquisition unit 12, and the gaze area determination in the present invention Means. Specific operations will be described later.

The photographing unit 14 is a means for photographing an image using the in-vehicle camera unit 18.
First, the in-vehicle camera unit 18 will be described. The in-vehicle camera unit 18 includes an external shooting camera mounted on the vehicle. One camera or a plurality of cameras may be mounted. In the case of a single camera, it is preferable to have a pan and tilt mechanism so that a designated direction can be photographed. When there are a plurality of cameras, it is preferable that the camera is composed of a plurality of cameras capable of photographing different angles, for example, for front, left and right sides, and rear. Of course, it may consist of a plurality of cameras having a pan and tilt mechanism.

  The imaging unit 14 is a unit that transmits an instruction for imaging the direction of the target object determined by the gaze area determination unit 13 to the in-vehicle camera unit 18. Specifically, after sending a command for changing the direction of the camera or a command for selecting a camera to be used among a plurality of cameras so that the direction determined by the gaze area determination unit 13 can be shot, Send.

The storage unit 15 is a disk device, a nonvolatile memory, or the like, and is means for temporarily storing captured images. The input / output unit 16 is a means for acquiring a captured image from the storage unit 15 and presenting it to the user, and accepting an operation on the image. The input / output unit 16 is assumed to be a touch panel screen, but may be configured with a screen and a keyboard or a screen and a pointing device as long as it can present an image to the user and accept an input operation.
The storage unit 15 is connected to a GPS device 22 for acquiring vehicle position information. The GPS device 22 has a function of acquiring the latitude and longitude of the current position and the name of the region where the vehicle is located. The storage unit 15 and the input / output unit 16 are image storage means in the present invention.

The communication unit 17 is means for transmitting the image stored in the storage unit 15 to the network. The communication unit 17 can perform communication with the outside using packet communication using a mobile phone network or wireless communication such as WiMAX (Worldwide Interoperability for Microwave Access). The image transmission destination may be a Web server or FTP server connected to the Internet, or a server that manages and operates the SNS. The communication destination is not necessarily a public network, and may be a wirelessly connected portable information device or the like.

<Process flowchart>
Next, the processing content is demonstrated, referring FIG. 3 which is a flowchart figure about the operation | movement which the vehicle-mounted apparatus 10 which concerns on 1st embodiment collects an image.
When the apparatus starts operation, the line-of-sight information acquisition unit 12 acquires the face image of the occupant from the connected camera and determines the direction of the line of sight (S11). In the first embodiment, only the line of sight of a person sitting in the passenger seat is detected, but a plurality of passengers may be targeted. When targeting a plurality of occupants, images corresponding to all seats may be acquired, and only images that can be determined to have a face are used. Information about the direction of the sight line of the occupant is transmitted to the gaze area determination unit 13.

Next, the gaze area determination unit 13 determines whether or not the gaze area, that is, the area that the occupant is paying attention can be specified from the acquired line-of-sight direction (S12).
This process will be described in detail. FIG. 4 is a diagram in which the locus of the line of sight of the occupant sitting in the passenger seat is projected onto a virtual projection surface 401 provided between the occupant and the camera. A broken line is a range of the windshield and represents a line of sight in which a curved line indicated by a dotted line moves. In FIG. 4, the projection surface is rectangular, but may have any shape and size as long as the movement of the line of sight of the occupant that can be captured by the camera can be expressed.

  In step S12, the gaze area determination unit 13 determines whether there is an area on the projection plane where it can be determined that the occupant is paying attention, that is, a gaze area. Specifically, the gaze area is determined by detecting the presence or absence of an area where the line of sight remains for a certain period of time. In the case of the example in FIG. 4, the region indicated by reference numeral 402 is a region that has been determined to have a long gaze retention time, and is a gaze region in this example. Hereinafter, in the description of the embodiments, “region” refers to a region on a virtual line-of-sight projection plane provided for each occupant.

The gaze area determination process will be described more specifically with reference to FIG.
FIG. 5 is a map in which the projection plane 401 is divided into a plurality of regions (blocks), and the time during which the line of sight stays within the latest unit time is given as a score for each region. In the present embodiment, when a line of sight stays for 3 seconds or more in the latest 10 seconds with respect to a certain area, the area is set as a gaze area. That is, the numerical value on the map represents the dwell time (seconds) of the line of sight in the last 10 seconds. If the gaze area determination unit 13 determines that there is a gaze area on the projection surface, the process proceeds to step S13. If it is determined that there is no gaze area, the process returns to step S11.

  In this embodiment, the line-of-sight projection plane is used as one method for specifying the direction of the target object. However, if it can be determined which direction outside the vehicle the occupant is watching, Other methods may be used. Further, the gaze area may be determined based on the movement frequency of the line of sight, that is, the number of times the line of sight enters and exits the area within the unit time, in addition to the gaze dwell time.

  In step S13, the gaze area determination unit 13 determines the direction of the target object with respect to the vehicle from the identified gaze area. The direction of the target object with respect to the vehicle can be obtained from the positional relationship between the position of the projection surface 401, the position of the gaze area 402, and the viewpoint of the passenger. For example, the determination is made that “the target object is in a direction of 340 degrees with respect to the vehicle” (when the angle is defined clockwise with the front of the vehicle being 0 degrees). The determined direction is transmitted to the imaging unit 14.

  In step S <b> 14, the photographing unit 14 gives a photographing instruction to the in-vehicle camera unit 18. When there is one on-vehicle camera, a command for directing the lens in the direction determined by the gaze area determination unit 13 and a command for performing shooting are transmitted continuously. When there are a plurality of in-vehicle cameras, a command for selecting a camera whose direction is determined by the gaze area determination unit 13 and a command for performing shooting are continuously transmitted. The captured image is stored in the storage unit 15. At this time, the storage unit 15 acquires information about the current position of the vehicle from the GPS device 22 and stores the information in association with the image.

Next, the processing after the image collection is completed will be described.
FIG. 6 is an example of a screen on which the input / output unit 16 presents a plurality of images stored in the storage unit 15 to the user and selects a processing target. On this screen, the shooting date and time and the shooting location are displayed together with the thumbnail of the image, and the user can select the desired image and perform operations such as transmission to the network and deletion. For example, when the user presses the upload button, the communication unit 17 transmits the selected image to the network, and when the user presses the transfer button, the selected image is transferred to the wirelessly connected portable information device. When uploading an image to the network, a comment input for the image may be accepted and transmitted simultaneously with the image.

  According to the first embodiment, it is possible to detect an occupant's line of sight, determine the direction of the target object, and automatically perform imaging. In addition, since the date / time and place where the images were acquired are displayed together with thumbnails, the user can organize the images while remembering the situation at the time of shooting. Note that the image display method is not limited to the exemplified method as long as the shooting date / time and location can be presented to the user. For example, a map may be displayed on the screen, and thumbnails may be displayed overlaid at the point where the image was taken.

  In the first embodiment, only the line of sight of the occupant seated in the passenger seat is acquired, but the lines of sight of a plurality of occupants seated in different seats are acquired, and the processes in steps S11 to S14 are performed in parallel. You may go. Further, the gaze area determination method may be changed depending on the seating position of the occupant. For example, for an occupant sitting in the driver's seat, the gaze area is determined using the movement frequency of the line of sight, and for the other occupants, the gaze area is determined using the dwell time of the line of sight. May be. This is because the driver is driving while paying attention to the front, and therefore, when the staying time of the line of sight is used, an erroneous determination may be caused.

Further, when the vehicle is traveling, even if attention is paid to the same object, the movement of the line of sight occurs, so that it may be determined whether or not photographing is necessary. FIG. 7 is a diagram (line-of-sight projection plane and score map) for explaining the movement of the line of sight. When the occupant is following an object outside the vehicle, the gaze area moves horizontally in the direction of the arrow. Therefore, the shooting process (S13 to S14) may be performed only when the gaze area is moving in the horizontal direction, and the shooting process may not be performed in other cases.
When the target object is far away, the movement amount of the line of sight for following the target object is small, and when it is close, the movement amount of the line of sight is large. Therefore, the required dwell time of the line of sight may be different for each region. In addition, even if the staying time of the line of sight for each region is less than or equal to the specified value, if the regions adjacent in the horizontal direction are continuously watched, it is determined that the object of interest is being followed, and shooting processing is performed. Also good.

(Second embodiment)
In the first embodiment, shooting is performed by determining the direction of the target object. However, when the target object is close to the vehicle, the direction of the target viewed from the vehicle is drastically changed. There is a possibility that the position will deviate from the expected. This point will be briefly described with reference to FIG.

For example, consider a case where an occupant of a traveling vehicle pays attention to an object denoted by reference numeral 801. Since the vehicle is running, the object gradually approaches and approaches the position indicated by reference numeral 802. At this time, it is assumed that the gaze area determination unit 13 detects the movement of the line of sight and gives an instruction for photographing. However, if there is a time lag in shooting and the object has moved to the position indicated by reference numeral 803 at the release timing, the object will be out of frame.
In order to deal with such a case, it is necessary to predict the movement of the object in advance and determine the shooting direction. That is, the direction calculated by the gaze area determination unit 13 must be corrected so that the direction indicated by reference numeral 803 can be photographed. In the second embodiment, the calculated shooting direction is corrected in order to minimize the influence of the time lag in shooting.

FIG. 9 shows a system configuration of the in-vehicle device according to the second embodiment. Moreover, FIG. 10 is a flowchart about the operation | movement which the vehicle equipment 10 which concerns on 2nd embodiment collects an image. In the second embodiment, the processes shown in steps S21 to S24 are added between steps S13 and S14.
The in-vehicle device 10 according to the second embodiment further includes a vehicle speed information acquisition unit 20 and a movement amount prediction unit 21. A vehicle speed sensor 19 is connected to the vehicle speed information acquisition unit 20.
The vehicle speed information acquisition unit 20 is means for acquiring vehicle speed information by acquiring vehicle speed pulses from the connected vehicle speed sensor 19. The movement amount prediction unit 21 is a unit that corrects the shooting direction determined by the gaze area determination unit 13 in consideration of a time lag when shooting. The movement amount prediction unit 21 also forms a gaze area determination unit in the present invention together with the gaze area determination unit 13.
The photographing unit 14 is different from the first embodiment in that an instruction to photograph the direction corrected by the movement amount predicting unit 21 is transmitted to the in-vehicle camera unit 18. Configurations other than the means described above are the same as those in the first embodiment.

The process following step S13 of the first embodiment will be described with reference to FIG. Steps S21 to S24 are steps for performing dynamic prediction of the target object.
First, the gaze area determination unit 13 acquires a change in the direction of the target object (S21). In step S21, the gaze area is detected a plurality of times during a unit time, the amount of movement of the gaze area is acquired, and the change in the direction of the target object with respect to the vehicle is acquired. For example, it can be determined that “at some point, when the target object is in the direction of 340 degrees with respect to the vehicle, it has moved in the direction of 250 degrees at the time of + n seconds”. The direction before and after the change and the required time are transmitted to the movement amount prediction unit 21.

  Next, the vehicle speed information acquisition unit 20 acquires the vehicle speed from the vehicle speed sensor 19 (S22). The vehicle speed can be obtained by taking the reciprocal of the vehicle speed pulse obtained from the vehicle speed sensor. The acquired speed is transmitted to the movement amount prediction unit 21.

Next, the movement amount prediction unit 21 acquires the distance between the target object and the vehicle (S23). A specific method will be described with reference to FIG. In this example, when the vehicle traveling at the speed ν is at the position (a), the angle between the target object and the vehicle is determined as θ _1, and after t seconds (
Consider a case in which the angle between the object of interest and the vehicle is determined to be θ ₂ when moving to b). In this case, x, which is the distance perpendicular to the road from the occupant to the object of interest at the time of (b), can be expressed as Equation 1. Also, y, which is the distance along the road from the occupant to the target object, can be expressed as Equation 2.

Next, the movement amount prediction unit 21 predicts the direction of the target object with respect to the vehicle at the photographing timing (S24). Here, if the time lag from when the angle θ ₂ is acquired to when the release is actually performed is T, the angle θ between the vehicle and the target object at the time when the release is performed is a value that satisfies Equation 3. The angle θ satisfying Equation 3 is transmitted to the photographing unit 14, and the photographing unit 14 transmits an instruction to photograph the direction of the angle θ to the in-vehicle camera unit 18.

  According to the second embodiment, even when the target object is located near the vehicle, it is possible to take a picture without causing the target object to be out of frame. In addition, since the relative distance between the passenger and the target object can be calculated by using the angle θ and the distance x, the distance may be used for focusing. Further, the position information (latitude / longitude) of the target object may be calculated using the position information of the vehicle acquired from the GPS device 22 and the calculated angle and relative distance. By storing the position information in the storage unit 15 in association with the image, the position information of the photographed object can be presented to the user together with the image.

(Third embodiment)
In the first embodiment, the direction of the target object is determined using the gaze dwell time or the line-of-sight movement frequency. However, when this method is used, the target object is excessively determined. There is. For example, since the driver's line of sight always pays attention to the traveling direction, there is a possibility that a front image will always be taken when the driver's line of sight is detected.

  FIG. 12 is an example of a line-of-sight projection plane for a driver and a score map corresponding to the projection plane. In the example of FIG. 12, when a line of sight enters and exits each rectangular area five times or more within a unit time, the area is set as a gaze area (the numerical value on the map is the number of times). In this example, the area 1202 is an area that should be extracted as the gaze area, but when the first embodiment is applied to the driver, the area 1201 that is the area of interest in normal driving is always extracted. And will be subject to photography. In order to deal with such a case, the third embodiment is a form in which an excluded area, that is, an area that is not treated as a gaze area is provided in the first embodiment. The system configuration of the in-vehicle device according to the third embodiment and the processes other than step S12 are the same as in the first embodiment.

  In the third embodiment, in step S12, the gaze area is determined by excluding the area that the occupant is continuously paying attention to. As a specific example, there is an area where the occupant keeps watching for a certain time or more (for example, 50% or more of the unit time), or an area where the line of sight has entered or exited a certain number of times (for example, 10 times or more) within the unit time, And when the vehicle is driving | running | working, it judges that it is not the gaze for tracking a target object, and memorize | stores the said area | region as an exclusion area | region. The excluded area is not treated as a gaze area even if the condition is met. If a target object appears during driving, the gaze area can be determined by the same processing as in the first embodiment.

  In the third embodiment, it is possible to more accurately determine the direction of the target object by excluding the region that is always watched during normal travel from the detection target. Note that even if an area is stored as an exclusion area, the designation of the exclusion area may be canceled if it has not been watched at all for a certain period of time (for example, 60 seconds or more).

(Fourth embodiment)
In the first embodiment, the direction of the target object is determined using the gaze retention time or the gaze movement frequency. However, when the target object is located near the vehicle, the gaze movement is fast and the gaze area May not be correctly identified. In order to cope with this, when it can be determined that a plurality of occupants are looking in the same direction, the form in which the corresponding area is determined as the gaze area regardless of the staying time and the movement frequency of the line of sight is the fourth embodiment. It is. The system configuration of the in-vehicle device according to the fourth embodiment and the processes other than steps S11 to S13 are the same as those in the first embodiment.

In the fourth embodiment, the line of sight of two or more passengers is detected in step S11. In step S12, a gaze area is specified for each occupant. At this time, the area where the line of sight is directed is always set as the gaze area, and the direction corresponding to the gaze area is calculated for each of the plurality of occupants in step S13. Then, when the calculated direction matches or is similar between a plurality of people (for example, two or more people), the direction is determined as the direction of the target object. For example, when the variation in the direction of the line of sight is within 10 degrees, shooting is performed by the method described in the first embodiment using the line of sight of at least one of the passengers.

  As described above, in the fourth embodiment, since the direction in which a plurality of occupants are gazing is regarded as the direction of the target object, shooting due to erroneous determination is unlikely to occur, and only the target object that the occupant is interested in is captured. Can be expected.

  Note that, in the fourth embodiment, as in the first embodiment, the photographing direction may be determined using the stay time of the line of sight and the movement frequency. For example, the required dwell time of the line of sight may be set as short as 1 second, and shooting may be performed when two or more people are simultaneously gazing at the same direction for 1 second or longer. As a result, it is possible to exclude a case where the line-of-sight directions coincide by chance.

(Modification)
The above description is merely an example for explaining the present invention, and each of the embodiments can be modified or combined in a range not departing from the gist of the present invention. In the description of each embodiment, the direction of the target object is specified only in the horizontal direction, but may be specified using the vertical direction. In this way, for example, even when the target object is in the sky, it is possible to take a picture by tilting the in-vehicle camera.

  Moreover, the vehicle-mounted camera 18 may be a camera that captures a moving image, and may be controlled to always face the direction of the target object determined by the gaze area determination unit 13. With this configuration, the in-vehicle camera can always capture the field of view seen by the driver, and can compensate for the drawbacks of conventional drive recorders that can only capture a fixed direction. it can.

  The storage unit 14 may automatically evaluate the stored image. For example, the stored image may be matched with a template image, and an image determined to include an object to be excluded, such as a traffic light or a road sign, may be suppressed from being presented to the user. You may delete automatically. Similar processing may be performed on an image that can be determined that the main subject is not captured. In addition, the importance may be calculated from the number of people who have paid attention and the time of gaze, and the image may be labeled.

DESCRIPTION OF SYMBOLS 10 In-vehicle apparatus 11 Eye-gaze camera part 12 Eye-gaze information acquisition part 13 Gaze area determination part 14 Imaging | photography part 15 Memory | storage part 16 Input / output part 17 Communication part 18 Car-mounted camera part 19 Vehicle speed sensor 20 Vehicle speed information acquisition part 21 Movement amount prediction part

Claims (8)

An imaging device for photographing an object outside a vehicle that is being watched by a vehicle occupant,
Line-of-sight detection means for detecting the direction of the line of sight of the occupant;
Based on the direction of the sight line of the occupant detected by the sight line detection means, a gaze area that is an area that the occupant is paying attention to is determined from the occupant's visual field area, the direction of the gaze area with respect to the vehicle, Gaze area determination means for determining timing;
Photographing means for photographing the direction of the gaze area with respect to the vehicle according to the determined timing;
An imaging apparatus having The gaze area determination unit calculates a movement frequency or a dwell time of the line of sight with respect to a specific area of the occupant's visual field area from the direction of the gaze detected by the line-of-sight detection unit, and the gaze movement frequency or the dwell time is predetermined. The imaging device according to claim 1, wherein the region is the gaze region when the value is larger than the value of the imaging region. The line-of-sight detection means detects the direction of the line of sight of a plurality of passengers,
When the gaze area determination unit determines that the plurality of occupants are facing the same or similar directions, the gaze area determination unit uses the gaze direction of at least one of the plurality of occupants to use the gaze area vehicle. The imaging apparatus according to claim 1, wherein a direction with respect to and a shooting timing are determined. A vehicle speed sensor for acquiring a vehicle speed;
The gaze area determination means calculates the amount of change in the direction of the line of sight per unit time from the direction of the line of sight of the occupant detected by the line of sight detection means, and from the amount of change and the vehicle speed acquired from the vehicle speed sensor, The imaging apparatus according to claim 1, further determining a relative distance between a vehicle and the object. The photographing means photographs the direction of the gaze area with respect to the vehicle corrected based on a relative distance between the vehicle and the object, a vehicle speed acquired from the vehicle speed sensor, and a time lag required for photographing by the photographing means. The imaging apparatus according to claim 4, wherein Position information acquisition means for acquiring position information;
Image storage means for storing an image captured by the imaging means, presenting the stored image to a user, and selecting an image to be processed;
Further comprising
The image storage means generates position information of the captured object from the position information acquired by the position information acquisition means and the relative distance between the vehicle and the object calculated by the gaze area determination means, The image pickup apparatus according to claim 4, wherein the image pickup apparatus is stored in association with the photographed image. The imaging apparatus according to claim 1, further comprising a communication unit that transmits an image captured by the imaging unit to a network. An imaging method performed by an imaging device that captures an object outside a vehicle that is being watched by a vehicle occupant,
Detecting the direction of the sight of the occupant;
Based on the detected direction of the line of sight of the occupant, a gaze area that is an area that the occupant is paying attention to is determined in the occupant's visual field area, and the direction of the gaze area with respect to the vehicle and the timing of shooting are determined. And steps to
Photographing the direction of the gaze area with respect to the vehicle according to the determined timing;
The imaging method characterized by including.

Images