JP5811918B2 - Gaze target estimation apparatus, method, and program - Google Patents

Description

  The disclosed technology relates to a gaze target estimation device, a gaze target estimation method, and a gaze target estimation program.

  2. Description of the Related Art Conventionally, an apparatus for estimating an object such as a signboard watched by a passenger who has boarded a moving body such as a vehicle is generally known. Such an apparatus analyzes the interest and interest of each passenger from the signboard watched by the passenger, and presents an effective advertisement, or displays an advertisement according to the degree of attention to the advertising agency presenting the signboard. It can be applied to digital signage, such as providing information such as pricing.

  As a simple example of the gaze target estimating apparatus as described above, the line-of-sight direction of the occupant of the moving body is estimated by the line-of-sight sensor, and the position of the moving body is acquired by the GPS sensor or the like. In addition, the installation position of an object such as a signboard is registered in advance in the database. And the apparatus which estimates which target object the passenger was gazing in three-dimensional space from these positional information can be considered. However, when estimating the gaze position three-dimensionally in this way, the installation position of the object, the position of the moving object, and the line-of-sight direction are each three-dimensionally affected as error factors. It is difficult to estimate the target object.

  Thus, an apparatus for estimating an object using a two-dimensional image is provided. Such an apparatus includes a line-of-sight sensor that estimates a line-of-sight direction of a passenger on a moving body, and a camera that captures the periphery of the moving body. And from the position corresponding to the sight line direction of the passenger on the image obtained by photographing with the camera, and the position on the image of the object such as a signboard recognized by performing image processing on the image, Estimates the object watched by the passenger.

  For example, the gaze state of the passenger's gaze behavior is determined from the temporal change in the passenger's visual field image generated sequentially every predetermined time, and if the gaze behavior is the gaze state, it is included in the gaze target object watched by the passenger There has been proposed a gaze target identifying device that recognizes character information on a viewed image.

  Further, the front image data, the line-of-sight direction data, and the imaging position coordinates are recorded in the memory, the image data corresponding to the attention area determined based on the line-of-sight direction data is extracted from the front image data, and the attention area is displayed on the liquid crystal display. Driving support systems that display images have been proposed.

JP 2007-172378 A JP 2006-172215 A

  However, for example, a vehicle occupant generally looks at the front of the vehicle so as not to look aside, and often looks at an object installed in front of the object such as a signboard. Further, the line-of-sight sensor for detecting the passenger's line-of-sight direction has high detection accuracy for the line-of-sight direction when the passenger is looking at a position corresponding to the vicinity of the center of the image. On the other hand, an object such as a signboard in front of the occupant who is gazing away from the camera has a smaller distance on the image. Since a camera mounted on a moving body such as a current on-vehicle camera has a low resolution, there is a problem that it is difficult to recognize an object having a small size on an image by image processing. On the other hand, an object having a large size on an image is an object installed at a short distance such as a lateral direction or obliquely forward for a passenger. Therefore, since the passenger of the vehicle looks at the front of the vehicle so as not to look aside, there is a problem that the possibility that the passenger is watching the object at such a short distance is low.

  An object of the disclosed technology is to estimate an object watched by the passenger even when the size of the object watched by the passenger boarding the moving body is small.

  The disclosed technology acquires each of image data of an image obtained by photographing the periphery of the moving object at each time, and each of the line-of-sight data indicating the line-of-sight direction of the passenger of the moving object at each time A data acquisition unit is provided. Further, the disclosed technology is a determination unit that determines the gaze position of the occupant in the image indicated by the image data at each time based on the image data and the line-of-sight data acquired by the data acquisition unit. It has. In addition, the disclosed technology includes a detection unit that detects an object region corresponding to a predetermined object from an image indicated by the image data. In addition, the disclosed technology includes an estimation unit that estimates whether or not the passenger is gazing at the object. The estimation unit includes: a position in the past image of the target region tracked from a past image taken at a time earlier than the image in which the target region is detected by the detection unit; and the determination unit. The determined gaze position of the passenger in the past image is compared.

  The disclosed technology has an effect that it is possible to estimate an object watched by the passenger even when the size of the object watched by the passenger boarding the moving body is small.

It is a functional block diagram of the gaze target estimating device concerning this embodiment. It is an image figure for demonstrating the detection of (a) target object area, (b) tracking of target object area, and (c) tracking position and gaze position. It is a schematic block diagram of the computer which functions as a gaze target object estimation apparatus. It is a flowchart of the gaze target object estimation process in this embodiment. It is a flowchart of the target object detection process in this embodiment. It is a flowchart of the retroactive range determination process in this embodiment. It is a flowchart of the object area | region tracking process in this embodiment. It is a flowchart of the tracking position / gaze position comparison process in the present embodiment. It is a functional block diagram which shows the other example of the gaze target object estimation apparatus which concerns on this embodiment.

  Hereinafter, an example of an embodiment of the disclosed technology will be described in detail with reference to the drawings.

  [First Embodiment]

  FIG. 1 shows a gaze target estimating apparatus 10 according to the present embodiment. The gaze target estimation device 10 is a device that estimates a target gaze by a passenger of a moving object. This embodiment demonstrates the case where the technique of indication is applied to the gaze target object estimation apparatus which estimates the signboard installed in the vehicle and the surroundings of the driving environment of the vehicle which the driver of the vehicle gazes at.

  The gaze target estimation device 10 includes a data acquisition unit 12, an image data storage unit 14, a gaze position determination unit 16, a gaze position data storage unit 18, a target detection unit 20, a target information storage unit 22, an estimation unit 24, a target An object information acquisition unit 26, an analysis unit 28, and a feedback unit 30 are provided. The gaze position determination unit 16 is an example of a determination unit in the disclosed technology, and the object detection unit 20 is an example of a detection unit in the disclosed technology.

  The data acquisition unit 12 includes position data indicating the current position of the vehicle detected by the GPS sensor 60, image data indicating an image obtained by photographing the periphery of the vehicle with the camera 62, and the vehicle detected by the line-of-sight sensor 64. The line-of-sight data indicating the line-of-sight direction of the driver is acquired. The camera 62 captures the same range as the field of view in the front of the vehicle, at least when the driver is facing forward. The line-of-sight sensor 64 can be constituted by, for example, a camera that captures an area including the driver's face, and can detect the direction of the driver's line of sight from a captured image using a known technique. . The data acquisition unit 12 acquires each data synchronously at each time.

  The image data storage unit 14 stores image data 32 acquired at each time by the data acquisition unit 12 in association with the time. In the image data storage unit 14, position data 34 acquired at the same time as the image data 32 is stored in association with the image data 32.

  The gaze position determination unit 16 determines the gaze position of the driver in the image indicated by the image data 32 at each time based on the image data 32 and the line-of-sight data 34 acquired by the data acquisition unit 12. A known technique can be used as a method for determining the gaze position. For example, when the temporal change of the line-of-sight position within a predetermined time with reference to a certain time is small using the time change of the line-of-sight position on the image indicating the driver's line-of-sight direction, the area around the line-of-sight position is set as the gaze position. Can be determined. The gaze position determination unit 16 stores the coordinates of a predetermined position (for example, the center) of the area determined as the gaze position in the gaze position data storage unit 18 as gaze position data 36 in association with the time.

  The image data storage unit 14 and the gaze position data storage unit 18 may store each data acquired at each time as much as possible, but at least as a retrospective range in the estimation unit 24 described later. Each data of the past time to be referred to is stored.

  The object detection unit 20 refers to the object information 38 stored in the object information storage unit 22 and detects a signboard to be detected from the image indicated by the image data 32 acquired by the data acquisition unit 12. The object information storage unit 22 stores object information, which is information used to identify the signboard that was being watched when the driver of the vehicle was watching the signboard while the vehicle was running. . For example, the object information includes an installation position and size of the corresponding signboard in the real world and a template for detecting the signboard from the image. Moreover, the detailed information and the related information which show the content of the signboard for using with the target object information acquisition part 26 and the analysis part 28 mentioned later are also contained.

  The object detection unit 20 compares the installation positions of the signboards included in the position data 34 acquired by the data acquisition unit 12 and the object information 38 stored in the object information storage unit 22, and the signboards present around the vehicle. Extract a template. Note that only signs that may enter the driver's field of view may be extracted as signs around the vehicle. In addition, the object detection unit 20 detects an object region indicating a signboard from the image by a matching process between the image acquired by the data acquisition unit 12 and the extracted template. Note that the detection method of the object region is not limited to template matching, and other methods may be used. For example, when a character on the image is recognized and matches a registered word, the periphery of the character may be detected as a target area. In the present embodiment, since the sign is detected from an image taken when the sign reaches the vicinity of the vehicle, for example, as shown in FIG. It can be detected with accuracy.

  The estimation unit 24 further includes a retrospective range determination unit 24a, an object region tracking unit 24b, and a tracking position / gaze position comparison unit 24c. The estimation unit 24 determines whether the passenger has been gazing at the object detected by the object detection unit 20 in the past, and the image data 32 and the gaze position data storage unit 18 stored in the image data storage unit 14. The gaze position data 36 stored in is referred to as appropriate. Hereinafter, each part is explained in full detail.

  When the object detection unit 20 detects the object area, the retrospective range determination unit 24a starts from the time when the image in which the object area is detected is captured with respect to the image data 32 and the corresponding gaze position data 36. Determine the reference range in the retroactive direction. Hereinafter, this range is referred to as “retroactive range”. The retroactive range may be until processing by the object area tracking unit 24b described later becomes impossible (the signboard disappears on the photographed image), but may be determined as follows, for example. The size of the character of the guide sign is set as, for example, a standard 20 cm. If the eyesight is 0.7 (sight required for obtaining a driver's license), the Landolt ring of this size is recognized from about 100 m ahead. it can. From this, it is possible to set the range of time corresponding to the position where the mobile body is 100 m back from the current position as the retroactive range. Specifically, assuming that the vehicle is traveling at a speed of about 40 km / h, it is assumed that the vehicle travels 100 meters, that is, about 10 seconds before, and 10 seconds before the time when the image where the sign is detected is captured. Can be determined as a retroactive range.

  The object region tracking unit 24b acquires from the image data storage unit 14 image data 32 indicating images taken at each time within the retroactive range determined by the retrospective range determining unit 24a. Hereinafter, an image in which a signboard is detected by the object detection unit 20 is referred to as a “detected image”, and an image acquired as an image at each time within the retroactive range is referred to as a “past image”. Based on the position (coordinates) of the object area in the detected image, the object area tracking unit 24b moves the object from the past image at each time in the direction of retroactively extending the retroactive range from the shooting time of the detected image. By detecting the area, the position of the object area is tracked. This is shown in FIG. Even if the size of the object area on the image is small at the past time and it is difficult to recognize the image, the object area detected when the signboard comes to the front is traced back in the past, so that the object area is small. It is also possible to detect an object region having a size.

  As with the object detection unit 20, template matching can be used as a method for detecting the object region from each past image. At this time, the template used in the detection of the signboard is appropriately modified, and template matching is performed on the past image. As a template deformation method, for example, from the installation position of the detected signboard, the position of the vehicle at the time when each of the detected image and the past image was photographed, at the time when each of the detected image and the past image was photographed. Estimate the size ratio of the object area. Then, the template is reduced by the estimated ratio.

  The tracking position / gaze position comparison unit 24 c acquires gaze position data 36 at each time within the retroactive range determined by the retrospective range determination unit 24 a from the gaze position data storage unit 18. The tracking position / gaze position comparing unit 24c also monitors the position of the object region tracked from each past image captured at each time within the retroactive range by the object region tracking unit 24b and the gaze at the corresponding time. Compare with position. This is shown in FIG. Based on the comparison result, the tracking position / gaze position comparing unit 24c determines whether or not the driver is gazing at the detected signboard at any time within the retroactive range. For example, if the distance between the position of the object area and the gaze position is equal to or less than a predetermined distance, it can be determined that the driver is gazing at the signboard indicating the detected object area.

  When a plurality of target areas are detected by the target detection unit 20, the processing of the estimation unit 24 may be performed in parallel or in order for each detected target area.

  The target object information acquisition unit 26 acquires target object information from the target object information storage unit 22 for the signboard that has been determined by the tracking position / gaze position comparison unit 24c to be watched by the driver. For example, detailed information and related information of a signboard stored in the object information storage unit 22 corresponding to the template used at the time of detection by the object detection unit 20 are acquired.

  The analysis unit 28 analyzes the interest and interest of the driver based on the object information acquired by the object information acquisition unit 26 and the comparison result by the tracking position / gaze position comparison unit 24c. For example, based on detailed information and related information of a signboard, a product advertised on a signboard watched by a driver can be analyzed as an object of interest of the driver.

  The feedback unit 30 feeds back information based on the analysis result by the analysis unit 28 to the driver. For example, information on a product analyzed as an object of interest of the driver can be fed back by being displayed on the display 52 (see FIG. 3).

  The gaze target object estimation device 10 can be realized by a computer 40 shown in FIG. 3, for example. The computer 40 includes a CPU 42, a memory 44, a nonvolatile storage unit 46, a keyboard 50, a display 52, and a communication IF (interface) 54, which are connected to each other via a bus 56. The GPS sensor 60, the camera 62, and the line-of-sight sensor 64 are connected to the computer 40 via the communication IF 54.

  The storage unit 46 can be realized by an HDD (Hard Disk Drive), a flash memory, or the like. In the storage unit 46 as a recording medium, a gaze target estimation program 70 for causing the computer 40 to function as the gaze target estimation device 10, image data 32, position data 34, gaze position data 36, and target object information 38. Is stored. The CPU 42 reads the gaze target object estimation program 70 from the storage unit 46 and develops it in the memory 44, and sequentially executes the processes of the gaze target object estimation program 70.

  The gaze target estimation program 70 includes a data acquisition process 72, a gaze position determination process 74, a target detection process 76, an estimation process 78, a target information acquisition process 80, an analysis process 82, and a feedback process 84. The CPU 42 operates as the data acquisition unit 12 illustrated in FIG. 1 by executing the data acquisition process 72. The CPU 42 operates as the gaze position determination unit 16 illustrated in FIG. 1 by executing the gaze position determination process 74. Further, the CPU 42 operates as the object detection unit 20 illustrated in FIG. 1 by executing the object detection process 76. The CPU 42 operates as the estimation unit 24 illustrated in FIG. 1 by executing the estimation process 78. The CPU 42 operates as the object information acquisition unit 26 shown in FIG. 1 by executing the object information acquisition process 80. The CPU 42 operates as the analysis unit 28 illustrated in FIG. 1 by executing the analysis process 82. Further, the CPU 42 operates as the feedback unit 30 illustrated in FIG. 1 by executing the feedback process 84.

  When the gaze target estimating apparatus 10 is realized by the computer 40, the storage unit 46 that stores the image data 32 and the position data 34 is used as the image data storage unit 14, and a partial area of the memory 44 is the image data 32 and the position. Used as a storage area for data 34. The storage unit 46 that stores the gaze position data 36 is used as the gaze position data storage unit 18, and a partial area of the memory 44 is used as a storage area for the gaze position data 36. The storage unit 46 that stores the object information 38 is used as the object information storage unit 22, and a partial area of the memory 44 is used as a storage area of the object information 38. As a result, the computer 40 that has executed the gaze target estimation program 70 functions as the gaze target estimation apparatus 10.

  Note that the gaze target estimation apparatus 10 can be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit) or the like.

  Next, a gaze target estimation process performed by the gaze target estimation apparatus 10 according to the present embodiment will be described with reference to the flowchart of FIG. Note that this gaze target object estimation process is repeatedly executed by the gaze target object estimation device 10 at regular time intervals.

  First, in step 100, the data acquisition unit 12 acquires position data (hereinafter referred to as “position Pt2”) indicating the position of the vehicle at time t2 detected by the GPS sensor 60. Further, the data acquisition unit 12 acquires image data (hereinafter referred to as “image It2”) indicating an image obtained by photographing the periphery of the vehicle at time t2 with the camera 62. The data acquisition unit 12 acquires line-of-sight data (hereinafter referred to as “line of sight Et2”) indicating the line-of-sight direction of the driver of the vehicle at time t2 detected by the line-of-sight sensor 64. Further, the data acquisition unit 12 stores the image It2, the position Pt2, and the time t2 in association with each other in the image data storage unit 14.

  Next, in step 102, the gaze position determination unit 16 determines a gaze position on the image It2 using a known technique based on the image It2 and the line of sight Et2 acquired by the data acquisition unit 12. The coordinates of the determined gaze position (hereinafter referred to as “gaze position Gt2”) are stored in the gaze position data storage unit 18 in association with the time t2.

  Next, in step 104, the target object detection unit 20 executes a target object detection process for detecting a target object region from the image. Here, the object detection processing will be described with reference to FIG.

  In step 200, the object detection unit 20 acquires the image It2 and the position Pt2 from the image data storage unit 14. Next, in step 202, the object detection unit 20 compares the acquired position Pt2 with the signboard installation position stored in the object information storage unit 22, and extracts a signboard template existing around the position Pt2. . For example, a signboard installed at a position close to the position Pt2 may be extracted. Here, the distance is short, for example, within 100 m from the position Pt2. As described above, 100 m means that the characters on the guide sign can be confirmed. When there are a plurality of signboards that meet the conditions, a plurality of corresponding templates are extracted, and a number Bk (k = 1,..., N, N is the number of extracted templates) is assigned to each template. And stored in a predetermined area of the memory 44.

  Next, in step 204, the object detection unit 20 sets 1 to the variable i corresponding to the number assigned to the detected object area. Next, the process proceeds to step 206, and the object detection unit 20 executes a loop process indicated by steps 208 to 216 in order from k = 1 for the template Bk. In the loop process, in step 208, the object detection unit 20 acquires the template Bk stored in a predetermined area of the memory 44. Next, in step 210, the object detection unit 20 performs matching processing between the image It2 and the template Bk, and calculates the similarity between each region on the image It2 and the template Bk.

  Next, in step 212, the object detection unit 20 acquires the maximum similarity that is the maximum value of the similarity calculated in step 210, and in the next step 214, the maximum similarity is greater than a predetermined threshold. It is determined whether or not. When the maximum similarity is larger than the threshold, the process proceeds to step 216, and it is recorded that the object area Ai indicating the signboard ai corresponding to the template Bk is detected. Here, while recording as Ai = Bk, the variable i is incremented by one. As a result, when the object region detection processing is completed for all of the extracted templates Bk, a list of object regions indicating signboards detected from the image It2 is obtained as Ai. If the maximum similarity is less than or equal to the threshold value, it is determined that there is no object region that matches the template Bk in the image It2, and step 216 is skipped. Note that the predetermined threshold may be experimentally set in advance.

  When k = N, that is, when the object region detection process is completed for all of the extracted templates Bk, the loop process is determined to end in step 218, and the gaze object estimation process (FIG. Return to 4).

  When the process returns to the gaze target object process, the process proceeds to step 106, and the retroactive range determination unit 24a determines whether or not a target area at time t2 is detected in the target object detection process of step 104. When the object area is not detected, the gaze object estimation process ends. When the object area is detected, the process proceeds to step 108, and the estimation unit 24, the object information acquisition unit 26, the analysis unit 28, and the feedback unit 30 sequentially perform steps from i = 1 for the object area Ai. A loop process indicated by 110 to 122 is executed.

  In the loop processing, in step 110, the retroactive range determining unit 24a executes a retroactive range determining process for determining the retroactive range. Here, the retroactive range determination process will be described with reference to FIG.

  First, in step 300, the retroactive range determining unit 24a sets a threshold D for the moving distance of the vehicle. As described above, the threshold value D can be set to 100 m, for example. Next, in step 302, the retroactive range determination unit 24a acquires the position Pt2 from the image data storage unit 14. Next, the process proceeds to step 206, where the retroactive range determining unit 24a performs the loop process indicated by steps 306 to 310 for each time t that reaches the time t = tmin by going back in the past by a fixed time from the time t = t2. Run. In the loop processing, in step 306, the retroactive range determination unit 24a acquires the position Pt of the vehicle at time t from the image data storage unit 14, and in the next step 308, the moving distance of the vehicle between time t2 and time t. Dt (Dt = | Pt−Pt2 |) is calculated. Next, in step 310, the retroactive range determining unit 24a determines whether or not the movement distance Dt is less than the threshold D set in step 300. If Dt <D, the loop process is repeated for the next past time t. If Dt ≧ D, the loop process is exited and the process proceeds to step 314. Even if t is tmin without a negative determination in step 310, the end of the loop process is determined in step 312 and the process proceeds to step 314. Note that, for example, tmin is set in advance as the time until the object area tracking unit 24b becomes unprocessable (the signboard disappears on the photographed image) as described above.

  In step 314, the retroactive range determining unit 24a determines the retroactive range t2 to t1 with the current processing time t as the retroactive end time t1 and the time t2 as the retroactive start time, and gaze target object estimation processing (FIG. 4). Return to

  When returning to the gaze target processing, the process proceeds to step 112, and the target region tracking unit 24b executes target region tracking processing for tracking the target region in the past image in the retroactive range. Here, the object region tracking process will be described with reference to FIG.

  In step 400, the object area tracking unit 24 b sets the retroactive range t <b> 2 to t <b> 1 determined in the retroactive range determining process in step 110 as a processing range. Next, in step 402, the object area tracking unit 24b acquires the position Pt2 from the image data storage unit 14, and in the next step 404, acquires the object area Ai detected in step 104. In step 406, the object area tracking unit 24b acquires the template T corresponding to the object area Ai from the object information storage unit 22. Next, in step 408, the object area tracking unit 24b acquires the installation position Ri of the signboard ai indicated by the object area Ai from the object information storage unit 22.

  Next, the process proceeds to step 410, and the object region tracking unit 24b executes a loop process indicated in steps 412 to 420 for each time t within a retroactive range in the direction retroactive from t = t2. In the loop processing, in step 412, the object region tracking unit 24 b acquires the image It and the position Pt at time t from the image data storage unit 14. Next, in step 414, the object area tracking unit 24b calculates a reduction ratio c of the template. Since the vehicle is moving forward and the object ai exists farther from the own vehicle than the time t2 at the time t retroactive from the time t2, the size of the object area Ai on the image It is indicated on the detection image It2. Smaller than the size of. Accordingly, the reduction rate c when generating the template for the past image by reducing the template T of the object area Ai at the time of detection is calculated by, for example, the following equation (1).

          c = | Pt2-Ri | / | Pt-Ri | (1)

  The reduction ratio c corresponding to the distance from the vehicle to the installation position of the signboard can be obtained from equation (1).

  Next, in step 416, the object region tracking unit 24 b generates a template T ′ obtained by reducing the template T acquired in step 406 using the reduction ratio c calculated in step 414. Next, in step 418, the object region tracking unit 24b performs a matching process between the image It and the template T ', and calculates the similarity between each region on the image It and the template T'. Next, in step 420, the object region tracking unit 24b determines the position on the image It where the similarity calculated in step 418 is maximized, that is, the position matching the template T ′ as the object region Ai at time t. Detect as. Then, the position (coordinates) is stored in a predetermined area of the memory 44 as the tracking position Qt. By storing the matched position as Qt, the tracking result of the object area Ai is obtained as Qt2 to Qt1 when the retroactive processing from t2 to t1 is completed. That is, the tracking position Qt is obtained at a certain time t (t2 to t1).

  When t = t1, that is, when the tracking process of the object region is completed for each time in the retroactive range, it is determined in step 422 that the loop process is finished, and the gaze target object estimation process (FIG. 4). Return to

  When the process returns to the gaze target processing, the process proceeds to step 114, and the tracking position / gaze position comparison unit 24c executes tracking position / gaze position comparison processing for comparing the tracking position of the target area with the gaze position. Here, the tracking position / gaze position comparison process will be described with reference to FIG.

  In step 500, the tracking position / gaze position comparing unit 24c sets the retroactive range t2 to t1 determined in the retroactive range determining process in step 110 as a processing range. Next, the process proceeds to step 502, and the tracking position / gaze position comparing unit 24c executes a loop process indicated by steps 504 to 512 for each time t within a retroactive range in the direction retroactive from t = t2.

  In the loop processing, in step 504, the tracking position / gaze position comparison unit 24c acquires the tracking position Qt at time t stored in a predetermined area of the memory 44, and in the next step 506, the tracking position / gaze position comparison unit 24c The gaze position Gt at time t is acquired. Next, in step 508, the tracking position / gaze position comparison unit 24c sets a threshold value d for the inter-pixel distance. The threshold value d is set in advance, for example, within the size of the template T ′ generated in step 416 of the object tracking process.

  Next, in step 510, the tracking position / gaze position comparing unit 24c calculates an inter-pixel distance dt (dt = | Qt−Gt |) between the tracking position Qt and the gaze position Gt. Next, in step 512, the tracking position / gaze position comparing unit 24c determines whether or not the inter-pixel distance dt is larger than the threshold value d set in step 508. If dt> d, the loop process is repeated for the next past time t. If dt ≦ d, the loop process is exited and the process proceeds to step 516.

  In step 516, the tracking position / gaze position comparing unit 24c determines that the driver is gazing at the sign ai at time t, and returns to the gaze target estimation process (FIG. 4). Also, when t becomes t1 without a negative determination in step 512, the loop process is determined to be ended in step 514, and the process returns to the gaze target estimation process (FIG. 4). Note that step 516 may be performed in a loop process, and it may be determined whether or not the object ai is being watched at all times within the retroactive range.

  When the process returns to the gaze target processing, the process proceeds to step 116, and the target object information acquisition unit 26 determines whether or not it is determined in the tracking position / gaze position comparison process in step 114 that the driver is gazing at the signboard ai. judge. If it is determined that the user is gazing, in step 118, the object information acquisition unit 26 acquires object information from the object information storage unit 22 for the sign ai that is determined to be being watched by the driver.

  Next, in step 120, the analysis unit 28 analyzes the interest and interest of the driver based on the object information acquired in step 118 and the comparison result in step 114. Next, in step 122, the feedback unit 30 feeds back information based on the analysis result in step 120 to the driver.

  When i = n (the number of target areas detected from the image It2), that is, when the gaze target determination process is completed for all detected target areas Ai, a loop process is performed in step 124. Is determined to end the gaze target estimation process.

  As described above, according to the gaze target estimating apparatus 10 of the present embodiment, when a target area indicating a signboard is detected from an image obtained by shooting the periphery of the vehicle, the shot is taken at a past time. The object area is tracked on the recorded image. Then, the position of the tracked object area is compared with the gaze position at the past time, and it is determined whether or not the driver has gazes at the signboard. Therefore, even when the size of the signboard watched by the driver is small, the signboard watched by the driver can be estimated.

  In the above-described embodiment, the case where the gaze target estimation device 10 is mounted on a vehicle has been described. However, the gaze target estimation device 10 may be configured as a server device installed in a management center or the like. In this case, each data acquired by the GPS sensor 60, the camera 62, and the line-of-sight sensor 64 mounted on the vehicle may be acquired by the data acquisition unit 12 via a network or the like.

  Further, when the disclosed technology is configured as a server device, as illustrated in FIG. 1, the configuration is not limited to a configuration in which all the components are closed in the terminal. For example, some storage units and processing units are provided on the in-vehicle device side, and other storage units and processing units are provided on the server side, and the above-described gaze target object estimation process is executed while communicating between the on-vehicle device and the server device. You may do it. An example of a schematic configuration in this case is shown in FIG. In the example of FIG. 9, the data acquisition unit 12, the image data storage unit 14, the gaze position determination unit 16, the gaze position data storage unit 18, the object detection unit 20, and the estimation unit 24 are provided on the in-vehicle device 10a side. The object information storage unit 22, the object information acquisition unit 26, the analysis unit 28, and the feedback unit 30 are provided on the server device 10b side. Note that FIG. 9 is an example, and any analogized device configuration is included in the disclosed technology.

  In addition, when a part or all of the configuration is provided on the server device side, information on a plurality of different drivers can be collected, so that the analysis unit 28 can also analyze data such as the degree of attention of the signboard. For example, for each signboard, the number of drivers who watched the signboard is counted, and the greater the number of drivers watched, that is, the more closely watched by various drivers, the more noticeable the signboard is. it can. Then, the feedback unit 30 can feed back data such as the degree of attention for each signboard installed by the advertising agency via the network to the advertising agency.

  Moreover, although the said embodiment demonstrated the case where a detection target object was made into a signboard, it is not limited to this, A road sign etc. is good also as a detection target object. In this case, it is possible to analyze the ease of visually recognizing the sign, or to feed back information such as an oversight of the sign to the driver. Further, the passenger is not limited to the driver but may be a passenger who has boarded the passenger seat, and the moving body is not limited to the vehicle, and may be a train or the like.

  Moreover, although the said embodiment demonstrated the case where the object area | region was detected by extracting the template corresponding to the signboard installed in the vehicle periphery using the position of a vehicle, it is not limited to this. The object region may be detected using all or some of the registered templates without using the vehicle position information.

  Moreover, although the said embodiment demonstrated the case where the retroactive range was determined uniformly, you may change a retroactive range according to the largest character size etc. in the signboard here.

  In the above description, the gaze target estimation program 70 is stored (installed) in the storage unit 46 in advance. However, the gaze target estimation program 70 is recorded on a recording medium such as a CD-ROM or a DVD-ROM. It is also possible to provide it in the form in which it is provided.

  All documents, patent applications and technical standards mentioned in this specification are to the same extent as if each individual document, patent application and technical standard were specifically and individually stated to be incorporated by reference. Incorporated by reference in the book.

  Regarding the above embodiment, the following additional notes are disclosed.

(Appendix 1)
A data acquisition unit that acquires each of image data of an image obtained by photographing the periphery of the moving object at each time, and each of the line-of-sight data indicating the line-of-sight direction of the passenger of the moving object at each time; ,
A determination unit that determines the gaze position of the passenger in the image indicated by the image data at each time based on the image data and the line-of-sight data acquired by the data acquisition unit;
A detection unit for detecting an object region corresponding to a predetermined object from an image indicated by the image data;
The position in the past image of the target region tracked from the past image taken at a time earlier than the image in which the target region is detected by the detection unit, and the determination unit An estimation unit that estimates whether or not the passenger was gazing at the object at the past time based on the gazing position of the passenger in a past image;
Gaze target estimation device including

(Appendix 2)
The estimator has a retroactive range indicating whether the object area is to be traced on the past image from the time when the image where the object area was detected was captured, to the time when the passenger The gaze target estimation apparatus according to supplementary note 1, wherein the gaze target estimation apparatus is determined based on a distance at which the target can be recognized and a moving speed of the moving body.

(Appendix 3)
The object corresponding to the object determined by the estimating unit to be watched by the occupant from the object information storage unit that stores the object information including at least one of the detailed information and the related information of the object The gaze target estimation device according to supplementary note 1 or supplementary note 2, including a target information acquisition unit that acquires target information.

(Appendix 4)
Based on the information related to the passenger estimated to have watched the object by the estimating unit and the object information acquired by the object information acquiring unit, the interest and interest of the passenger are analyzed. The gaze target object estimation device according to supplementary note 3, including an analysis unit.

(Appendix 5)
The gaze target estimation apparatus according to supplementary note 4, including a feedback unit that feeds back information based on an analysis result by the analysis unit to the passenger or a related business operator of the target.

(Appendix 6)
A data acquisition step of acquiring each of image data of images obtained by photographing the periphery of the moving body at each time, and each of line-of-sight data indicating a line-of-sight direction of a passenger of the moving body at each time; ,
A determination step of determining a gaze position of the passenger in an image indicated by the image data at each time based on the image data and the line-of-sight data acquired by the data acquisition step;
A detection step of detecting an object region corresponding to a predetermined object from an image indicated by the image data;
The position in the past image of the object area tracked from the past image taken at a time earlier than the image in which the object area is detected by the detection step, and the position determined by the determination step An estimation step for estimating whether or not the occupant was gazing at the object at the past time based on the gazing position of the occupant in the past image;
Gaze target estimation method.

(Appendix 7)
In the estimation step, the passenger indicates a retroactive range indicating from which time the image in which the object area was detected was captured to what time to track the object area on the past image. The gaze target object estimation method according to appendix 6, wherein the gaze target object determination method is determined based on a distance at which the target object can be recognized and a moving speed of the moving body.

(Appendix 8)
The object corresponding to the object determined to have been watched by the occupant from the object information storage unit storing object information including at least one of detailed information of the object and related information. The gaze target object estimation method according to Supplementary Note 6 or Supplementary Note 7, which includes a target information acquisition step for acquiring target information.

(Appendix 9)
Based on the information related to the passenger estimated to have watched the object in the estimation step and the object information acquired in the object information acquisition step, the interest and interest of the passenger are analyzed. The gaze target object estimation method according to appendix 8, which includes an analysis step to perform.

(Appendix 10)
The gaze target object estimation method according to appendix 9, including a feedback step of feeding back information based on the analysis result of the analysis step to the passenger or a related business operator of the target object.

(Appendix 11)
On the computer,
A data acquisition step of acquiring each of image data of images obtained by photographing the periphery of the moving body at each time, and each of line-of-sight data indicating a line-of-sight direction of a passenger of the moving body at each time; ,
A determination step of determining a gaze position of the passenger in an image indicated by the image data at each time based on the image data and the line-of-sight data acquired by the data acquisition step;
A detection step of detecting an object region corresponding to a predetermined object from an image indicated by the image data;
The position in the past image of the object area tracked from the past image taken at a time earlier than the image in which the object area is detected by the detection step, and the position determined by the determination step An estimation step for estimating whether or not the occupant was gazing at the object at the past time based on the gazing position of the occupant in the past image;
Gaze target estimation program for executing a process including.

(Appendix 12)
In the estimation step, the passenger indicates a retroactive range indicating from which time the image in which the object area was detected was captured to what time to track the object area on the past image. The gaze target object estimation program according to appendix 11, which is determined based on a distance at which the target object can be recognized and a moving speed of the moving body.

(Appendix 13)
The object corresponding to the object determined to have been watched by the occupant from the object information storage unit storing object information including at least one of detailed information of the object and related information. The gaze target object estimation program according to supplementary note 11 or supplementary note 12, which includes a target information acquisition step for acquiring target information.

(Appendix 14)
Based on the information related to the passenger estimated to have watched the object in the estimation step and the object information acquired in the object information acquisition step, the interest and interest of the passenger are analyzed. The gaze target object estimation program according to supplementary note 13, including an analysis step to perform.

(Appendix 15)
15. The attention object estimation program according to supplementary note 14, including a feedback step of feeding back information based on the analysis result of the analysis step to the passenger or a related business operator of the object.

DESCRIPTION OF SYMBOLS 10 Gaze target estimation apparatus 12 Data acquisition part 14 Image data storage part 16 Gaze position determination part 18 Gaze position data storage part 20 Object detection part 22 Object information storage part 24 Estimation part 24a Retrospective range determination part 24b Object area tracking Unit 24c tracking position / gaze position comparison unit 26 object information acquisition unit 28 analysis unit 30 feedback unit 32 image data 34 position data 36 gaze position data 38 object information 40 computer 42 CPU
44 memory 46 storage unit

Claims (7)

A data acquisition unit that acquires each of image data of an image obtained by photographing the periphery of the moving object at each time, and each of the line-of-sight data indicating the line-of-sight direction of the passenger of the moving object at each time; ,
A determination unit that determines the gaze position of the passenger in the image indicated by the image data at each time based on the image data and the line-of-sight data acquired by the data acquisition unit;
A detection unit for detecting an object region corresponding to a predetermined object from an image indicated by the image data;
The position in the past image of the target region tracked from the past image taken at a time earlier than the image in which the target region is detected by the detection unit, and the determination unit An estimation unit that estimates whether or not the passenger was gazing at the object at the past time based on the gazing position of the passenger in a past image;
Gaze target estimation device including   The estimator has a retroactive range indicating whether the object area is to be traced on the past image from the time when the image where the object area was detected was captured, to the time when the passenger The gaze target estimation device according to claim 1, wherein the gaze target estimation device is determined based on a distance at which the target can be recognized and a moving speed of the moving body.   The object corresponding to the object determined by the estimating unit to be watched by the occupant from the object information storage unit that stores the object information including at least one of the detailed information and the related information of the object The gaze target estimation device according to claim 1, further comprising a target information acquisition unit that acquires target information.   Based on the information related to the passenger estimated to have watched the object by the estimating unit and the object information acquired by the object information acquiring unit, the interest and interest of the passenger are analyzed. The gaze target estimation device according to claim 3, further comprising an analysis unit that performs the analysis.   The gaze target object estimation apparatus according to claim 4, further comprising a feedback unit that feeds back information based on an analysis result by the analysis unit to the passenger or a related business operator of the target object. A data acquisition step of acquiring each of image data of images obtained by photographing the periphery of the moving body at each time, and each of line-of-sight data indicating a line-of-sight direction of a passenger of the moving body at each time; ,
A determination step of determining a gaze position of the passenger in an image indicated by the image data at each time based on the image data and the line-of-sight data acquired by the data acquisition step;
A detection step of detecting an object region corresponding to a predetermined object from an image indicated by the image data;
The position in the past image of the object area tracked from the past image taken at a time earlier than the image in which the object area is detected by the detection step, and the position determined by the determination step An estimation step for estimating whether or not the occupant was gazing at the object at the past time based on the gazing position of the occupant in the past image;
Gaze target estimation method. On the computer,
A data acquisition step of acquiring each of image data of images obtained by photographing the periphery of the moving body at each time, and each of line-of-sight data indicating a line-of-sight direction of a passenger of the moving body at each time; ,
A determination step of determining a gaze position of the passenger in an image indicated by the image data at each time based on the image data and the line-of-sight data acquired by the data acquisition step;
A detection step of detecting an object region corresponding to a predetermined object from an image indicated by the image data;
The position in the past image of the object area tracked from the past image taken at a time earlier than the image in which the object area is detected by the detection step, and the position determined by the determination step An estimation step for estimating whether or not the occupant was gazing at the object at the past time based on the gazing position of the occupant in the past image;
Gaze target estimation program for executing a process including.