JP2017016287A - Transmission device, server device, control method, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a transmission device and server device suitable to collect the information of an object such as a signboard to be visually recognized by a user.SOLUTION: A server device 5 acquires signboard information Is related to a signboard viewed by a driver of a vehicle Ve and driver attribute information Ip from an on-vehicle device 2 and registers them in a signboard observation information DB 32. The server device 5 acquires purchase history information Ib of the driver from a portable terminal 1 and acquires visit history information Iv from the on-vehicle device 2. The server device 5 searches the record of the signboard observation information DB 32 corresponding to the purchase history information Ib and additionally registers a purchase history. In the same way, the server device 5 searches the record of the signboard observation information DB 32 corresponding to the visit history information Iv and additionally registers a visit history.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for collecting information on an object that is visually recognized by a user such as a signboard.

  Conventionally, a technique for collecting information on an object visually recognized by a user is known. For example, Patent Document 1 discloses a system that detects a signboard watched by a vehicle occupant and totals it with a server device.

JP 2013-200247 A

  In order to judge the effectiveness of a signboard, it is necessary to consider whether the user who viewed the signboard has purchased an item related to the signboard or visited a facility. It may not be enough to detect and collect that information.

  The present invention has been made to solve the above-described problems, and mainly provides a transmission device and a server device suitable for collecting information on an object such as a signboard that is visually recognized by a user. Objective.

  The invention according to the claim is a transmission device, which is an object viewed by the user from an image acquired from a camera that captures the front of the user and data detected by a detection unit that detects the user's line of sight Object identification means for identifying an object, first transmission means for transmitting information on the object to a server, purchase history of products related to the object, or visit history to a facility related to the object An acquisition means for acquiring and a second transmission means for transmitting the purchase history or the visit history to the server are provided.

  Further, the invention described in claim, in a server device connected to a plurality of transmission devices, first acquisition means for acquiring information on an object viewed by a user of the transmission device from each of the transmission devices; Second acquisition means for acquiring the purchase history or visit history of the user, determination means for determining the purchase history or visit history related to the object, and purchase history or visit history determined by the determination means, Storage means for storing the information in the storage unit in association with information on the related object.

  The invention described in claim is a control method executed by the transmission device, and includes an image acquired from a camera that captures the front of the user and data detected by a detection unit that detects the user's line of sight. , An object identification step for identifying an object viewed by the user, a first transmission step for transmitting information on the object to a server, a purchase history of products related to the object, or an association with the object An acquisition step of acquiring a visit history to the facility to be performed, and a second transmission step of transmitting the purchase history or the visit history to the server.

  The invention described in the claims is a control method executed by a server device connected to a plurality of transmission devices, and acquires information on an object viewed by a user of the transmission device from each of the transmission devices. The first acquisition step, the second acquisition step of acquiring the purchase history or visit history of the user, the determination step of determining the purchase history or visit history related to the object, and the determination step A storage step of storing a purchase history or a visit history in a storage unit in association with information on the related object.

  The invention described in the claims is a program executed by a computer, and includes an image acquired from a camera that captures the front of the user, and data detected by a detection unit that detects the user's line of sight. Object identifying means for identifying an object viewed by a user, first transmission means for transmitting information on the object to a server, purchase history of products related to the object, or facilities related to the object The computer is caused to function as an acquisition unit that acquires a visit history to the server and a second transmission unit that transmits the purchase history or the visit history to the server.

  The invention described in the claims is a program executed by a computer connected to a plurality of transmission devices, wherein information on an object viewed by a user of the transmission device is acquired from each of the transmission devices. 1 acquisition means, second acquisition means for acquiring the purchase history or visit history of the user, determination means for determining the purchase history or visit history related to the object, and purchase history determined by the determination means Alternatively, the computer is caused to function as storage means for storing a visit history in a storage unit in association with information on the related object.

The structural example of the signboard information integration system which concerns on an Example is shown. It is a block diagram of a vehicle equipment. It is a block diagram of a server apparatus. An example of the data structure of signboard basic information DB is shown. An example of the data structure of signboard observation information DB is shown. It is a flowchart which shows the procedure of the addition process of the record to signboard observation information DB performed when a driver looks at a signboard. It is an example of the flowchart which shows the procedure of the process which adds a purchase history to signboard observation information DB. It is an example of the flowchart which shows the procedure of the process which adds a visit log | history to signboard observation information DB. The signboard search result list based on signboard observation information DB is shown. (A) shows an example of the data structure of the signboard observation information DB according to the modification. (B) shows the signboard search result list according to the modification.

  In one preferred embodiment of the present invention, the transmission device is viewed by the user from an image acquired from a camera that captures the front of the user and data detected by a detection unit that detects the user's line of sight. Object specifying means for specifying an object, first transmission means for transmitting information on the object to a server, purchase history of products related to the object, or visit history to facilities related to the object Acquisition means for acquiring information, and second transmission means for transmitting the purchase history or the visit history to the server.

  The transmission device includes an object specifying unit, a first transmission unit, an acquisition unit, and a second transmission unit. The object specifying unit specifies an object viewed by the user from an image acquired from a camera that captures the front of the user and data detected by a detecting unit that detects the user's line of sight. Here, the “object” refers to a medium that advertises a product or facility, and for example, a stationary advertisement medium such as a signboard. The first transmission means transmits information on the object to the server. The acquisition means acquires a purchase history of a product related to the object or a visit history to a facility related to the object. The second transmission means transmits the purchase history or the visit history to the server. According to this aspect, the transmitting device collects information on an object such as a signboard in addition to the information on the object viewed by the user, or purchase performed by the user viewing the object or Accumulate visit history. Therefore, the server can suitably accumulate information that is a material for determining the validity of the object.

  In one aspect of the transmission apparatus, the first transmission unit transmits the attribute information of the user to the server together with the information on the object. According to this aspect, the transmission device can suitably store the history in which the object is observed in the server so that the history can be classified according to the user attribute.

  In another aspect of the transmission device, the first transmission unit includes information on an object that the user has not seen among objects displayed on the image, and information on the object that the user has viewed. A distinction is transmitted to the server. According to this aspect, the transmission device can suitably accumulate information on the object on the server so that the frequency at which the object is observed can be calculated.

  In another aspect of the transmission device, the first transmission unit transmits an image of the object captured by the camera to the server together with the information on the object. Thereby, the transmission apparatus can make a server apparatus analyze suitably the appearance etc. of a target object.

  In another aspect of the transmission device, the object specifying unit detects a signboard from the image, and specifies a signboard present on the user's line of sight as an object viewed by the user. To do. According to this aspect, the transmission device can suitably store the information on the signboard viewed by the user and the purchase or visit history after viewing the signboard on the server.

  In another preferred embodiment of the present invention, in a server device connected to a plurality of transmission devices, first acquisition means for acquiring information on an object viewed by a user of the transmission device from each of the transmission devices; Second acquisition means for acquiring purchase history or visit history of the user, determination means for determining the purchase history or visit history related to the object, and purchase history or visit history determined by the determination means. Storage means for storing the information in the storage unit in association with information on the related object.

  The server apparatus includes a first acquisition unit, a second acquisition unit, a determination unit, and a storage unit. A 1st acquisition means acquires the information of the target object which the user of the transmission apparatus saw from each of the transmission apparatuses. The second acquisition unit acquires the purchase history or visit history of the user. In this case, a 2nd acquisition means may acquire the above-mentioned purchase history or visit history from the above-mentioned transmission apparatus, and may acquire from the terminal or server which manages a purchase history and a visit history. The determining means determines a purchase history or a visit history related to the object viewed by the user. The storage means stores the purchase history or the visit history determined by the determination means in association with the information on the related object. In this aspect, the server device accumulates the history of purchases or visits executed by the user viewing the target object in addition to the information on the target object viewed by the user. Therefore, the server device can suitably accumulate information that is a material for determining the validity of the object.

  In one aspect of the server device, the first acquisition unit acquires an image of the object captured from each of the transmission devices, and the determination unit recognizes how the object is seen from the image. The storage unit stores the appearance of the object in association with the information of the object. According to this aspect, the server device can suitably accumulate information on how the object is viewed by the user.

  In another aspect of the server device, the first acquisition unit acquires the attribute information of the user from each of the transmission devices together with information on the object viewed by the user, and the storage unit includes: The attribute information of the user is stored in association with the information of the object viewed by the user. According to this aspect, the server device can preferably accumulate the history in which the object is observed so that it can be classified according to the user's attribute.

  In still another embodiment of the present invention, the control method executed by the transmission device includes: an image acquired from a camera that captures the front of the user; and data detected by a detection unit that detects the user's line of sight. A target identification step for identifying a target viewed by a user; a first transmission step for transmitting information on the target to a server; a purchase history of products related to the target; or a facility related to the target An acquisition step of acquiring a visit history to the second, and a second transmission step of transmitting the purchase history or the visit history to the server. By executing this control method, the transmission device can suitably store information that is a material for determining the validity of the object in the server.

  In still another embodiment of the present invention, the control method is executed by a server device connected to a plurality of transmission devices, and information on an object viewed by a user of the transmission device is acquired from each of the transmission devices. The first acquisition step, the second acquisition step of acquiring the purchase history or visit history of the user, the determination step of determining the purchase history or visit history related to the object, and the determination step A storage step of storing a purchase history or a visit history in a storage unit in association with the information of the related object. By executing this control method, the server device can suitably accumulate information used as a material for determining the validity of the object in the storage unit.

  In still another embodiment of the present invention, the program is executed by a computer, and is obtained from an image acquired from a camera that captures the front of the user and data detected by a detection unit that detects the user's line of sight. Object identifying means for identifying an object viewed by a user, first transmission means for transmitting information on the object to a server, purchase history of products related to the object, or facilities related to the object The computer is caused to function as an acquisition unit that acquires a visit history to the server and a second transmission unit that transmits the purchase history or the visit history to the server. By executing this program, the computer can suitably store information serving as a material for determining the validity of the object in the server.

  In still another embodiment of the present invention, there is provided a program executed by a computer connected to a plurality of transmission devices, wherein information on an object viewed by a user of the transmission device is acquired from each of the transmission devices. 1 acquisition means, second acquisition means for acquiring the purchase history or visit history of the user, determination means for determining the purchase history or visit history related to the object, and purchase history determined by the determination means Alternatively, the computer is caused to function as storage means for storing a visit history in the storage unit in association with information on the related object. By executing this program, the computer can suitably store information used as a material for determining the validity of the object in the storage unit. Preferably, the program is stored in a storage medium.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[Schematic configuration]
(1) System Configuration FIG. 1 shows a configuration example of a signboard information accumulation system 100 according to the embodiment. The signboard information integration system 100 includes a mobile terminal 1 used by a driver of the vehicle Ve, an in-vehicle device 2, a line-of-sight detection camera 3, an external photographing camera 4, a server device 5, and a browsing terminal 6. The signboard information accumulation system 100 is a system that accumulates and uses data for determining the usefulness of signboards.

  The mobile terminal 1 is a portable terminal such as a smartphone used by the driver of the vehicle Ve, and executes a product purchase process based on electronic commerce or a product purchase process using electronic money. Then, the mobile terminal 1 transmits information (also referred to as “purchase history information Ib”) indicating the purchase history of the product obtained by the purchase process described above to the server device 5. The purchase history information Ib includes identification information (user ID) of a user who is a purchaser, a product ID which is identification information of a product purchased by the driver, and date / time information indicating the date and time of purchase.

  The in-vehicle device 2 has a navigation function for performing route guidance from the departure place to the destination. The in-vehicle device 2 may be, for example, a stationary navigation device installed in the vehicle Ve, or a portable terminal such as a PND (Portable Navigation Device). The in-vehicle device 2 is electrically connected to the line-of-sight detection camera 3 and the external photographing camera 4, and based on the images acquired from these cameras, whether or not the driver has viewed a signboard installed in front of the vehicle such as the signboard 70 To determine. Then, the in-vehicle device 2 uses information (referred to as “signboard information Is”) related to the signboard that the driver has determined to view as information (“driver attribute information Ip”) that represents the driver attributes such as age and gender. And transmitted to the server device 5. In addition, when the vehicle Ve stops at a facility other than the destination set at the start of driving, the in-vehicle device 2 stores information (also referred to as “visit history information Iv”) indicating that the vehicle has stopped. To 5. The visit history information Iv includes a user ID that is identification information of the driver, a facility ID that is identification information of the facility that visited, and information on the date and time when the facility stopped. The in-vehicle device 2 is an example of the “transmission device” in the present invention.

  The line-of-sight detection camera 3 is installed at a position where the driver's face is in the shooting range, and transmits the generated image (also referred to as “in-vehicle shot image Imi”) to the in-vehicle device 2. The in-vehicle photographed image Imi is used to detect the driver's line-of-sight direction. The line-of-sight detection camera 3 is an example of the “detection means” in the present invention.

  The external photographing camera 4 is a camera directed in front of the vehicle Ve, and transmits a generated image (also referred to as “ex-vehicle photographed image Imo”) to the in-vehicle device 2. Preferably, the installation position, the installation angle, and the like of the external shooting camera 4 are adjusted so that the shooting range of the external shooting camera 4 substantially matches the field of view of the driver. The outside-vehicle photographed image Imo is used for detecting a signboard or the like. The external camera 4 is an example of the “camera” in the present invention.

  The server device 5 is electrically connected to the mobile terminal 1, the in-vehicle device 2, and a browsing terminal 6 described later via a network or the like. The server device 5 has a signboard observation information DB 32 in which information on a signboard viewed by the driver, an attribute of the driver who viewed the signboard, and an action history after viewing the signboard are recorded in association with each other. And the server apparatus 5 updates signboard observation information DB32 based on the information received from the portable terminal 1 and the vehicle equipment 2. FIG. Further, when there is a search request specifying a predetermined search condition from the browsing terminal 6, the server device 5 displays screen information generated based on the record in the signboard observation information DB 32 that matches the search condition. 6 to send.

  The browsing terminal 6 is a terminal that is browsed by, for example, a person who determines the effectiveness of a signboard or a person who is considering the form of a signboard to be installed next, and accepts user input that specifies a search condition related to the signboard. And the browsing terminal 6 receives the screen information which shows a search result from the server apparatus 5, by transmitting the designated search condition to the server apparatus 5, and performs the display based on the said screen information.

(2) Configuration of the in-vehicle device FIG. 2 is a block diagram of the in-vehicle device 2. The in-vehicle device 2 mainly includes a display unit 21 such as a display, an input unit 22, a storage unit 23, a communication unit 24 that performs data communication, a control unit 25, a positioning unit 26, and a sound output unit 27. . Each of these elements is connected to each other via a bus line 20.

  Based on the control of the control unit 25, the display unit 21 displays an image indicating a map around the current position, information on audio being reproduced, and the like. The input unit 22 is, for example, a button, a switch, a touch panel, a voice input device, or the like, and receives user input necessary for setting a destination based on the control of the control unit 25.

  The storage unit 23 stores a program for controlling the operation of the in-vehicle device 2 and holds information necessary for the operation of the in-vehicle device 2. In the present embodiment, the storage unit 23 stores map data necessary for route guidance. The storage unit 23 stores driver attribute information Ip. Here, the “driver attribute information Ip” includes the user ID, gender, age, vehicle type of the vehicle Ve, and the like given to the driver.

  The communication unit 24 receives the in-car image Imi from the line-of-sight detection camera 3 and supplies it to the control unit 25. Further, the communication unit 24 receives the outside-captured image Imo from the external camera 4 and supplies it to the control unit 25. Furthermore, the communication unit 24 establishes wireless communication with the server device 5 based on the control of the control unit 25, and transmits signboard information Is, driver attribute information Ip, visit history information Iv, and the like to the server device 5.

  The positioning unit 26 is a self-supporting positioning device such as a gyro sensor or a distance sensor, a GPS receiver, or the like, and performs positioning such as the current location of the vehicle and the traveling direction. The sound output unit 27 outputs a voice for route guidance based on the control of the control unit 25.

  The control unit 25 includes a CPU (not shown) and performs various controls on each component in the in-vehicle device 2. In the present embodiment, the control unit 25 detects a signboard that matches the driver's line of sight. Then, the control unit 25 uses the detected signboard position information, the signboard detection date / time information, and the image of the external photographing camera 4 on which the signboard is displayed (also referred to as “signboard image”) to the signboard information Is. Is transmitted to the server device 5 by the communication unit 24. In addition, when signboard basic information DB31 described later is stored in the storage unit 23 of the vehicle-mounted device, the signboard ID of the signboard that matches the driver's line of sight is transmitted to the server apparatus 5 by the communication section 24 as the signboard information Is. Also good. In addition, the control unit 25 transmits the driver attribute information Ip stored in the storage unit 23 together with the signboard information Is to the server device 5. Further, when the control unit 25 determines that the vehicle Ve has stopped at a facility other than the destination that is initially set after the vehicle Ve is started, the control unit 25 transmits information about the facility to the server device 5 as visit history information Iv. To do.

  Here, a method for detecting a signboard that matches the driver's line of sight will be described. The control part 25 extracts the signboard which exists in the position which a driver | operator can visually recognize by extracting a signboard from the imaging | photography image Imo outside a vehicle. Further, the control unit 25 recognizes the driver's line-of-sight direction from the in-vehicle photographed image Imi based on a known line-of-sight recognition technique. And the control part 25 detects the signboard which exists on the driver | operator's eyes | visual_axis recognized from the in-vehicle captured image Imi among the signboards extracted from the imaging | photography image Imo outside a vehicle as a signboard which matched the eyes | visual_axis of a driver | operator. Thereafter, the control unit 25 calculates the position information of the signboard based on, for example, the current position and traveling direction measured by the positioning unit 26 and the position of the signboard in the outside-vehicle photographed image Imo, and includes the signboard information Is.

  The control unit 25 is an example of a “target specifying unit”, “first transmission unit”, “acquisition unit”, “second transmission unit”, and “computer” that executes a program in the present invention.

(3) Configuration of Server Device FIG. 3 is a block diagram of the server device 5. The server device 5 mainly includes a storage unit 53, a communication unit 54 that performs data communication, and a control unit 55. These elements are connected to each other via a bus line 50.

  The storage unit 53 stores a program for controlling the operation of the in-vehicle device 2 and holds information necessary for the operation of the in-vehicle device 2. In the present embodiment, the storage unit 53 stores in advance a signboard basic information DB 31 in which basic information such as installation positions and sizes of signboards installed throughout the country is recorded. In addition, the storage unit 53 stores a signboard observation information DB 32 generated based on information received from the mobile terminal 1 and the in-vehicle device 2. In the signboard observation information DB 32, the attribute of the driver who viewed the signboard, the purchase history of products related to the signboard after viewing the signboard, the visit history of the facility, and the like are recorded. Detailed data structures of the signboard basic information DB 31 and the signboard observation information DB 32 will be described in the section [Data Structure].

  The communication unit 54 establishes communication with the vehicle-mounted device 2 based on the control of the control unit 55, and receives signboard information Is, driver attribute information Ip, visit history information Iv, and the like. Further, the communication unit 54 establishes communication with the mobile terminal 1 based on the control of the control unit 55 and receives purchase history information Ib and the like.

  The control unit 55 includes a CPU (not shown) and the like, and performs various controls on each component in the server device 5. In the present embodiment, the control unit 55 updates the signboard observation information DB 32 based on information received from the mobile terminal 1 and the in-vehicle device 2. In addition, when there is a search request specifying a predetermined search condition from the browsing terminal 6, the control unit 55 displays the screen information generated from the record in the signboard observation information DB 32 that matches the search condition, and the communication unit 54. Is transmitted to the browsing terminal 6. The control unit 55 is an example of the “first acquisition unit”, “second acquisition unit”, “determination unit”, “storage unit”, and “computer” that executes a program in the present invention.

[data structure]
FIG. 4 shows an example of the data structure of the signboard basic information DB 31. The signboard basic information DB 31 is a database in which basic information for each signboard is registered. In the example of FIG. 4, “signboard ID”, “position”, “height”, “size”, “design”, “category” "," Product (facility) name ", and" distance to facility ".

  Here, each item which signboard basic information DB31 has is demonstrated. In the item “position”, latitude / longitude information indicating the installation position of the signboard is registered. In the item “height”, the height from the ground surface where the signboard is installed is registered. In the item “size”, the vertical and horizontal lengths of the signboard are registered. In the item “design”, information indicating the color of the signboard and other design features is registered. In the item “category”, the category of the product or facility service advertised by the signboard is registered. In the “product (facility) name”, the name of the product advertised by the signboard or the facility guided by the signboard and the corresponding product ID or facility ID are registered. In the item “distance to facility”, when the signboard is related to the facility, the distance from the installation position of the signboard to the facility related to the signboard is registered.

  FIG. 5 shows an example of the data structure of the signboard observation information DB 32. In the example of FIG. 5, the signboard observation information DB 32 includes “user ID”, “signboard ID”, “sex”, “age”, “vehicle type”, “date / time”, “appearance”, “presence / absence of purchase”, and Includes each item of “visited”. Here, in the item “date and time”, the date and time when the driver saw the target signboard is registered. Here, in the item “how to see”, information indicating how the target signboard looked to the driver is registered. In addition, in the item “presence / absence of purchase”, information indicating whether or not a related product has been purchased after the driver sees the sign is registered when the target signboard is related to a specific product. In addition, in the item “presence / absence of visit”, information indicating whether or not the driver has visited the related facility after viewing the sign is registered when the target signboard is a signboard related to the facility.

  Here, a method of generating each record in the signboard observation information DB 32 illustrated in FIG. 5 will be described.

  When the server device 5 receives the signboard information Is and the driver attribute information Ip from the in-vehicle device 2, the server device 5 refers to the signboard basic information DB 31 based on the signboard position information included in the signboard information Is, so that the driver can The signboard ID corresponding to the signboard with the line of sight is recognized. The server device 5 associates the recognized signboard ID with the user ID, gender, age, vehicle type, and signboard detection date / time included in the signboard information Is included in the driver attribute information Ip, and records in the signboard observation information DB 32 Register as When signboard ID is received as signboard information Is, the received signboard ID is associated with the user ID, gender, age, vehicle type, and signboard detection date / time included in signboard information Is included in driver attribute information Ip. And registered as a record in the signboard observation information DB 32.

  Further, the server device 5 identifies the content to be registered in the item “appearance” by performing image analysis based on the signboard image included in the signboard information Is. In the example of FIG. 5, when the server device 5 determines that the signboard image included in the signboard information Is indicates the entire signboard based on a known image recognition technology, the item “how to see” When the signboard image is determined to show only a part of the signboard, the item “how to see” is set to “partially seen”. In addition, even when the signboard image shows the entire signboard, the server device 5 determines that the signboard contents are not visible due to the presence or absence of irradiation of the western sun, the time zone, etc. , The item “how to see” is “the contents are not visible”.

  Further, when the server device 5 receives the purchase history information Ib, if there is a record corresponding to the purchase history information Ib among the records registered in the signboard observation information DB 32, ”Item is set to“ Yes ”. In this case, first, the server device 5 refers to the items “product name” and “signboard ID” of the signboard basic information DB 31 and recognizes the corresponding signboard ID from the product ID included in the purchase history information Ib. Next, the server device 5 searches for a record in the signboard observation information DB 32 in which a combination of the recognized signboard ID and the user ID included in the purchase history information Ib is defined. The server device 5 further refers to the item “date and time” of the retrieved record, and records the date and time indicated by the item before the purchase date and time indicated by the purchase history information Ib and within a predetermined time difference (for example, 30 minutes). On the other hand, the item “purchased” is set to “present”. Thereby, the server apparatus 5 can register suitably only the purchase history which resulted in the purchase of the goods relevant to the said signboard by seeing the signboard in signboard observation information DB32.

  Similarly, when the server device 5 receives the visit history information Iv, when there is a record corresponding to the visit history information Iv among the records registered in the signboard observation information DB 32, the “visit of visit” is recorded. Set “Presence” to “Yes”. In this case, first, the server device 5 refers to the items “facility name” and “signboard ID” of the signboard basic information DB 31 and recognizes the corresponding signboard ID from the facility ID included in the visit history information Iv. Next, the server device 5 searches for a record in the signboard observation information DB 32 in which a combination of the recognized signboard ID and the user ID included in the visit history information Iv is defined. Then, the server device 5 further refers to the item “date and time” of the retrieved record, and the date and time indicated by the item is before the visit date and time specified from the visit history information Iv and within a predetermined time difference (for example, 30 minutes). On the other hand, the item “visited / not visited” is set to “present”. Thereby, the server apparatus 5 can register suitably only the visit history which resulted in the visit of the facility relevant to the said signboard by seeing the signboard in signboard observation information DB32.

[Processing flow]
(1) Processing for adding records to the signboard observation information DB 32 FIG. 6 is a flowchart showing a procedure for adding records to the signboard observation information DB 32 that is executed when the driver views the signboard. The in-vehicle device 2 repeatedly executes the process shown in FIG. 6 after the vehicle Ve is started.

  First, the in-vehicle device 2 determines whether or not a signboard has been detected from the outside-vehicle shot image Imo by performing predetermined image processing on the outside-vehicle shot image Imo supplied from the external shooting camera 4 (step S101). And the vehicle equipment 2 will advance a process to step S102, when a signboard is detected from the imaging | photography image Imo outside a vehicle (step S101; Yes). Further, in this case, the in-vehicle device 2 prepares for the determination process in step S103 to be described later, in accordance with the display position of the signboard in the outside-vehicle photographed image Imo, the current position of the vehicle Ve measured by the positioning unit 26, the traveling direction, and the like. Based on this, the detected position of the signboard is calculated. On the other hand, when the signboard cannot be detected from the outside-captured image Imo (step S101; No), the in-vehicle device 2 performs the signboard detection process again on the outside-captured image Imo supplied from the external shooting camera 4 after a predetermined time.

  Next, the vehicle-mounted device 2 detects the driver's line-of-sight direction from the in-vehicle photographed image Imi supplied from the line-of-sight detection camera 3 (step S102). In this case, the vehicle-mounted device 2 detects the relative line-of-sight direction of the driver with respect to the line-of-sight detection camera 3 from the in-vehicle captured image Imi using a known line-of-sight recognition technique, and further, the installation position of the line-of-sight detection camera 3 in the vehicle Ve, The driver's line-of-sight direction is detected by taking into account the installation angle, the traveling direction of the vehicle Ve, and the like.

  And the vehicle equipment 2 determines whether the driver was looking at the signboard detected at step S101 (step S103). In this case, for example, the vehicle-mounted device 2 recognizes the position on the driver's line of sight based on the current position of the vehicle Ve and the driver's line-of-sight direction detected in step S102, and the position of the signboard detected in step S101 is determined. It is determined whether or not the position overlaps with the position on the driver's line of sight. In addition, when a plurality of signboards are detected in step S101 from the outside-captured image Imo, the in-vehicle device 2 executes the process of step S103 for each detected signboard.

  And when the vehicle equipment 2 judges that the driver was looking at the signboard (step S103; Yes), the signboard information Is for specifying the signboard detected at step S101 and the driver stored in the storage unit 23 The attribute information Ip is transmitted to the server device 5 (step S104). The signboard information Is includes a signboard image in which the signboard portion detected in step S101 is extracted from the outside-vehicle photographed image Imo. On the other hand, the vehicle equipment 2 returns a process to step S101, when it is judged that the driver did not look at the signboard (step S103; No).

  After step S104, the server device 5 receives the signboard information Is and the driver attribute information Ip transmitted from the in-vehicle device 2 (step S201). Then, the server device 5 specifies the signboard ID of the signboard viewed by the driver by referring to the signboard basic information DB 31 based on the position information of the signboard included in the signboard information Is (step S202). Thereafter, the server device 5 analyzes the appearance of the signboard from the signboard image included in the signboard information Is (step S203). Thereby, the server apparatus 5 determines the content to be registered in the “appearance” item of the signboard observation information DB 32. Then, the server device 5 adds a new record to the signboard observation information DB 32 based on the analysis result of step S202 and the received signboard information Is and driver attribute information Ip (step S204). Note that the record added in step S204 does not yet include records corresponding to the items “purchased” and “visited” in FIG. The addition of the record corresponding to the item “presence / absence of purchase” will be described with reference to FIG. 7, and the addition of the record corresponding to the item “presence / absence of visit” will be described with reference to FIG.

(2) Purchase History Addition Processing FIG. 7 is an example of a flowchart showing a procedure of processing for adding a purchase history to the signboard observation information DB 32. The mobile terminal 1 repeatedly executes the process of the flowchart of FIG.

  First, the mobile terminal 1 determines whether or not purchase behavior has been detected (step S111). For example, the mobile terminal 1 determines that there is a purchase action when there is a settlement with electronic money using the mobile terminal 1 or an electronic commerce using the Internet or the like. And the portable terminal 1 transmits purchase history information Ib to the server apparatus 5, when a purchase action is detected (step S111; Yes) (step S112). Here, the mobile terminal 1 sends, as the purchase history information Ib, for example, the user ID of the driver who is the user of the mobile terminal 1, the product ID indicating the purchased product, and purchase date / time information to the server device 5. Send.

  Then, the server device 5 receives the purchase history information Ib transmitted from the mobile terminal 1 (step S211). Then, the server device 5 determines whether or not there is a record corresponding to the received purchase history information Ib in the signboard observation information DB 32 (step S212). In this case, first, the server device 5 recognizes the corresponding signboard ID by referring to the signboard basic information DB 31 from the product ID included in the purchase history information Ib. Then, the server device 5 determines whether or not a record in which the recognized signboard ID is associated with the user ID included in the purchase history information Ib exists in the signboard observation information DB 32. In this case, preferably, the server device 5 may search only records for which the date and time indicated in the item “date and time” is before the purchase date and time indicated by the purchase history information Ib and within a predetermined time difference.

  When the record corresponding to the received purchase history information Ib exists in the signboard observation information DB 32 (step S212; Yes), the server device 5 additionally records the purchase history in the corresponding signboard observation information DB 32 (step). S213). That is, the server device 5 sets the item “purchased” in the corresponding record to “present”. On the other hand, the server apparatus 5 complete | finishes the process of a flowchart, when the record corresponding to the received purchase history information Ib does not exist in signboard observation information DB32 (step S212; No).

(3) Visit History Adding Process FIG. 8 is an example of a flowchart showing a procedure of a process for adding a visit history to the signboard observation information DB 32. The in-vehicle device 2 repeatedly executes the process of the flowchart of FIG. 8 after executing the transmission process of the signboard information Is and the like in step S104 of FIG.

  First, after transmitting the signboard information Is or the like in step S104 in FIG. 6, the in-vehicle device 2 determines whether or not the vehicle Ve is detected to stop at a facility other than the destination set before the transmission (step S104). S121). For example, when the vehicle-mounted device 2 arrives at the destination or stopover set after the transmission of the signboard information Is or the like in step S104 of FIG. 6, or the vehicle-mounted device 2 exceeds a predetermined time at a position off the set route. When there is parking or the like, it is determined that the vehicle Ve has stopped at the facility near the arrival place or the parking position.

  And the vehicle equipment 2 will transmit the visit history information Iv to the server apparatus 5, when the stop of the vehicle Ve to a facility is detected (step S121; Yes) (step S122). In this case, the in-vehicle device 2 includes, as the visit history information Iv, for example, the user ID of the driver who is the user of the in-vehicle device 2, the facility ID that is identification information of the facility that visited, and the date / time information indicating the date and time of the stop Is transmitted to the server device 5.

  And the server apparatus 5 receives the visit history information Iv transmitted from the vehicle equipment 2 (step S221). Then, the server device 5 determines whether or not there is a record corresponding to the received visit history information Iv in the signboard observation information DB 32 (step S222). In this case, first, the server device 5 recognizes the corresponding signboard ID by referring to the signboard basic information DB 31 from the facility ID included in the visit history information Iv. Then, the server device 5 determines whether or not a record in which the recognized signboard ID and the user ID included in the visit history information Iv are associated exists in the signboard observation information DB 32. In this case, preferably, the server device 5 may search only records for which the date and time indicated in the item “date and time” is before the visit date and time indicated by the visit history information Iv and within a predetermined time difference.

  Then, when the record corresponding to the received visit history information Iv exists in the signboard observation information DB 32 (step S222; Yes), the server device 5 additionally records the visit history in the corresponding signboard observation information DB 32 (step). S223). That is, the server device 5 sets the item “visited / not visited” of the corresponding record to “present”. On the other hand, the server apparatus 5 complete | finishes the process of a flowchart, when the record corresponding to the received visit history information Iv does not exist in signboard observation information DB32 (step S222; No).

[Display example]
Next, a usage example of the signboard observation information DB 32 will be described. Hereinafter, a case where the server device 5 transmits a search result list (also referred to as “signboard search result list”) to the browsing terminal 6 based on a search request from the browsing terminal 6 will be described.

  FIG. 9A is displayed by the browsing terminal 6 when the user of the browsing terminal 6 who intends to install a signboard of a restaurant whose main target is a woman in his twenties specifies a predetermined search condition. It is an example of a signboard search result list. In this example, the user of the browsing terminal 6 specifies that the age is in his twenties, the gender is female, the category is restaurant, and the number of visitors is 5 or more as a search condition.

  In this case, the server device 5 that has received the search request including the above-described search conditions from the browsing terminal 6 extracts records from the signboard observation information DB 32 in which the item “age” is in the twenties and the item “gender” is female. . Furthermore, the server device 5 refers to each signboard ID of the extracted record, and further extracts a record in which the item “category” in the signboard basic information DB 31 corresponding to the signboard ID is a restaurant. Then, the server device 5 divides the extracted records into the signboard IDs, and recognizes the signboard ID in which five or more records having the item “visited / not visited” are “present”. And the server apparatus 5 produces | generates the signboard search result list | wrist shown to FIG. 9 (A) by extracting the information corresponding to signboard ID in which five or more visit histories exist from signboard basic information DB31. In the signboard search result list in FIG. 9A, in addition to the information extracted from the signboard basic information DB 31, an item “number of visits” indicating the total number of visit histories is provided. Then, the browsing terminal 6 receives information on the signboard search result list from the server device 5 as a search result, and displays the signboard search result list shown in FIG.

  In this case, the user of the browsing terminal 6 can preferably grasp trends such as the size, design, installation position, etc. of a signboard effective when installing a signboard of a restaurant whose main target is a woman in his twenties. it can. Therefore, the user of the browsing terminal 6 can suitably install an effective restaurant signboard by referring to the signboard search result list.

  FIG. 9B is displayed by the browsing terminal 6 when a user of the browsing terminal 6 who intends to install a beer sign whose main target is a man in his 30s specifies a predetermined search condition. It is an example of a signboard search result list. In this example, the user of the browsing terminal 6 specifies that the age is 30's, the sex is male, the category is beer, and the number of purchasers is 100 or more as a search condition.

  In this case, the server device 5 that has received the search request including the above-described search conditions from the browsing terminal 6 extracts records from the signboard observation information DB 32 in which the item “age” is 30s and the item “gender” is male. . Furthermore, the server device 5 refers to each signboard ID of the extracted record, and further extracts a record in which the item “category” in the signboard basic information DB 31 corresponding to the signboard ID is beer. Then, the server device 5 divides the extracted records into the signboard IDs, and recognizes the signboard ID in which 100 or more records having the item “purchased” are “present”. And the server apparatus 5 produces | generates the signboard search result list | wrist shown to FIG. 9 (B) by extracting the information corresponding to signboard ID with 100 or more purchase histories from signboard basic information DB31. In the signboard search result list of FIG. 9B, in addition to the information extracted from the signboard basic information DB 31, an item “number of purchasers” indicating the total number of purchase histories is provided. Then, the browsing terminal 6 receives information on the signboard search result list from the server device 5 as a search result, and displays the signboard search result list shown in FIG. 9B.

  In this case, the user of the browsing terminal 6 can appropriately grasp the trends in the size, design, installation position, etc. of the signboard effective when installing a signboard of beer whose main target is a man in his thirties. it can. Therefore, the user of the browsing terminal 6 can suitably install an effective beer signboard by referring to the signboard search result list.

[Modification]
Hereinafter, modified examples suitable for the above-described embodiments will be described. The following modifications may be applied in any combination to the above-described embodiments.

(Modification 1)
The server device 5 may register in the signboard observation information DB 32 not only the record in which the driver has adjusted his gaze, but also the record in which the driver has not adjusted his gaze.

  FIG. 10A shows an example of the data structure of the signboard observation information DB 32 according to this modification. In the signboard observation information DB 32 of FIG. 10A, an item of “matching line of sight” is provided. When the line of sight matches, “match”, and when the line of sight does not match, “mismatch”. It is recorded. Thus, in the example of FIG. 10A, in addition to the record related to the sign that matches the line of sight, the record related to the sign that does not match the line of sight is also added to the signboard observation information DB 32.

  Here, a specific example of a method for adding records in which the item of “match of line of sight” is “mismatch” will be described. In the first example, the in-vehicle device 2 executes the transmission process of step S104 regardless of the determination result of whether or not the signboard was viewed in step S103 of FIG. In this case, the vehicle-mounted device 2 includes the determination result in step S103 in the signboard information Is to be transmitted, thereby causing the server device 5 to identify whether or not the line of sight matches. In the second example, the server device 5 acquires information related to the position of the vehicle Ve from the in-vehicle device 2 and extracts a signboard that exists in a range that is visible from the vehicle Ve from the signboard basic information DB 31 based on the information. Then, the server device 5 recognizes a signboard other than the signboard specified by the signboard information Is transmitted from the in-vehicle device 2 among the extracted signboards as a signboard that does not match the line of sight, and stores it in the signboard observation information DB 32. sign up.

  FIG. 10B is an example of a signboard search result list generated based on the signboard observation information DB 32 of FIG. FIG. 10B is an example of a signboard search result list whose search condition is age 20s, gender female, category restaurant, and line-of-sight matching probability (line-of-sight matching probability) of 10% or more. .

  In this case, first, the server device 5 extracts records satisfying the age, sex, and category conditions from the signboard observation information DB 32. Next, the server device 5 calculates the line-of-sight coincidence probability for each signboard ID by referring to the item “line-of-sight match”, and identifies the signboard ID having the probability of 10% or more. And the server apparatus 5 produces | generates the signboard search result list | wrist shown to FIG. 10 (B) by extracting the information corresponding to specified signboard ID from signboard basic information DB31. In the signboard search result list of FIG. 10B, in addition to the information extracted from the signboard basic information DB 31, an item “line-of-sight matching probability” indicating the calculated line-of-sight matching probability is provided.

  Thus, by registering the signboard that does not match the line of sight in the signboard observation information DB 32, the server device 5 can also accurately determine the effectiveness of the signboard in consideration of the frequency with which the line of sight matches. Become.

(Modification 2)
The configuration of the signboard information accumulation system 100 shown in FIG. 1 is an example, and the configuration to which the present invention can be applied is not limited to this.

  For example, the external photographing camera 4 may be a camera attached to a wearable terminal such as a smart glass worn by the driver. In this case, the vehicle-mounted device 2 obtains an outside-vehicle photographed image Imo that is a photographing range corresponding to the driver's field of view from the wearable terminal. Similarly, the line-of-sight detection camera 3 may be attached to a wearable terminal such as a smart glass worn by the driver. In this case, the in-vehicle device 2 receives the image generated by the line-of-sight detection camera 3 or information on the line-of-sight direction detected by the wearable terminal based on the image. The line-of-sight detection camera 3 is not limited to the camera, and may be various sensors used to detect the line of sight.

(Modification 3)
In the embodiment, the server device 5 adds the purchase history to the signboard observation information DB 32 based on the purchase history information Ib acquired from the mobile terminal 1. In addition to or instead of this, the server device 5 is connected to the POS system via the network, and acquires purchase history information Ib corresponding to the user ID registered in the signboard observation information DB 32 from the POS system. Also good. In this case, similarly to the embodiment, the server device 5 searches for a record in the signboard observation information DB 32 corresponding to the received purchase history information Ib, and records a purchase history for the searched record.

  In addition to this, or instead of this, the server device 5 is connected to the credit card purchase history server via the network, and the user ID registered in the signboard observation information DB 32 is registered from the credit card purchase history server. Corresponding purchase history information Ib may be acquired. In this case, similarly to the embodiment, the server device 5 searches for a record in the signboard observation information DB 32 corresponding to the received purchase history information Ib, and records a purchase history for the searched record.

(Modification 4)
The in-vehicle device 2 may store the signboard basic information DB 31 and use it to detect the signboard from the outside-vehicle photographed image Imo.

  For example, in this case, in step S101 in FIG. 6, when the vehicle-mounted device 2 detects the signboard area from the outside-vehicle photographed image Imo, it calculates the installation position and size of the detected signboard, and matches the calculation result. It is determined whether there is a record in the signboard basic information DB 31 to be performed. Then, when there is a record in the signboard basic information DB 31 that matches the calculation result, the in-vehicle device 2 determines whether or not a signboard corresponding to the record exists on the driver's line of sight. By doing in this way, the vehicle equipment 2 can detect a signboard with high precision.

(Modification 5)
The server device 5 may be a server system including a plurality of servers. In this case, the servers mutually exchange data necessary for executing a process assigned in advance, and execute a necessary process.

(Modification 6)
The mobile terminal 1 may transmit the purchase history information Ib to the in-vehicle device 2 instead of transmitting to the server device 5. In this case, the in-vehicle device 2 transfers the purchase history information Ib received from the mobile terminal 1 to the server device 5.

DESCRIPTION OF SYMBOLS 1 Portable terminal 2 Car-mounted apparatus 3 Eye-gaze detection camera 4 External imaging camera 5 Server apparatus 6 Browsing terminal 100 Signboard information integration system

Claims (13)

An object identifying unit that identifies an object viewed by the user from an image acquired from a camera that captures the front of the user and data detected by a detecting unit that detects the user's line of sight;
First transmission means for transmitting information of the object to a server;
An acquisition means for acquiring a purchase history of a product related to the object or a visit history to a facility related to the object;
Second transmission means for transmitting the purchase history or the visit history to the server;
A transmission device comprising:   The transmission device according to claim 1, wherein the first transmission unit transmits the attribute information of the user to the server together with the information on the object.   The first transmission means transmits information on an object not viewed by the user among objects displayed on the image to the server separately from information on the object viewed by the user. The transmission device according to claim 1 or 2.   The transmission according to any one of claims 1 to 3, wherein the first transmission unit transmits an image of the object captured by the camera to the server together with information on the object. apparatus.   The object identifying unit detects a signboard from the image, and identifies a signboard present on the line of sight of the user among the detected signboards as an object viewed by the user. 5. The transmission device according to any one of 4. In a server device connected to a plurality of transmission devices,
First acquisition means for acquiring information on an object viewed by a user of the transmission device from each of the transmission devices;
Second acquisition means for acquiring purchase history or visit history of the user;
Determining means for determining the purchase history or visit history related to the object;
Storage means for storing a purchase history or a visit history determined by the determination means in a storage unit in association with information on the related object;
The server apparatus characterized by having. The first acquisition means acquires an image of the object photographed from each of the transmission devices,
The determining means recognizes how the object is seen from the image,
The server device according to claim 6, wherein the storage unit stores the appearance of the object in association with information on the object. The first acquisition unit acquires the attribute information of the user from each of the transmission devices together with information on the object viewed by the user,
The server device according to claim 6, wherein the storage unit stores the attribute information of the user in association with information on an object viewed by the user. A control method executed by a transmitting device,
An object identification step for identifying an object viewed by the user from an image acquired from a camera that captures the front of the user and data detected by a detection unit that detects the line of sight of the user;
A first transmission step of transmitting information on the object to a server;
An acquisition step of acquiring a purchase history of a product related to the object or a visit history to a facility related to the object;
A second transmission step of transmitting the purchase history or the visit history to the server;
A control method characterized by comprising: A control method executed by a server device connected to a plurality of transmission devices,
A first acquisition step of acquiring information on an object viewed by a user of the transmission device from each of the transmission devices;
A second acquisition step of acquiring the purchase history or visit history of the user;
A determination step of determining the purchase history or visit history associated with the object;
A storage step of storing a purchase history or a visit history determined by the determination step in a storage unit in association with information on the related object,
A control method characterized by comprising: A program executed by a computer,
An object identifying unit that identifies an object viewed by the user from an image acquired from a camera that captures the front of the user and data detected by a detecting unit that detects the user's line of sight;
First transmission means for transmitting information of the object to a server;
An acquisition means for acquiring a purchase history of a product related to the object or a visit history to a facility related to the object;
A program for causing the computer to function as second transmission means for transmitting the purchase history or the visit history to the server. A program executed by a computer connected to a plurality of transmission devices,
First acquisition means for acquiring information on an object viewed by a user of the transmission device from each of the transmission devices;
Second acquisition means for acquiring purchase history or visit history of the user;
Determining means for determining the purchase history or visit history related to the object;
A program that causes the computer to function as a storage unit that stores a purchase history or a visit history determined by the determination unit in a storage unit in association with information on the related object.   A storage medium storing the program according to claim 11 or 12.

Images