JP2008047015A - Information processing device, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To properly determine the validity of an image. <P>SOLUTION: A server 2 presumes a presumed photographing state of a transmission image G1 from a portable terminal 1-1. In this case, a display image actually displayed in an information display device 3-3 in the presumed photographing state is an image corresponding to the transmission image G1. Then, in such a case, the server 2 recognizes that the transmission image G1 is a valid image. Unlike the transmission image G1, in the case of presuming a presumed photographing state of a transmission image G2 from a portable terminal 1-2, the presumed photographing state is different from a state in which the portable terminal 1-1 photographs the display device 3-3. Then, since a display image actually displayed in the information display device 3-3 in the presumed photographing state in such a state is an image different from an image corresponding to the transmission image G2, the server 2 recognizes that the transmission image G2 is not a valid image. This is applicable to a server that distributes related information of an advertisement image. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an information processing apparatus, method, and program, and more particularly to an information processing apparatus that provides services to a mobile terminal that has captured and transmitted an image displayed on an image display apparatus. The present invention relates to an information processing apparatus, method, and program that can more accurately determine the legitimacy of a transmission image from a computer.

  Advertisement images such as posters and signboards (including videos such as moving images) to be pasted on the street have a long production period, and the labor required for changing the advertisement images is very large. For this reason, in recent years, a relatively large image display device capable of displaying various images has been installed on the street so that different advertisement images can be displayed on the image display device, thereby greatly reducing the labor required for changing the advertisement image. In addition, an advertising system has been put into practical use that can greatly shorten the production period of these advertising images (see, for example, Patent Document 1).

Furthermore, by using the image recognition function, the advertisement image displayed on the image display device at the street corner is simply photographed with a camera such as a user's mobile terminal and sent to the server, and the URL (Uniform There are services that download related information such as (Resource Locator) and coupons (see, for example, Patent Documents 2 and 3). As such an image recognition device, an image recognition device that is resistant to illumination changes and occlusion has been developed (see Patent Document 4).
JP 2002-230410 A JP 2004-145416 A JP 2005-215922 A JP 2004-326693 A

  However, in conventional recognition technologies including Patent Documents 1 to 4, a server that provides services to a mobile terminal that has captured and transmitted an advertisement image displayed on an image display device, the mobile terminal It is difficult to determine the validity of a transmission image, that is, whether or not the transmission image is a legitimate image obtained as a result of directly capturing an image displayed on the image display device.

  Specifically, for example, an advertisement is valuable only when it is viewed by many people. Therefore, for example, the advertisement image displayed on the image display device installed at the entrance of a certain shopping street is photographed with a mobile terminal or the like and transmitted to the server. It is conceivable to provide a service of sending a ticket. If it does so, it will become possible to make a consumer actually go to the front of an image display apparatus, ie, a shopping street.

  However, for example, an advertisement image displayed on the image display device is shot by a mobile terminal or the like having a high-resolution shooting function, the shot image is distributed over a network, and the shot image is displayed on another person's display device. Even if an image displayed on another person's display device is photographed by another person's mobile terminal or the like and transmitted to a server or the like, the image transmitted from the other person's mobile terminal cannot be obtained by the conventional recognition technology alone. Is recognized as a legitimate image. As a result, the coupon is also transmitted to the portable terminal of another person who has not visited the image display device. That is, it is necessary to distribute coupons only to those who have actually visited the image display device, but it has been difficult to achieve this with only conventional recognition technology.

  The present invention has been made in view of such a situation, and in a server that provides services to a mobile terminal that has captured and transmitted an image displayed on an image display device, the mobile terminal Therefore, it is possible to more accurately determine the legitimacy of the transmission image from.

  An information processing apparatus according to an aspect of the present invention determines whether a transmission image from a terminal is a legitimate image obtained as a result of the terminal directly capturing an image displayed on a display device as a subject. An information processing apparatus comprising: a reception unit that receives the transmission image; an acquisition unit that acquires information about the transmission image received by the reception unit; and the transmission based on the information acquired by the acquisition unit When the estimated shooting situation of the image is estimated and the display image actually displayed on the display device under the estimated shooting situation is an image corresponding to the transmission image, the transmission image is a valid image. An image recognizing unit that recognizes and recognizes that the transmitted image is an illegal image when the image is not an image corresponding to the transmitted image;

  The image recognizing means estimates an estimated shooting time zone as one of the estimated shooting situations when it is assumed that the transmission image is shot by the terminal that has transmitted the transmitted image.

  The acquisition unit acquires the reception time of the transmission image in the reception unit as one of the information related to the transmission image, and the image recognition unit is a fixed time period before the reception time acquired by the acquisition unit. Is estimated as the estimated photographing time zone.

  The image recognizing means estimates an estimated photographing location as one of the estimated photographing situations when it is assumed that the transmission image is photographed by the terminal that has transmitted the transmitted image.

  The image recognition unit further includes an image storage unit that stores a plurality of stored images that can be the display image of the display device, and the image recognition unit is the display image that is actually displayed on the display device under the estimated photographing condition. If one or more of the stored images that can be determined to be extracted from the image storage means and an image corresponding to the transmission image is included in the one or more extracted stored images, the transmission image is a valid image When the image corresponding to the transmission image is not included in the one or more extracted accumulated images extracted or the accumulated image is not extracted, the transmission image is an invalid image. Recognize that there is.

  A transmission unit that transmits the related information of the display image to the terminal; and when the transmission image is recognized as a valid image by the image recognition unit, permits transmission of the related information from the transmission unit; The image recognition means further comprises a transmission permission judgment means for prohibiting transmission of the related information from the transmission means when the transmission image is recognized as an illegal image.

  In the information processing method according to one aspect of the present invention, it is determined whether or not a transmission image from a terminal is a legitimate image obtained as a result of the terminal directly capturing an image displayed on a display device as a subject. An information processing method for an information processing apparatus, wherein the transmission image is received, information about the received transmission image is acquired, an estimated photographing situation of the transmission image is estimated based on the acquired information, When the display image that is actually displayed on the display device under the estimated photographing condition is an image corresponding to the transmission image, the transmission image is recognized as a valid image and corresponds to the transmission image. If it is not an image, the method includes a step of recognizing that the transmission image is an illegal image.

  A program according to one aspect of the present invention is a program corresponding to the information processing method according to one aspect of the present invention described above.

  In the information processing apparatus and method, and the program according to one aspect of the present invention, whether or not the transmission image from the terminal is a legitimate image obtained as a result of direct imaging by the terminal using the display image of the display device as a subject. The legitimacy of is determined as follows. Information on the received transmission image is acquired, an estimated shooting situation of the transmission image is estimated based on the acquired information, and a display image actually displayed on the display device under the estimated shooting situation When the image corresponds to the transmission image, the transmission image is recognized as a valid image, and when the image does not correspond to the transmission image, the transmission image is recognized as an illegal image.

  As described above, according to the present invention, it is possible to provide an information processing apparatus that provides a service to a terminal that has captured and transmitted an image displayed on an image display apparatus. In particular, the information processing apparatus can more accurately determine the validity of the image transmitted from the terminal.

  Embodiments of the present invention will be described below. Correspondences between the configuration requirements of the present invention and the embodiments described in the detailed description of the present invention are exemplified as follows. This description is to confirm that the embodiments supporting the present invention are described in the detailed description of the invention. Accordingly, although there are embodiments that are described in the detailed description of the invention but are not described here as embodiments corresponding to the constituent elements of the present invention, It does not mean that the embodiment does not correspond to the configuration requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  Further, this description does not mean that all the inventions corresponding to the specific examples described in the embodiments of the invention are described in the claims. In other words, this description is an invention corresponding to the specific example described in the embodiment of the invention, and the existence of an invention not described in the claims of this application, that is, in the future, a divisional application will be made. Nor does it deny the existence of an invention added by amendment.

An information processing apparatus according to one aspect of the present invention (for example, the server 2 in FIG. 3)
A transmission image (for example, the transmission image G in FIG. 3) transmitted from a terminal (for example, the portable terminal 1 in FIG. 3) is directly displayed on the display device (for example, the information display device 3 in FIG. 3). An information processing apparatus that determines whether a legitimate image obtained as a result of shooting is valid,
Receiving means for receiving the transmission image (for example, the receiving unit 24 in FIG. 3);
Acquisition means (for example, transmission image information acquisition unit 25 in FIG. 3) for acquiring information related to the transmission image received by the reception means;
An estimated shooting situation of the transmission image is estimated based on the information acquired by the acquisition means, and a display image actually displayed on the display device under the estimated shooting situation is an image corresponding to the transmission image If the transmission image is a legitimate image, the image recognition means recognizes that the transmission image is an illegal image if it is not an image corresponding to the transmission image (for example, the image recognition in FIG. 3). Part 26).

The image recognizing means estimates an estimated shooting time zone as one of the estimated shooting situations when it is assumed that the transmission image is shot by the terminal that has transmitted the transmitted image.
As one of the information related to the transmission image, the acquisition unit receives the transmission image at the reception unit (for example, in the example of FIG. 2, the reception time of the transmission image G1 is the time tb. On the other hand, the transmission image G2 Is received at time tc)
The image recognizing unit estimates a certain time zone before the reception time acquired by the acquiring unit as the estimated shooting time zone (for example, in the example of FIG. 2, the estimated time zone of the transmission image G1 is the time tb. (The estimated time zone of the transmission image G2 is the time zone of the time TC before the time tc).

  The image recognizing unit is configured to assume an estimated shooting location when the transmission image is shot by the terminal that has transmitted the transmission image (for example, in the example of FIG. 1, of the information display devices 3-1 to 3-3). Any installation location) is estimated as one of the estimated shooting situations.

Image storage means (for example, the image storage unit 21 in FIG. 3) that stores a plurality of stored images that can be the display image of the display device,
The image recognizing unit extracts one or more accumulated images from the image accumulating unit that can be determined to be the display image that is actually displayed on the display device under the estimated photographing situation, and the extracted one or more If the image corresponding to the transmission image is included in the stored image, the transmission image is recognized as a valid image, and the transmission image is included in the one or more extracted stored images. When the corresponding image is not included or when the stored image is not extracted, the transmission image is recognized as an illegal image.

Transmission means (for example, the transmission unit 29 in FIG. 3) for transmitting the related information of the display image (for example, related information stored in the related information storage unit 28 in FIG. 3) to the terminal;
When the transmission image is recognized as a valid image by the image recognition means, the transmission of the related information from the transmission means is permitted, and the transmission image is recognized as an illegal image by the image recognition means. In this case, it further comprises transmission permission determining means (for example, transmission information determining section 27 in FIG. 3) for prohibiting transmission of the related information from the transmitting means.

An information processing method according to one aspect of the present invention (for example, a method corresponding to the server-side related information providing process in FIGS. 5 and 6) is as follows.
A transmission image (for example, the transmission image G in FIG. 3) transmitted from a terminal (for example, the portable terminal 1 in FIG. 3) is directly displayed on the display device (for example, the information display device 3 in FIG. 3). An information processing method of an information processing apparatus (for example, the server 2 in FIG. 3) that performs a validity determination as to whether or not a legitimate image obtained as a result of photographing,
The transmission image is received (for example, step S22 in FIG. 5 or step S42 in FIG. 6),
Obtain information about the received transmission image,
Based on the acquired information, the estimated shooting situation of the transmission image is estimated (for example, “specified past time” in step S25 of FIG. 5 or step S43 of FIG. 6),
When the display image that is actually displayed on the display device under the estimated photographing condition is an image corresponding to the transmission image, the transmission image is recognized as a valid image and corresponds to the transmission image. If it is not an image, the transmission image is recognized as an illegal image (for example, steps S23 to S25 in FIG. 5 and steps S44 to S46 in FIG. 6).

  The program according to one aspect of the present invention is a program corresponding to the information processing method according to one aspect of the present invention described above, and is executed by, for example, the computer of FIG.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows a configuration example of an information processing system to which the present invention is applied.

  In the example of FIG. 1, the information processing system includes a mobile terminal 1-1 of user U1, a mobile terminal 1-2 of user U2, a server 2, and information display devices 3-1 to 3-3.

  The number of mobile terminals 1 constituting the information processing system is not limited to two in FIG. 1 and may be any number as long as it is one or more. Similarly, the number of information display devices 3 constituting the information processing system is not limited to three in FIG. 1, and may be any number as long as it is one or more. Further, the number of servers 2 constituting the information processing system is not limited to one in FIG. 1 and may be a plurality.

  Hereinafter, the portable terminals 1-1 and 1-2 will be referred to as portable terminals 1 when it is not necessary to distinguish them individually. Accordingly, when the mobile terminal 1 is referred to, the users U1 and U2 are simply referred to as the user U without being distinguished from each other. Similarly, when it is not necessary to individually distinguish the information display devices 3-1 to 3-3, they are simply referred to as the information display device 3.

  The information display device 3 displays an image corresponding to the image signal provided from the server 2 via the network 4.

  In the following description, the image signal is regarded as a change in form when the image is transmitted, and is simply referred to as an image is provided from the server 2 to the information display device 3. That is, when various information such as images is transmitted from a predetermined one of a plurality of devices constituting the information processing system to another one, the information is converted into a signal in a predetermined form. This is referred to as information being transmitted. For the same purpose, hereinafter, the information is simply referred to as being recorded on a predetermined recording medium. In other words, since the form of information transmission / recording is not particularly limited, the form is not particularly referred to, and is simply referred to as information transmission / recording.

  In addition, the image displayed by the information display device 3 is not particularly limited. For example, in the present embodiment, the image is an advertisement image in order to facilitate comparison with the conventional example in [Background Art] described above. To do. The image here is an image of a broad concept including both a still image and a moving image. Furthermore, each display content of the information display devices 3-1 to 3-3 is basically independent of the others. That is, although details will be described later, in each of the information display devices 3-1 to 3-3, the same advertisement image may be displayed in the same time zone, or separate advertisement images may be displayed. There is also.

  The server 2 holds a plurality of advertisement images and performs at least display destination management and time management in order to display the plurality of advertisement images on the information display device 3. In the example of FIG. 1, information display devices 3-1 to 3-3 exist as display destination candidates for each advertisement image. That is, the process of designating one or more information display devices 3-1 to 3-3 as the display destination of the predetermined advertisement image is executed as one of display destination management. Further, a process for designating a time zone for displaying the predetermined advertisement image is executed as one of the time managements for the information display device 3 designated as the display destination.

  That is, by the time management and display destination management by the server 2, a predetermined advertisement image is displayed at the designated display destination in the designated time zone.

  It should be noted here that the designated display destination of a predetermined advertisement image is not limited to one information display device 3. That is, depending on how the display destination is managed, the predetermined advertisement image may be displayed only on a predetermined one of the information display devices 3-1 to 3-3, or may be displayed only on the predetermined two. Or displayed on all three devices. Furthermore, even when the predetermined advertisement image is displayed on a plurality of information display devices 3, depending on the combination with time management, it may be displayed in the same time zone, or in different time zones. Sometimes it is displayed.

  Furthermore, the time management and display destination management itself by the server 2 are not particularly limited, and the server 2 may independently determine and manage, or may manage according to a predefined rule. An example of the former management is management performed by the real-time image distribution unit 22 in FIG. 3 to be described later. On the other hand, an example of the latter management is management performed in accordance with a display table held in the display table holding unit 32 of FIG.

  In this way, each of the plurality of advertisement images held in the server 2 is managed so as to be displayed on the designated information display device 3 in the designated time zone. Further, some of these advertisement images are associated with predetermined information, and information associated with the advertisement image (hereinafter referred to as related information) is also held in the server 2. The content of the related information and the method of associating are not particularly limited.

  For example, in this embodiment, in order to facilitate comparison with the conventional example in [Background Art] described above, a coupon ticket distributed only to the user U who has seen a predetermined advertisement image is Is stored in the server 2 as one of the related information of the advertisement image.

  In this case, as a proof that the user has actually seen the advertisement image, the user U uses the mobile terminal 1 to photograph the advertisement image displayed on the information display device 3 as a subject, and uses the photographed image as a transmission image in the server. 2 to send. Then, the server 2 determines the validity of the transmission image. That is, the server 2 determines whether or not the transmission image is obtained as a result of directly capturing an advertisement image. When the server 2 determines that the transmission image is a legitimate image, that is, when the server 2 determines that the transmission image is obtained as a result of directly capturing the advertisement image, the transmission source mobile terminal 1 Send coupons to. On the other hand, when the server 2 determines that the transmission image is an illegal image, that is, when the transmission image is determined not to be obtained as a result of directly capturing the advertisement image, the transmission source The portable terminal 1 is prohibited from transmitting coupon coupons. That is, the coupon is distributed only to the mobile terminal 1 that has transmitted a legitimate image, and is not distributed to the mobile terminal 1 that has transmitted an illegal image.

  Here, the portable terminal 1 is considered separately according to the example of FIG. In this case, the portable terminal 1-1 is operated by the user U1 who actually viewed the advertisement image displayed on the information display device 3-3. That is, the portable terminal 1-1 directly captures the advertisement image displayed on the information display device 3-3, and directly transmits the captured image as the transmission image G1 to the server 2.

  On the other hand, the portable terminal 1-2 is operated by the user U2 who has not seen the advertisement image displayed on the information display device 3-3. However, the mobile terminal 1-2 receives the transmission image G1 transferred from the mobile terminal 1-1, and then transmits the transmission image G2 corresponding to the transmission image G1 to the server 2.

  That is, the transmission image G1 from the portable terminal 1-1 is a valid image, while the transmission image G2 from the portable terminal 1-2 is an illegal image. Accordingly, the coupon ticket is originally allowed to be distributed only to the mobile terminal 1-1 and must be prohibited from being distributed to the mobile terminal 1-2.

  However, when only the legitimacy determination based on the conventional image feature is performed, the coupon is also distributed to the mobile terminal 1-2. That is, according to the conventional legitimacy determination, since the transmission image G1 and the transmission image G2 have almost no difference in image characteristics, it is determined that not only the transmission image G1 but also the transmission image G2 is a legitimate image. The As a result, the related information I as a coupon is distributed not only to the mobile terminal 1-1 but also to the mobile terminal 1-2.

  Therefore, the transmission image G1 from the portable terminal 1-1 is valid, but the transmission image G2 from the portable terminal 1-2 is correctly determined to be invalid, that is, the accuracy of the validity determination of the transmission image is increased. The following method has been invented by the present inventors to achieve this. Hereinafter, when the portable terminal 1-1 and the portable terminal 1-2 are referred to as the portable terminal 1 without being distinguished from each other, the transmission images G1 and G2 are also simply referred to as the transmission image G without being distinguished from each other. In this case, it is assumed that the transmission image G is an image obtained as a result of the mobile terminal 1 directly photographing the display image of the information display device 3 at the time of reception by the server 2. Under this assumption, the server 2 estimates the shooting situation of the transmission image G. Then, the server 2 determines whether or not the transmission image G matches the advertisement image actually displayed on the information display device 3 under the estimated photographing situation, that is, whether or not the transmission image G is an image corresponding to the advertisement image. Thus, the validity of the transmission image G is determined. That is, when the transmission image G matches the advertisement image actually displayed on the information display device 3 under the estimated shooting situation, the transmission image G is determined to be a valid image. Otherwise, it is determined that the transmission image G is an illegal image. The technique for realizing the above series of processes is the technique invented by the present inventors.

  Note that the estimated shooting situation may be any situation that can be estimated using information that can be directly acquired by the server 2 from the transmission image G, or information that can be indirectly acquired from the reception status of the transmission image G. For example, an estimated shooting time when it is assumed that the transmission image G was shot by the mobile terminal 1 that has transmitted the transmission image G can be adopted as the estimated shooting situation. However, considering that it is difficult to specify the estimated shooting time, in this embodiment, it is assumed that an estimated shooting time zone having a certain time width is adopted as an example of the estimated shooting situation.

  Therefore, referring to FIG. 2, the transmission image G1 from the mobile terminal 1-1 and the transmission image G2 from the mobile terminal 1-2 in FIG. A processing example of the validity determination of the server 2 for each of the above will be described.

  In FIG. 2, the outline of each timing chart of the portable terminal 1-1, the portable terminal 1-2, the server 2, and the information display device 3-3 is shown in order from the top. That is, each arrow drawn to the right from each block indicating the mobile terminal 1-1, the mobile terminal 1-2, the server 2, and the information display device 3-3 indicates the same time axis. A square on the arrow indicates a predetermined process executed in that time zone.

  For example, if the approximate time of the processing time required for the portable terminal 1 to capture and transmit the image and the transmission time of the image from the portable terminal 1 to the server 2 can be grasped, the processing time A fixed time TC obtained by adding a grace time to the transmission time can be defined, and a time zone corresponding to the fixed time TC before the reception time of the transmission image G in the server 2 can be adopted as the estimated shooting time zone.

  In this case, for example, an advertisement image is displayed on the information display device 3-3 at time ta, the advertisement image is directly captured by the mobile terminal 1-1, the captured image is transmitted as the transmission image G1, and at time tb. Suppose that the server 2 receives the request. Then, the server 2 determines a time zone for a certain time TC before the reception time tb of the transmission image G1 as an estimated shooting time zone of the transmission image G1.

  In this case, if the above-described fixed time TC is appropriately set, as shown in FIG. 2, the actual shooting time ta is included in the estimated shooting time zone of the transmission image G1. Therefore, among the one or more advertisement images actually displayed on the information display device 3-3 within the estimated shooting time zone of the transmission image G1, it is actually displayed on the information display device 3-3 at time ta. Naturally, the advertisement image that had been included will be included. Accordingly, the advertisement image and the transmission image G1 match, and as a result, the transmission image G1 is determined to be valid.

  On the other hand, the advertisement image displayed on the information display device 3-3 at the time ta is captured by the mobile terminal 1-1, and the captured image is transmitted as the transmission image G1 from the mobile terminal 1-1 to the mobile terminal 1-2. , And then a transmission image G2 corresponding to the transmission image G1 is transmitted from the portable terminal 1-2 and received by the server 2 at time tc. Further, the time from time ta to time tc is described as TF.

  In this case, the server 2 determines a time zone for a certain time TC before the reception time tc of the transmission image G2 as an estimated shooting time zone.

  However, as shown in FIG. 2, the time TF includes the time required for the transfer operation performed by the user U1 to transfer the transmission image G1 from the mobile terminal 1-1 to the mobile terminal 1-2. The time required for the transfer operation performed by the user U2 to transfer the transmission image G2 corresponding to the transmission image G1 from the portable terminal 1-2 to the server 2 is included. Since the time required for each transfer operation of the users U1 and U2 is generally much longer than the time TC, the time TF is also much longer than the time TC.

  Therefore, as shown in FIG. 2, the time ta is not included in the estimated shooting time zone of the transmission image G2, that is, in the time zone of the past fixed time TC from the reception time tc of the transmission image G2. Become. That is, the image that matches the transmission image G2 is the advertisement image that was displayed on the information display device 3-3 at the time ta as in the case of the transmission image G1, but the advertisement image is the estimated shooting time of the transmission image G2. It is not displayed on the information display device 3-3 in the band. For this reason, the transmission image G2 does not match any advertisement image actually displayed on the information display device 3 within the estimated shooting time period, and as a result, the transmission image G2 is determined to be illegal. become.

  The estimated shooting time zone of the transmission image G is estimated based on the reception time of the transmission image G in the example of FIG. 2, but is not particularly limited to the example of FIG. For example, if the transmission image G itself or accompanying information such as meta information thereof includes information that can specify the shooting time, the estimated shooting time zone of the transmission image G is estimated using such information. You can also

  The example of FIG. 2 is an example in the case where the estimated shooting time zone is adopted as the estimated shooting situation. Therefore, the partner of the matching process with the transmission image G1 from the portable terminal 1-1 of FIG. To the past fixed time TC, that is, not only the advertisement image displayed on the information display device 3-3 within the estimated shooting time zone of the transmission image G1, but also the information display device 3 within the estimated shooting time zone. The advertisement image displayed on each of -1, 3-2 is also included. The same applies to the transmission image G2 from the portable terminal 1-2.

  In this case, when the advertisement image associated with the related information I is displayed only on the information display device 3-3 at the time ta, the advertisement displayed on each of the information display devices 3-1 and 3-2. The matching process between the image and the transmission image G is essentially unnecessary. Therefore, in order to omit such an originally unnecessary matching process, an estimated shooting location of the transmission image G may be adopted as the estimated shooting situation in addition to the estimated shooting time zone of the transmission image G. That is, if the information display device 3-3 can be specified as the estimated shooting location of the transmission images G1 and G2, it is displayed on the information display device 3-3 within the estimated shooting time zone as a partner for matching processing with the transmission images G1 and G2. The advertisement image displayed on each of the information display devices 3-1 and 3-2 within the estimated shooting time period can be excluded.

  Note that the estimation method of the estimated shooting location is not particularly limited. For example, information that can specify the shooting location, specifically, for example, text information indicating the display location (hereinafter, such information is referred to as specific information) is included in the advertisement image, and the mobile terminal 1 captures the advertisement image. Then, the transmission image G is transmitted, and the server 2 can extract the specific information from the transmission image G and estimate the estimated shooting location from the specific information. Further, for example, if the mobile terminal 1 has a function of outputting its own position information by GPS (Global Positioning System) or the like, the mobile terminal 1 transmits the position information to the server 2 together with the transmission image G, A method in which the server 2 estimates an estimated shooting location from the position information can be employed.

  The method of the present invention described above, that is, the method of determining the validity of the transmission image G in consideration of not only the image characteristics of the transmission image G but also the estimated shooting situation of the transmission image G is the information processing system of FIG. For example, the mobile terminal 1 and the server 2 can be realized so as to have the functional configurations shown in FIG. That is, FIG. 3 shows an example of each functional configuration of the mobile terminal 1 and the server 2 to which the present invention is applied.

  The mobile terminal 1 is configured to include an image acquisition unit 11 to a related information processing unit 15.

  The image acquisition unit 11 acquires the captured image by capturing the subject. The acquired captured image is provided to the transmission unit 12. That is, the image acquisition unit 11 can photograph the advertisement image displayed on the information display device 3 as a subject and provide the photographed image to the transmission unit 12.

  The transmission unit 12 transmits the captured image from the image acquisition unit 11 to the server 2 as a transmission image G. Note that the transmission timing of the transmission image G is not particularly limited, but when the estimated shooting time period described above is used for determining the validity of the transmission image G, the timing as early as possible from the shooting timing of the image acquisition unit 11. It is good to do. Therefore, for example, in the present embodiment, when the captured image is provided from the image acquisition unit 11, the transmission unit 12 does not wait for the operation of the user U, but immediately, in other words, automatically the captured image. Is transmitted as a transmission image G.

  On the other hand, although not shown, when the transmission unit 12 transmits information other than the captured image from the image acquisition unit 11, the transmission timing is assumed to be a timing when a specific operation is performed by the user. For example, the transmission unit 12 of the portable terminal 1-2 in FIG. 1 transmits the transmission image G2 when the user U2 performs a transfer operation after receiving the transmission image G1 transferred from the portable terminal 1-1. Then.

  In this way, a photographed image obtained as a result of photographing by the image acquisition unit 11 with the information display device 3 as a subject in a state where a predetermined advertisement image is displayed on the information display device 3 is transmitted from the transmission unit 12. The image G is transmitted and received by the server 2. Then, as described above, the server 2 determines that the transmission image G is a legitimate image, and the predetermined advertisement image corresponding to the transmission image G, that is, the predetermined advertisement displayed on the information display device 3. Image related information I is transmitted from the server 2.

  Therefore, the receiving unit 13 receives the related information I and causes the related information recording unit 14 to record it. The related information processing unit 15 executes processing related to the related information I recorded in the related information recording unit 14.

  Here, some specific examples of processing related to the related information I will be described.

  For example, when the coupon is transmitted as the related information I as in the above-described example, the process of displaying the coupon on the display (not shown) of the mobile terminal 1 is an example of the process related to the related information I. is there.

  Further, for example, it is assumed that the store in FIG. 1 performs a member service that gives points to the person who actually visited the store for the purpose of increasing the number of visitors. In this case, the validity determination of the transmission image G can be used as a determination as to whether or not the user U who performed the transmission operation of the transmission image G is a person who actually visited the store. In this case, a point addition permission flag can be employed as the related information I. That is, when the server 2 determines that the transmission image G is a valid image, the server 2 determines that the user U who performed the transmission operation of the transmission image G is a person who actually visited the store, and sets the point addition permission flag As information I, it can transmit with respect to the portable terminal 1 of the user U operation object. In addition, since the point addition permission flag is basically independent information from the coupon, it may be transmitted together with the coupon or may be transmitted separately.

  When such a point addition permission flag is transmitted as the related information I, the related information processing unit 15 executes a process of adding a specified point to the point information of the user U as a process related to the related information I. be able to. However, there is a case where such a process is restricted to be executed only once a day.

  In addition, the concept itself of such a store visit point addition system has existed conventionally. However, the conventional store point addition system has been realized with a medium such as a magnetic card or non-contact communication. Therefore, the following various effects that could not be achieved by the conventional store visit point adding system are as follows. The store visit point adding system using the information processing system to which the present invention is applied (hereinafter referred to as the store visit point adding system of the present invention). Will be able to play.

  That is, in the conventional store visit point addition system, as a point addition operation, an operation using a dedicated terminal capable of receiving a medium such as a magnetic card or non-contact communication is required. For this reason, when there are a plurality of users U, there is a problem that there are many cases of waiting in order to use the dedicated terminal.

  The store visit point adding system of the present invention can achieve the effect that such a problem can be solved. This is because, in the store visit point addition system of the present invention, as the point addition operation, only the operation of photographing the information display device 3 from a remote location using the portable terminal 1 is sufficient. In other words, this point addition operation can be executed by several users U at the same time.

  In addition, in the store visit point addition system of the present invention, all the users U inevitably see the advertisement image of the information display device 3 when performing the point addition operation, and the advertisement effect of the advertisement image is enhanced. Such effects can be achieved.

  Furthermore, in the store visit point addition system of the present invention, the user U can obtain various related information in addition to the point addition permission flag. Examples of the related information herein include information such as advertisements and announcements desired by the user U, in addition to the coupons described above. Further, for example, the server 2 determines the legitimacy of the transmission image G, and as a result, the display destination of the advertising image that matches the transmission image G, that is, the location where the user U exists can be known. Therefore, for example, information indicating a route from the location where the user U exists to another location can be adopted as one of the related information. Thereby, the route can be presented on the user's portable terminal 1.

  For the mobile terminal 1 having such a functional configuration, the server 2 is configured to include an image storage unit 21 to a transmission unit 29.

  The image storage unit 21 stores one or more advertisement images (still images / moving images).

  The real-time image distribution unit 22 appropriately reads a predetermined advertisement image stored in the image storage unit 21 and distributes it to the information display device 3 in real time via the network 4. That is, the real time image distribution unit 22 autonomously performs display destination management and time management in real time for each of one or more advertisement images stored in the image storage unit 21.

  The advertisement image distributed from the real-time image distribution unit 22 is provided to the information display device 3 via the network 4 and displayed by the information display device 3.

  The display information acquisition unit 23 acquires information (hereinafter referred to as display information) related to the display of the advertisement image on the information display device 3 from the information display device 3 via the network 4 or from the real-time image distribution unit 22. To do.

  As the display information, for example, the management information of the real-time image distribution unit 22, that is, each distribution time zone or each distribution location of one or more advertisement images stored in the image storage unit 21 is acquired by the display information acquisition unit 23. The

  Further, for example, a time required for a predetermined advertisement image to be delivered from the real-time image delivery unit 22, provided to the information display device 3 via the network 4, and displayed on the information display device 3 (hereinafter referred to as display) (Referred to as delay time) is acquired by the display information acquisition unit 23 as display information.

  In this case, the time zone in which the predetermined advertisement image is actually displayed on the information display device 3 (hereinafter referred to as the display time zone) can be calculated by adding the display delay time to the distribution time zone. In addition, the display location of the predetermined advertisement image, that is, the information display device 3 displaying the predetermined advertisement image can be specified from the distribution location.

  Therefore, the display information acquisition unit 23 identifies each display time zone and each display location for each of the one or more advertisement images stored in the image storage unit 21 from the acquired display information, and sends them to the image recognition unit 26. provide.

  On the other hand, the advertisement image displayed on the information display device 3 is captured by the mobile terminal 1, and the captured image is transmitted from the mobile terminal 1 as the transmission image G. Therefore, the reception unit 24 of the server 2 receives this transmission image G and provides it to the image recognition unit 26.

  At that time, the transmission image information acquisition unit 25 acquires information on the transmission image G received by the reception unit 24 (hereinafter referred to as transmission image information) and provides the information to the image recognition unit 26 in an appropriate form.

  Here, the transmission image information is information that can be acquired indirectly based on the reception state of the transmission image G in the receiving unit 24 in addition to information that can be directly acquired from the transmission image G (hereinafter referred to as direct acquisition information). (Hereinafter referred to as indirect acquisition information).

  For example, each piece of pixel information constituting the transmission image G, metadata of the transmission image G, information generated based on them, and the like are examples of directly acquired information.

  Specifically, for example, as described above, when specific information that can specify the shooting location is included in the advertisement image, each pixel information about the image region indicating the specific information included in the transmission image G is directly acquired. Information is acquired by the transmission image information acquisition unit 25 as information. Then, the transmission image information acquisition unit 25 recognizes specific information from each pixel information and provides the specific information to the image recognition unit 26.

  Further, for example, when position information such as GPS information is included in the metadata, the transmission image information acquisition unit 25 acquires the position information directly as acquisition information and provides it to the image recognition unit 26.

  On the other hand, the reception time of the transmission image G in the reception unit 24 is an example of indirect acquisition information, and the reception time is also provided to the image recognition unit 26.

  Based on the transmission image information provided from the transmission image information acquisition unit 25 and the display information provided from the display information acquisition unit 23, the image recognition unit 26 estimates an estimated shooting state of the transmission image G provided from the reception unit 24. Is identified. The image recognizing unit 26 confirms whether the transmission image G matches the advertisement image actually displayed on the information display device 3 under the estimated shooting condition, thereby verifying the validity of the transmission image G. Make a decision. The determination result is provided to the transmission information determination unit 27.

  In this case, as the estimated shooting situation, for example, in the present embodiment, as described above, an estimated shooting time zone and an estimated shooting location are employed. The estimated shooting time zone can be estimated from the reception time at the reception unit 24 of the transmission image G of the transmission image information provided from the transmission image information acquisition unit 25, as described with reference to FIG. is there. As for the estimated shooting location, for example, as described above, of the transmission image information provided from the transmission image information acquisition unit 25, the specific information recognized from the transmission image G and the position extracted from the metadata It can be estimated from information.

  Further, as described above, display information provided from the display information acquisition unit 23 is used to specify an advertisement image that is actually displayed in the estimated shooting time zone at the estimated shooting location (predetermined information display device 3). Can be used.

  In this way, when the validity of the transmission image G is determined by the image recognition unit 26, the transmission information determination unit 27 restricts transmission of the related information I associated with the advertisement image based on the determination result. Determine whether or not.

  When it is determined that the transmission image G is a valid image, the transmission information determination unit 27 determines that transmission of the related information I is not restricted, and transmits the related information I from the related information storage unit 28. The data is read out and provided to the transmission unit 29. Then, the transmission unit 29 transmits the related information I to the mobile terminal 1 that has transmitted the transmission image G.

  On the other hand, when the transmission image G is recognized as an unauthorized image, the transmission information determination unit 27 determines to limit transmission of the related information I. That is, transmission of the related information I from the transmission unit 29 is prohibited.

  Hereinafter, an example of a series of processes performed until the user U operates the mobile terminal 1 to acquire related information among the processes performed between the mobile terminal 1 having the functional configuration and the server 2. explain. Hereinafter, of the series of processes, a process example executed on the mobile terminal 1 side will be referred to as “related information acquisition process on the mobile terminal side”, and a process example executed on the server 2 side will be referred to as “server side This is referred to as “related information provision processing”.

  First, the “related information acquisition process on the portable terminal side” will be described with reference to the flowchart of FIG.

  In step S <b> 1, the mobile terminal 1 determines whether acquisition of related information is instructed.

  If it is determined in step S1 that acquisition of related information is not instructed, the process returns to step S1 again. In other words, the loop process of step S1 is repeatedly executed until acquisition of related information is instructed.

  Thereafter, when a button (not shown) is pressed by the user U and acquisition of related information is instructed, it is determined as YES in the process of step S1, and the process proceeds to step S2.

  In step S <b> 2, the mobile terminal 1 executes a process for starting application software for acquiring related information. When such application software starts up, the process proceeds to step S3.

  In step S <b> 3, the mobile terminal 1 determines whether the camera shooting mode is selected.

  If it is determined in step S3 that the camera shooting mode is not selected, the process returns to step S3 again. That is, until the camera shooting mode is selected, the loop process of step S3 is repeatedly executed.

  Thereafter, when a button (not shown) or the like is pressed by the user U and the camera shooting mode is selected, it is determined as YES in the process of step S3, and the process proceeds to step S4.

  In step S4, the mobile terminal 1 determines whether or not an image is captured by the image acquisition unit 11 (FIG. 3).

  If it is determined in step S4 that no image has been taken, the process returns to step S4 again. That is, until the image is captured, the loop process in step S4 is repeatedly executed.

  Thereafter, when the user U presses a shutter button (not shown) or the like and the image acquisition unit 11 captures the subject and provides the captured image to the transmission unit 12, it is determined YES in the process of step S4. The process proceeds to step S5.

  In step S <b> 5, the mobile terminal 1 transmits the captured image as a transmission image G from the transmission unit 12 to the server 2.

  In step S <b> 6, the mobile terminal 1 determines whether the related information I has been transmitted from the server 2.

  The captured image obtained as a result of the process of step S4, that is, the transmission image G transmitted to the server 2 in the process of step S5 is an image obtained as a result of directly capturing the advertisement image displayed on the information display device 3. If there is, the related information I of the advertisement image is transmitted to the portable terminal 1 by the process of step S26 (FIG. 5) or S47 (FIG. 6) of the “server-side related information providing process” described later.

  In such a case, the portable terminal 1 determines YES in Step S6, and receives the related information I in the receiving unit 13 (FIG. 3) in Step S7.

  When the related information is recorded in the related information recording unit 14, the process proceeds to step S8. That is, in step S8, the related information processing unit 15 (FIG. 3) of the mobile terminal 1 executes processing related to the related information I. In addition, since the specific example of the process regarding the related information I was mentioned above, the description is abbreviate | omitted here.

  When the process of step S8 ends, that is, when the process related to related information ends, the entire “related information acquisition process on the mobile terminal side” also ends.

  On the other hand, the captured image obtained as a result of the process in step S4, that is, the transmission image G transmitted to the server 2 in the process of step S5 is different from the advertisement image displayed on the information display device 3. If the image is obtained as a subject, transmission of the related information I to the mobile terminal 1 by the processing in step S27 (FIG. 5) or S48 (FIG. 6) of “related information providing processing on the server side” described later. Is prohibited. Here, prohibiting transmission of the related information I to the mobile terminal 1 means that no information is transmitted from the server 2 to the mobile terminal 1, and error information or the like is used instead of the related information I. This is a broad concept including that the information is transmitted from the server 2 to the portable terminal 1.

  In such a case, the mobile terminal 1 determines NO in step S6 and executes a predetermined error process. The contents of error processing are not particularly limited. For example, the mobile terminal 1 may perform a predetermined error display as the process of step S9.

  Although not shown in the flowchart, a case where an image acquired by an unauthorized means is transmitted to the server 2 as the transmission image G, as in the transmission image G2 of the portable terminal 1-2 in FIG. The transmission of the related information I to the portable terminal 1 is prohibited by the process of step S27 (FIG. 5) or S48 (FIG. 6) of the “related information providing process on the server side”. Accordingly, even in such a case, error processing similar to that in step S9 is executed.

  Two examples of the “related information providing process on the server side” are shown in the flowcharts of FIGS. 5 and 6 in contrast to the “related information acquiring process on the portable terminal side” in FIG.

  First, the “related information providing process on the server side” in the example of FIG. 5 will be described.

  In step S <b> 21, the receiving unit 24 of the server 2 in FIG. 3 determines whether an image has been transmitted from the mobile terminal 1.

  If it is determined in step S21 that no image has been transmitted from the mobile terminal 1, the process returns to step S21 again. That is, until the image is transmitted from the mobile terminal 1, the loop process of step S21 is repeatedly executed.

  It should be noted that although only one mobile terminal 1 is shown in the example of FIG. 3, a plurality of mobile terminals 1 may actually exist. That is, the loop process of step S21 is repeatedly executed only when no image is transmitted from any of the plurality of portable terminals 1.

  On the other hand, when the transmission image G is transmitted from one or more of the plurality of mobile terminals 1, it is determined as YES in the process of step S21, and each transmission image G has been transmitted 1 For each of the mobile terminals 1 described above, the processing after the next step S22 is executed independently and in parallel. However, hereinafter, for the sake of simplicity, it is assumed that the transmission image G has been transmitted from one mobile terminal 1, and the description after step S22 will be made.

  In step S <b> 22, the reception unit 24 receives the transmission image G from the mobile terminal 1 and provides it to the image recognition unit 26.

  In step S <b> 23, the image recognition unit 26 executes a matching process between the transmission image G and each of all the stored images in the image storage unit 21. Note that the accumulated image here refers to the advertisement image in the above-described example. However, as described above, the advertisement image is merely an example of an image to be displayed on the information display device 3, and therefore the term “stored image” is simply used in the examples of FIGS. 5 and 6.

  In step S24, the image recognition unit 26 determines whether there is an accumulated image that matches the transmission image.

  If there is an accumulated image that matches the transmitted image, it is determined YES in the process of step S24, and the process proceeds to step S25. In step S <b> 25, the image recognition unit 26 determines whether the stored image is an image displayed on the information display device 3 within a certain past time.

  That is, the “predetermined time period” in the process of step S25 indicates the estimated shooting time zone for the transmission image G acquired in the process of step S22. Specifically, for example, when the processing timing of step S22 is the time tb in the example of FIG. 2, the time zone TC before the time tb is the “predetermined past time”. As described above, the “predetermined time period” can be specified by the processing timing of step S22, that is, the reception time of the transmission image G in the reception unit 24. Further, as described above, “the image displayed on the information display device 3 within a certain past time” can be specified by the display information from the display information acquisition unit 23.

  The “information display device 3” in the process of step S25 refers to the information display device 3 specified as the estimated shooting location when the estimated shooting location is specified, and the estimated shooting location is specified. When there is not, it means all of the one or more information display devices 3 managed as display destination candidates by the real-time image distribution unit 22.

  That is, if the estimated shooting location is not specified, whether or not an accumulated image exists in one or more images displayed on each of the one or more information display devices 3 within the past certain time. Is executed as the process of step S25.

  The case where the stored image that matches the transmission image is an image displayed on the information display device 3 within a certain period of time in the past means that the transmission image G is a valid image. Therefore, in such a case, the determination result of YES in the process of step S25 is provided from the image recognition unit 26 to the transmission information determination unit 27 as the determination result that the transmission image G is a valid image. Thereby, a process progresses to step S26.

  That is, in step S <b> 26, the transmission information determination unit 27 reads the related information I from the related information storage unit 28 and transmits the related information I from the transmission unit 29 to the mobile terminal 1. Thereby, the “related information providing process on the server side” is completed.

  Specifically, for example, a photographed image obtained as a result of the process in step S4 in FIG. 4 described above, that is, a transmission image G transmitted to the server 2 in the process in step S5 is displayed on the information display device 3. If it is an image obtained as a result of directly capturing an image, it is determined YES in each process of steps S24 and S25, and the related information I is transmitted to the portable terminal 1 in the process of step S26. Become.

  On the other hand, the photographed image obtained as a result of the process in step S4 in FIG. 4, that is, the transmission image G transmitted to the server 2 in the process in step S5 is the advertisement image displayed on the information display device 3. When it is an image obtained by using another object as an object, or when an image obtained by an unauthorized means such as the portable terminal 1-2 in FIG. 1 is transmitted to the server 2 as the transmission image G, It is determined NO in the process of step S24 or S25, and the process proceeds to step S27.

  That is, the case where it is determined as NO in step S24 or S25 means that the transmission image G is an illegal image. Therefore, in such a case, the determination result of NO in step S24 or S25 is provided from the image recognition unit 26 to the transmission information determination unit 27 as the determination result that the transmission image G is an illegal image. . Thereby, a process progresses to step S27.

  In step S27, the transmission information determining unit 27 prohibits transmission of the related information I. As described above, the process in step S27 is a process for prohibiting transmission of related information I, and is not a process for prohibiting transmission of other information such as error information. Therefore, as a result of the processing in step S27, no information may be transmitted from the transmission unit 29 to the portable terminal 1, or other information such as error information may be transmitted from the transmission unit 29 to the portable terminal 1. There is also.

  In this way, even when the process of step S27 is completed, the “relevant information providing process on the server side” ends.

  In the “related information provision processing on the server side” in the example of FIG. 5, the partner of matching processing for the transmission image G is all the stored images in the image storage unit 21. Therefore, when a large number of stored images are stored in the image storage unit 21, the matching process, that is, the process of step S23, is a process that requires a long time with a high load.

  Therefore, when it is desired to make the matching process even shorter and lighter, the server 2 executes, for example, the process of the example of FIG. 6 instead of the example of FIG. 5 as the “related information providing process on the server side”. can do.

  Note that the processes in steps S41 and S42 in FIG. 6 are basically the same as the processes in steps S21 and S22 in FIG. 5 described above. Therefore, hereinafter, the processing after step S43 will be described.

  In step S43, the image recognition unit 26 in FIG. 3 is based on the transmission image information from the transmission image acquisition unit 25, and the estimated shooting situation (the estimated shooting time zone and the estimated shooting of the transmission image G received in the process of step S43). Location).

  In addition, since the specific example of the specific method of the estimated shooting situation (such as the estimated shooting time zone and the estimated shooting location) has been described above, the description thereof is omitted here.

  In step S <b> 44, the image recognition unit 26 extracts one or more stored images that are actually displayed on the information display device 3 under the estimated shooting status from all the stored images of the image storage unit 21. Note that “one or more stored images actually displayed on the information display device 3 under the estimated photographing situation” can be specified by the display information from the display information acquisition unit 23 as described above.

  In step S45, the image recognizing unit 26 performs a matching process between the transmission image G and each of the one or more accumulated images extracted in the process of step S44.

  In the process of step S46, the image recognition unit 26 determines whether there is an accumulated image that matches the transmission image.

  The case where YES is determined in the process of step S46 means a case where the transmission image G is a valid image. Therefore, in such a case, the determination result of YES in the process of step S46 is provided from the image recognition unit 26 to the transmission information determination unit 27 as the determination result that the transmission image G is a valid image. Thereby, a process progresses to step S47.

  That is, in step S 47, the transmission information determination unit 27 reads the related information I from the related information storage unit 28 and transmits the related information I from the transmission unit 29 to the mobile terminal 1. Thereby, the “related information providing process on the server side” is completed.

  Specifically, for example, a photographed image obtained as a result of the process in step S4 in FIG. 4 described above, that is, a transmission image G transmitted to the server 2 in the process in step S5 is displayed on the information display device 3. If the image is obtained as a result of directly capturing an image, it is determined YES in the process of step S46, and the related information I is transmitted to the mobile terminal 1 in the process of step S47.

  On the other hand, the photographed image obtained as a result of the process in step S4 in FIG. 4, that is, the transmission image G transmitted to the server 2 in the process in step S5 is the advertisement image displayed on the information display device 3. When it is an image obtained by using another object as an object, or when an image obtained by an unauthorized means such as the portable terminal 1-2 in FIG. 1 is transmitted to the server 2 as the transmission image G, If it is determined as NO in the process of step S46, the process proceeds to step S48.

  That is, the case where it is determined as NO in the process of step S46 means a case where the transmission image G is an illegal image. Therefore, in such a case, the determination result of NO in the process of step S46 is provided from the image recognition unit 26 to the transmission information determination unit 27 as a determination result that the transmission image G is an illegal image. Thereby, the process proceeds to step S48.

  In step S48, the transmission information determination unit 27 prohibits transmission of the related information I. Note that the process in step S48 is a process for prohibiting transmission of related information I, as in the process in step S27 of FIG. 5 described above, and is not a process for prohibiting transmission of other information such as error information. Accordingly, as a result of the processing in step S48, no information may be transmitted from the transmission unit 29 to the portable terminal 1, or other information such as error information may be transmitted from the transmission unit 29 to the portable terminal 1. There is also.

  Thus, also when the process of step S48 is completed, the “relevant information providing process on the server side” ends.

  As described above, in the “relevant information providing process on the server side” in the example of FIG. 6, the process of steps S43 and S44 causes the partner of the matching process for the transmission image G to be “information display device under estimated shooting conditions”. 3 is reduced to “one or more accumulated images actually displayed”. Therefore, even when a large number of accumulated images are accumulated in the image accumulating unit 21, the matching process, that is, the process of step S45 is more lightly loaded than the process of step S23 of FIG. It is possible to achieve an effect that the processing can be performed in a short time.

  Such an effect becomes more prominent as the range of the estimated shooting situation specified in the process of step S43 is narrowed. This is because the number of “one or more stored images actually displayed on the information display device 3 under the estimated photographing situation” is reduced accordingly. Therefore, ultimately, if the estimated shooting situation is limited to the estimated shooting time instead of the estimated shooting time zone and can be limited to the place where the user U was exactly as the estimated shooting location, the counterpart of the matching process for the transmission image G, That is, the “one or more accumulated images that were actually displayed on the information display device 3 under the estimated shooting situation” are displayed on the information display device 3 of the estimated shooting location at the timing of the estimated shooting time. It becomes possible to be unique.

  The information processing system having the functional configuration shown in FIG. 3 has been described as an information processing system to which the present invention is applied. However, the present invention can take various embodiments other than the example of FIG.

  Specifically, for example, FIG. 7 is a functional configuration example of an information processing system to which the present invention is applied, and shows an example different from the example of FIG.

  7, portions corresponding to those in FIG. 3 are denoted by corresponding reference numerals, and description of such portions is omitted as appropriate.

  The mobile terminal 1 in the example of FIG. 7 has the same functional configuration as that in the example of FIG.

  In the server 2 in the example of FIG. 7, the real-time image distribution unit 22 and the display information acquisition unit 23 are omitted from the functional configuration in the example of FIG. 3. Instead, the display information distribution unit 31 and the display table are omitted. A holding part 32 is provided.

  That is, as described above, the time management and display destination management method of each stored image (advertisement image in the above example) stored in the image storage unit 21 is not particularly limited, and the server 2 determines independently. And may be managed according to a predefined rule.

  In this case, an example in which the former is managed is the server 2 in FIG. That is, as described above, the former management is performed by the real-time image distribution unit 22 of FIG.

  On the other hand, an example of performing the latter management is the server 2 in FIG. That is, in the example of FIG. 7, the display time zone and the display destination (specific information display device 3) of each accumulated image are set in advance, and the display table in which each setting content is described is stored in the display table. Held in the section 32. The display information distribution unit 31 distributes in advance the display table held in the display table holding unit 32 and each stored image stored in the image storage unit 21 to the information display device 3 via the network 4. .

  That is, in the example of FIG. 7, each stored image and display table on the server 2 are downloaded in advance to the information display device 3 via the network 4. For this reason, the information display device 3 in the example of FIG. 7 is configured to include an image storage unit 41 to an information display unit 44. That is, each accumulated image distributed in advance from the server 2 via the network 4 is accumulated in the image accumulation unit 41. On the other hand, the display table distributed in advance from the server 2 via the network 4 is held in the display table holding unit 42. The display content management unit 43 appropriately reads each stored image from the image storage unit 41 according to the content of the display table held in the display table holding unit 42 and causes the information display unit 44 to display it.

  Accordingly, in the example of FIG. 7, the image recognition unit 26 specifies the information processing apparatus when identifying the partner of the matching process with the transmission image G received by the reception unit 24, that is, under the estimated shooting situation of the transmission image G. 3, when one or more images actually displayed are specified, the contents of the display table held in the display table holding unit 32 are referred to.

  In the information processing system of the example of FIG. 3 and FIG. 7 described above, the processing result of the image recognition unit 26 of the server 2, that is, the validity judgment result of the transmission image G considering the estimated photographing state is Used to determine whether transmission is restricted. However, the validity judgment result of the transmission image G in consideration of the estimated photographing situation is not particularly limited to the examples of FIGS.

  Specifically, for example, FIG. 8 shows an information processing system to which the present invention is applied, and adds the result of determining the validity of the transmission image G in consideration of the estimated shooting situation to the uses of the examples of FIGS. 2 shows an example of the functional configuration of an information processing system used for another purpose.

  8, portions corresponding to those in FIG. 3 are denoted by corresponding reference numerals, and description of such portions is omitted as appropriate.

  Each of the mobile terminal 1 and the information display device 3 in the example of FIG. 8 has the same functional configuration as each of the examples in FIG.

  In the server 2 in the example of FIG. 8, a display content changing unit 51 and a marketing data storage unit 52 are further provided in addition to the functional configuration in the example of FIG.

  When the validity of the transmission image G is determined by the image recognition unit 26, the display content changing unit 51 displays the display content on the information display device 3 displaying the stored image that matches the transmission image G. Control to be changed. Note that the content of the display change is not particularly limited. For example, in order to further improve the accuracy of the validity determination of the transmission image G, a change that gives feedback to the information display device 3 again, specifically, for example, The display name changing unit 51 can execute a display change to a specific image so that the mobile terminal 1 of the user U takes a new image. That is, in this case, at the stage where the first transmission image G is transmitted from the mobile terminal 1, the server 2 uses the display content changing unit 51 to transmit the specific image described above without sending the related information. Control to display on the display device 3 is performed. Then, the user U operates the mobile terminal 1 to capture the specific image, and transmits the captured image to the server 2 as the second transmission image G. The server 2 transmits the related information I to the mobile terminal 1 only when the second transmission image G matches the specific image.

  The target to which feedback is applied in this case is a specific image in the example of FIG. 8, but is not limited to such an image, and may be, for example, voice. Specifically, for example, although not shown, when the information display device 3 is provided with a speaker and the mobile terminal 1 is provided with a microphone, the first transmission image G is transmitted from the mobile terminal 1. At this stage, the server 2 performs control to output a specific sound from the speaker of the information display device 3 without sending the related information. And the user U operates the portable terminal 1, acquires this specific sound with a microphone, and transmits the specific sound to the server 2 as a transmission sound. The server 2 transmits the related information I to the mobile terminal 1 only when the transmission sound matches the specific sound.

  The marketing data storage unit 52 stores the processing result of the image recognition unit 26 as marketing data. For example, when the number of subsets of the user U who viewed the advertisement image displayed on the information display device 3 is used as marketing data, the transmission image G is determined to be a valid image. While the number of subsets is incremented by 1, when it is determined that the transmission image G is an illegal image, the number of such subsets is stored in the marketing data storage unit 52 as the number of such subsets is not changed. Can be made.

  As described above, whether or not the transmission image G from the mobile terminal 1 is a direct image of the image actually displayed on the information display device 3 by the server 2 to which the present invention is applied. Make accurate decisions. Furthermore, the server 2 can restrict the download of the related information I based on the determination result, and as a result, the consumer (user U of the portable terminal 1) at the place where the information display device 3 is installed. It is possible to increase the value of the information display device 3 and its surroundings. Further, the determination result can be used for various other purposes. Specifically, for example, as described above, the marketing data can be used for counting the number of subsets of people (users U) who came in front of the information display device 3 and watched the advertisement. Further, for example, it can be used for the purpose of applying feedback to the screen in front of the consumer (user U) (display surface of the information display device 3).

  By the way, the series of processes described above can be executed by hardware, but can also be executed by software.

  In this case, for example, a personal computer shown in FIG. 9 may be employed as at least a part of the server 2 or the like described above.

  In FIG. 9, a CPU (Central Processing Unit) 101 executes various processes according to a program recorded in a ROM (Read Only Memory) 102 or a program loaded from a storage unit 108 to a RAM (Random Access Memory) 103. To do. The RAM 103 also appropriately stores data necessary for the CPU 101 to execute various processes.

  The CPU 101, ROM 102, and RAM 103 are connected to each other via a bus 104. An input / output interface 105 is also connected to the bus 104.

  The input / output interface 105 includes an input unit 106 including a keyboard and a mouse, an output unit 107 including a display, a storage unit 108 including a hard disk, and a communication unit 109 including a modem and a terminal adapter. It is connected. The communication unit 109 controls communication performed with other devices (not shown) via a network including the Internet.

  A drive 110 is connected to the input / output interface 105 as necessary, and a removable medium 111 made up of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached, and a computer program read from them is loaded. These are installed in the storage unit 108 as necessary.

  When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, a general-purpose personal computer is installed from a network or a recording medium.

  As shown in FIG. 9, the recording medium including such a program is distributed to provide a program to the user separately from the apparatus main body, and a magnetic disk (including a floppy disk) on which the program is recorded. Removable recording media (package) consisting of optical disks (including compact disk-read only memory (CD-ROM), DVD (digital versatile disk)), magneto-optical disk (including MD (mini-disk)), or semiconductor memory (Media) 111, but also a ROM 102 on which a program is recorded and a hard disk included in the storage unit 108 provided to the user in a state of being incorporated in the apparatus main body in advance.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the order, but is not necessarily performed in chronological order, either in parallel or individually. The process to be executed is also included.

  Further, in the present specification, the system represents the entire apparatus including a plurality of apparatuses and processing units.

It is a figure which shows the structural example of the information processing system to which this invention is applied. 2 is a timing chart illustrating an outline of one operation of the information processing system in FIG. 1. It is a functional block diagram which shows the functional structural example of each apparatus of the information processing system to which this invention is applied. It is a flowchart explaining the related information acquisition process example by the side of the portable terminal of FIG. It is a flowchart explaining the related information provision process example of the server side of FIG. FIG. 6 is a flowchart illustrating an example of related information providing processing on the server side in FIG. 3, which is an example different from FIG. 5. It is a functional block diagram which is an example of a functional structure of each apparatus of the information processing system to which this invention is applied, Comprising: The example different from the example of FIG. It is a functional block diagram which is an example of a functional structure of each apparatus of the information processing system to which this invention is applied, and shows an example different from the example of FIG. 3 and FIG. It is a block diagram which shows the structural example of the personal computer as an information processing apparatus with which this invention is applied.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Mobile terminal, 2 Server, 3 Information display apparatus, 4 Network, 11 Image acquisition part, 12 Transmission part, 13 Reception part, 14 Related information recording part, 15 Related information processing part, 21 Image storage part, 22 Real time image delivery part , 23 display information acquisition unit, 24 reception unit, 25 transmission image information acquisition unit, 26 image recognition unit, 27 transmission information determination unit, 28 related information storage unit, 29 transmission unit, 31 display information distribution unit, 32 display table holding unit , 41 Image storage unit, 42 Display table holding unit, 43 Display content management unit, 44 Information display unit, 101 CPU, 102 ROM, 103 RAM, 108 storage unit, 111 Removable media

Claims (8)

An information processing apparatus that determines whether or not a transmission image from a terminal is a legitimate image obtained as a result of direct imaging by the terminal using a display image of a display device as a subject,
Receiving means for receiving the transmission image;
Obtaining means for obtaining information on the transmission image received by the receiving means;
An estimated shooting situation of the transmission image is estimated based on the information acquired by the acquisition means, and a display image actually displayed on the display device under the estimated shooting situation is an image corresponding to the transmission image An image recognition unit that recognizes that the transmission image is a legitimate image and recognizes that the transmission image is an illegal image when the transmission image is not an image corresponding to the transmission image. The information processing apparatus according to claim 1, wherein the image recognition unit estimates an estimated shooting time zone when the transmission image is shot by the terminal that has transmitted the image as one of the estimated shooting situations. . The acquisition means acquires the reception time of the transmission image in the reception means as one of the information related to the transmission image,
The information processing apparatus according to claim 2, wherein the image recognition unit estimates a certain time period before the reception time acquired by the acquisition unit as the estimated shooting time period. The information processing apparatus according to claim 1, wherein the image recognizing unit estimates an estimated shooting location when the transmission image is shot by the terminal that has transmitted the transmission image as one of the estimated shooting situations. Image storage means for storing a plurality of stored images that can be the display image of the display device;
The image recognizing unit extracts one or more accumulated images from the image accumulating unit that can be determined to be the display image that is actually displayed on the display device under the estimated photographing situation, and the extracted one or more If the image corresponding to the transmission image is included in the stored image, the transmission image is recognized as a valid image, and the transmission image is included in the one or more extracted stored images. The information processing apparatus according to claim 1, wherein when the corresponding image is not included or when the stored image is not extracted, the transmission image is recognized as an illegal image. Transmitting means for transmitting related information of the display image to the terminal;
When the transmission image is recognized as a valid image by the image recognition means, the transmission of the related information from the transmission means is permitted, and the transmission image is recognized as an illegal image by the image recognition means. The information processing apparatus according to claim 1, further comprising: a transmission permission determination unit that prohibits transmission of the related information from the transmission unit. An information processing method for an information processing apparatus for determining whether a transmission image from a terminal is a legitimate image obtained as a result of direct imaging of the display image of a display device as a subject,
Receiving the transmitted image;
Obtain information about the received transmission image,
Based on the acquired information, guess the estimated shooting situation of the transmission image,
When the display image that is actually displayed on the display device under the estimated photographing condition is an image corresponding to the transmission image, the transmission image is recognized as a valid image and corresponds to the transmission image. An information processing method including a step of recognizing that the transmitted image is an illegal image when the image is not an image. A program that is executed by a computer that controls a server that determines whether or not a transmission image from a terminal is a legitimate image obtained as a result of direct imaging of the display image of a display device as a subject. And
Obtaining information about the transmitted image received by the server;
Based on the acquired information, guess the estimated shooting situation of the transmission image,
When the display image that is actually displayed on the display device under the estimated photographing condition is an image corresponding to the transmission image, the transmission image is recognized as a valid image and corresponds to the transmission image. A program comprising a step of recognizing that the transmitted image is an illegal image if it is not an image.

Images