JP5262929B2 - Image processing program and image processing apparatus - Google Patents

Description

  The present invention relates to an advertisement information providing system for transmitting appropriate information according to an individual attribute to the individual optical communication device, and image processing for identifying the owner of the optical communication device based on an image taken by the imaging device More specifically, the present invention relates to a program, an image processing apparatus, and an image processing method. More specifically, when there are a plurality of persons around the optical communication apparatus in optical communication, the optical communication apparatus and each person are photographed from above by an imaging apparatus. Then, the photographed image is analyzed, and the optical communication device owner candidate is selected based on the positional relationship between the optical communication device and each person with respect to the reference position of the communication destination device installed at a predetermined position. The present invention relates to an image processing program, an image processing apparatus, and an image processing method.

  2. Description of the Related Art There is known an optical communication technology that realizes information transmission using light emission patterns such as LEDs (Light Emitting Diodes) (including light emission patterns not only visible light but also invisible light). Since such optical communication does not require a transmission medium such as an optical fiber, it can be easily used in various places.

  As a receiving device that receives optical communication, an imaging device that can capture an image can be used in addition to a light receiving sensor that can detect only blinking of light. When the imaging apparatus is used as a receiving device for optical communication, information transmitted by optical communication can be decoded while the face and figure of a sender transmitting information by optical communication are photographed by the imaging apparatus.

  As an application of the technique for performing optical communication while photographing the sender's face and the like as described above, a technique for displaying information about the sender near the light emission position used for optical communication has been proposed. In this proposal, an image taken from the front is analyzed, and a person shown in the vicinity of the light emission position is identified as a sender.

JP 2008-199569 A

  By applying the above-described technology, it is conceivable to provide advertisement information or the like according to the attribute of the sender of optical communication. In that case, it is necessary to specify the sender by analyzing the video, and further specify the attribute of the sender.

  However, since the above-mentioned conventional technology makes a determination from the image from the front, if there are multiple persons in the image, who is the sender in the image depending on the positional relationship of those persons It was sometimes difficult to identify correctly. For example, when a plurality of persons are positioned one behind the other when viewed from the imaging device, the person closest to the light emission position may not necessarily be the sender, but it is difficult to grasp this. It was.

  The disclosed technology has been made in view of the above, and when there are a plurality of persons around the optical communication apparatus in optical communication, the positional relationship between the optical communication apparatus, a predetermined reference position, and each person is determined. It is an object of the present invention to provide an image processing program, an image processing device, and an image processing method that can identify the owner of the device.

  In one aspect, an image processing apparatus disclosed in the present application analyzes an image captured by an imaging apparatus in an advertisement information providing system that transmits appropriate information according to an individual attribute to the individual optical communication apparatus. An image processing apparatus for identifying a holder of an optical communication apparatus, wherein the optical communication apparatus analyzes the image captured by the imaging apparatus that captures a predetermined space in which the optical communication apparatus and the owner exist. From the optical signal transmitted by the device, a detection unit that detects a plurality of human regions that can be candidates for the holder of the optical communication device, and the plurality of human regions detected in the image by the detection unit, A person area specifying unit for specifying a person area of a holder of the optical communication device based on a positional relationship between a light emission position of the optical signal in the image, a predetermined reference position determined in advance, and each person area;

  According to one aspect of the image processing program, the image processing apparatus, and the image processing method disclosed in the present application, when there are a plurality of persons around the optical communication apparatus, the positions of the optical communication apparatus and the communication destination apparatus, and each person Based on the positional relationship, it is possible to select an owner candidate for the optical communication device.

FIG. 1 is a block diagram illustrating the configuration of the advertisement management system according to the present embodiment. FIG. 2 is a sequence diagram illustrating the operation of the advertisement management system according to the present embodiment. FIG. 3 is a block diagram illustrating a configuration of the mobile terminal device. FIG. 4 is a diagram illustrating an example of a light emission pattern of a signal transmitted by optical communication. FIG. 5 is a block diagram illustrating the configuration of the advertisement management apparatus according to the present embodiment. FIG. 6 is a diagram illustrating an example of an advertisement display schedule. FIG. 7 is a diagram illustrating an example of distribution information selection conditions. FIG. 8 is a diagram illustrating an example of viewer attribute data. FIG. 9 is a flowchart illustrating a processing procedure of optical communication decoding processing. FIG. 10 is a diagram illustrating an example of image data captured by the imaging device on the ceiling side. FIG. 11 is a diagram illustrating an example of image data captured by a wall-side imaging device. FIG. 12 is a diagram illustrating an example of image data captured by another imaging device on the wall side. FIG. 13 is a flowchart showing the processing procedure of the face area specifying process. FIG. 14 is a diagram illustrating an example of the light emission position information. FIG. 15 is a diagram illustrating an example of head region information. FIG. 16 is a diagram illustrating an example of face area information. FIG. 17 is a diagram illustrating an example of the association between the imaging device and the head region. FIG. 18 is a flowchart showing the processing procedure of the viewer tracking process. FIG. 19 is a functional block diagram illustrating a computer that executes an image processing program.

  Embodiments of an image processing program, an image processing apparatus, and an image processing method disclosed in the present application will be described below in detail with reference to the drawings. In the following embodiments, an example in which the image processing program, the image processing apparatus, and the image processing method disclosed in the present application are applied to an advertisement management system is shown. However, the image processing program, the image processing apparatus, and the image processing method disclosed in the present application are described. Is not limited by this embodiment.

  First, an advertisement management system (advertisement information providing system) 1 according to the present embodiment will be described. The advertisement management system 1 displays an advertisement for a person on the elevator. The advertisement management system 1 provides more detailed information regarding the advertisement to a person who is interested in the advertisement, and acquires and records attribute information of the person. The attribute information acquired by the advertisement management system 1 is used as important material for marketing or the like.

  FIG. 1 is a block diagram showing the configuration of the advertisement management system 1. As shown in FIG. 1, the advertisement management system 1 includes an advertisement management device 10, imaging devices 20a to 20c, and display devices 30a to 30b.

  The imaging devices 20a to 20c shoot a space in the elevator and generate moving image data (hereinafter referred to as “image data”) including a plurality of still images (frames). Specifically, the imaging device 20a is installed above the entrance of the elevator, and images the space in the elevator from the viewpoint of viewing the back side from the entrance side of the elevator. The imaging device 20b is installed above the back of the elevator, and images the space in the elevator from the viewpoint of viewing the entrance side from the back of the elevator.

  The imaging device 20c is set downward on the ceiling of the elevator, and images the space in the elevator from the viewpoint of looking down inside the elevator. The imaging devices 20a to 20c may also serve other roles such as a security camera for crime prevention.

  Display device 30a-30b displays an advertisement etc. with respect to the person on the elevator. The display device 30a is installed above the entrance of the elevator, and the display device 30b is installed above the back of the elevator. In the following description, the imaging devices 20a to 20c may be collectively referred to as the imaging device 20, and the display devices 30a to 30b may be collectively referred to as the display device 30.

  The advertisement management apparatus 10 is an apparatus that executes various processes such as displaying advertisements and acquiring attribute information of persons who are interested in the advertisement, and includes a function as an image processing apparatus disclosed in the present application. The advertisement management device 10 is connected to the imaging devices 20a to 20c and the display devices 30a to 30b via a network 50 such as a LAN (Local Area Network).

  The advertisement management apparatus 10 is also connected to a wireless base station 70 of a wireless communication company via the Internet 60. For this reason, the advertisement management apparatus 10 can communicate with the mobile terminal device 40 possessed by the person P riding on the elevator via the Internet 60 and the wireless base station 70.

  The mobile terminal device 40 is a portable information processing device, for example, a mobile phone terminal. The mobile terminal device 40 has a function as an optical communication device that transmits information by optical communication in addition to a function as a wireless communication device that performs wireless communication with the wireless base station 70. In the advertisement management system 1, the imaging device 20a or the imaging device 20b receives a signal transmitted from the mobile terminal device 40 through optical communication, and the advertisement management device 10 decodes the received signal.

  Next, the operation of the advertisement management system 1 shown in FIG. 1 will be described. FIG. 2 is a sequence diagram showing the operation of the advertisement management system 1. As shown in FIG. 2, the advertisement management apparatus 10 transmits advertisement content to the display apparatus 30 (step S01), and the display apparatus 30 displays the transmitted advertisement content (step S02). The advertising content displayed on the display device 30 includes a message such as “If you want to know more about the products introduced in this advertisement, please send an e-mail address to the horizontal imaging device”.

  When the person P in the elevator is interested in the product introduced in the advertisement, the person P holds the portable terminal device 40 toward the imaging device 20 and transmits the mail address assigned to the portable terminal device 40 by optical communication. A predetermined operation is performed on the mobile terminal device 40. As a result, a mail address is transmitted from the mobile terminal device 40 to the imaging device 20 by optical communication (step S03).

  The imaging device 20 sends image data obtained by photographing the space in the elevator to the advertisement management device 10 (step S04). The advertisement management device 10 executes the decoding process by analyzing the light emission pattern of the light emission points while referring to the image data taken by the imaging device 20 in time series, and acquires the transmitted mail address (step S05). .

  Subsequently, the advertisement management apparatus 10 executes a face area specifying process described later, and acquires the face area of the sender who performed the optical communication from the image data captured during the optical communication (step S06). The face area is an area of the face portion of a person. Then, the advertisement management apparatus 10 analyzes the acquired face area and acquires attribute information such as the sender's age and gender (step S07). The attribute information acquired here is recorded in a predetermined storage unit in association with the transmitted mail address.

  Subsequently, the advertisement management apparatus 10 acquires information corresponding to the attribute information from the distribution information prepared in advance for each advertising content (step S08), and the mail address from which the acquired distribution information is acquired in step S05. The e-mail is transmitted to the address (step S09). The transmitted electronic mail is transmitted to the mobile terminal device 40 via the Internet 60 and the wireless base station 70.

  In this way, by sending an email address to a person interested in the advertisement by optical communication, detailed information about the advertisement is individually transmitted to a person in a public place such as an elevator, and the advertisement is sent. The effect can be improved. Also, by extracting the sender's face area from the image data taken during optical communication and analyzing the sender's age, etc., information corresponding to the age etc. is sent to the person interested in the advertisement And the effect of the advertisement can be further improved.

  Further, the advertisement management apparatus 10 tracks the movement of the face area acquired in step S06. When the face image moves outside the image data, the advertisement management apparatus 10 determines that the sender has exited the elevator and determines the floor number at that time. Obtain (step S10). The number of floors that get off the elevator is a clue to know where the sender is working and the store where you are shopping. Then, the advertisement management apparatus 10 records the acquired floor number as a part of the attribute information in association with the transmitted mail address in a predetermined storage unit (step S11).

  Thus, by recording the attribute information of a person who is interested in the advertisement, it is possible to obtain information useful for evaluating the effectiveness of the advertisement and marketing.

  Next, the configuration of the mobile terminal device 40 shown in FIG. 1 will be described. FIG. 3 is a block diagram illustrating a configuration of the mobile terminal device 40. As shown in FIG. 3, the mobile terminal device 40 includes a user interface unit 401, a mail address transmission unit 402, an encoding unit 403, a light emission control unit 404, a light emission unit 405, a wireless communication unit 406, a mail transmission / reception unit 407, a WEB browser. 408.

  The user interface unit 401 includes an input device such as a button for accepting a user operation, and a display device such as a liquid crystal panel for displaying characters and images. Note that the user interface unit 401 may include a device in charge of both input and display, such as a touch panel.

  When a predetermined operation is performed in the user interface unit 401, the mail address transmission unit 402 outputs a mail address previously assigned to the portable terminal device 40 to the encoding unit 403, and the mail address is optically communicated. Instruct to send by. The encoding unit 403 executes an encoding process for transmitting the instructed mail address by optical communication.

  The light emission control unit 404 causes the light emitting unit 405 to emit light according to the light emission pattern obtained by encoding the mail address. The light emitting unit 405 includes a light emitting device such as an LED, and turns on the light emitting device or stops the lighting according to the control of the light emission control unit 404. Note that an incoming call notification lamp, a liquid crystal panel of the user interface unit 401, and the like can also be used as a light emitting device for optical communication.

  The wireless communication unit 406 realizes wireless communication with the wireless base station 70. The mail transmission / reception unit 407 performs transmission / reception of electronic mail through wireless communication realized by the wireless communication unit 406, and displays and edits electronic mail according to operations accepted by the user interface unit 401. The WEB browser 408 acquires a WEB page through wireless communication in accordance with an operation accepted by the user interface unit 401 and displays the WEB page on the user interface unit 401.

  Here, an example of the light emission pattern of the signal transmitted by the optical communication of the portable terminal device 40 is shown in FIG. As shown in FIG. 4, when transmitting a mail address by optical communication, a transmission preparation signal is first transmitted. The transmission preparation signal is a signal indicating the start of optical communication, and includes a light emission pattern that repeats blinking in a short cycle.

  A transmission start signal is transmitted following the transmission preparation signal. The transmission start signal is a signal indicating the start of transmission of information to be transmitted, and includes, for example, a light emission pattern in which light emission continues for a predetermined time.

  Following the transmission start signal, a light emission pattern corresponding to the encoded mail address is transmitted. In the example shown in FIG. 4, a combination of the light emission time and the non-light emission time is defined in advance for each character code, and a light emission pattern in which a mail address is encoded according to this definition is transmitted. Note that not only the combination of the light emission time and the non-light emission time, but also the light emission pattern corresponding to each character code may be defined by combining the intensity and color of the emitted light. Further, the issuance pattern may be defined in units of bits, such as 1 if light emission continues for a predetermined period and 0 if light emission does not occur for a predetermined period.

  When transmission of the encoded mail address is completed, a transmission end signal is transmitted. The transmission end signal is a signal indicating the end of optical communication, and includes, for example, a light emission pattern in which light emission continues for a predetermined time.

  Next, the configuration of the advertisement management apparatus 10 shown in FIG. 1 will be described. FIG. 5 is a block diagram illustrating a configuration of the advertisement management apparatus 10. As illustrated in FIG. 5, the advertisement management apparatus 10 includes a network interface unit 110, a storage unit 120, and a control unit 130. The network interface unit 110 realizes communication via the Internet 60 or the like.

  The storage unit 120 is a storage device that stores various types of information. The storage unit 120 stores advertisement data 121, an advertisement display schedule 122, image data 123, distribution information 124, distribution information selection conditions 125, and viewer attribute data 126. Remember.

  The advertisement data 121 is data including advertisement content displayed on the display device 30. As described above, some advertisement contents included in the advertisement data 121 require that a mail address be transmitted by optical communication.

  The advertisement display schedule 122 is data in which a definition indicating which advertisement should be displayed when is stored. An example of the advertisement display schedule 122 is shown in FIG. As shown in FIG. 6, the advertisement display schedule 122 has items such as display time, advertisement ID, and advertisement data name. The display time is the time when the advertisement display starts and ends. The advertisement ID is an identification number for identifying the advertisement. The advertisement data name is the name of the data file in which the advertisement content is stored.

  The first line of the advertisement display schedule 122 shown in FIG. 6 starts displaying the advertisement identified by the identification number “CM001” at “9:00: 00 (9: 00: 0)” “9:00:30 (9: 0: 30)” indicates that the process should be completed. The record on the first line indicates that the content of the advertisement identified by the identification number “CM001” is stored in the data file named “CM001.mpeg”.

  The image data 123 is image data captured by the imaging device 20. The distribution information 124 is information that is individually distributed to a person who is interested in the advertisement.

  The distribution information selection condition 125 is data in which a condition indicating which distribution information should be selected and transmitted when a mail address is transmitted by optical communication is stored. An example of the distribution information selection condition 125 is shown in FIG. As illustrated in FIG. 7, the distribution information selection condition 125 includes items such as an advertisement ID, age, sex, and distribution information name. The advertisement ID is an identification number for identifying the advertisement. The age and gender are the age and gender of the sender of the email address. The distribution information name is the name of distribution information to be transmitted.

  The first to third lines of the distribution information selection condition 125 shown in FIG. 7 select and transmit which distribution information when an e-mail address is transmitted while an advertisement identified by the identification number “CM001” is displayed. This is a condition indicating what should be done. Specifically, the first line indicates that if the sender's age is in the range of “10 to 29” and the gender is “female”, distribution information named “CM001-F1.txt” should be transmitted. ing. The second line indicates that distribution information named “CM001-F2.txt” should be transmitted when the age of the sender is outside the above range and the gender is “female”. The third line indicates that distribution information with the name “CM001-M.txt” should be transmitted when the gender is “male” regardless of the age of the sender.

  The viewer attribute data 126 is data in which attribute information acquired for the sender of the mail address is stored. FIG. 8 shows an example of the viewer attribute data 126. As shown in FIG. 8, the viewer attribute data 126 includes items such as an email address, age, sex, floor number, advertisement ID, light emission time, image ID, and face area.

  The mail address is a mail address transmitted by optical communication. Age and gender are the results of analysis of the age and gender of the sender of the email address. The number of floors is the number of floors where the sender of the mail address has exited the elevator. The advertisement ID is an identification number for identifying the advertisement displayed when the e-mail address was transmitted.

  The light emission time is the time when a transmission preparation signal for transmitting a mail address by optical communication is detected. The image ID is an identification number for identifying image data used to analyze the age and sex of the sender of the mail address. The face area is the sender's face area specified in the image data identified by the image ID, and indicates the upper left coordinates, width, and height of the face area.

  The first line of the viewer attribute data 126 shown in FIG. 8 shows that the sender of the email address “suzuki@example1.com” is analyzed as “19F” for age “40” and gender “male”. It shows that I got off the elevator. The first line indicates that the advertisement displayed when the e-mail address is transmitted is an advertisement identified by the identification number “CM001”.

  The first line indicates that the image data used for analyzing the age and sex of the sender is an image identified by an identification number “N1-081201090021-11”. In the first line, X = “302” and Y = “436” are the upper left coordinates in the image, and an area having a width of “183” and a height of “196” is specified as the face area of the sender. It has been shown.

  The control unit 130 is a control unit that totally controls the advertisement management apparatus 10, and includes an advertisement display control unit 131, an image acquisition unit 132, a decoding unit 133, a face region specifying unit 134, a face region detecting unit 134a, A head region detection unit 134b, a head region selection unit 134c, a position conversion unit 134d, a tracking unit 135, a floor acquisition unit 136, an attribute analysis unit 137, an information distribution unit 138, and a mail transmission unit 139; Have

  The advertisement display control unit 131 displays the advertisement data 121 on the display device 30 according to the schedule defined in the advertisement display schedule 122. The image acquisition unit 132 acquires image data from the imaging device 20 and stores the acquired image data in the storage unit 120 as the image data 123.

  The decryption unit 133 decrypts the light emission pattern of the light emission points included in the image data acquired by the image acquisition unit 132 and acquires the mail address. In order to acquire the mail address, the decoding unit 133 first detects a transmission preparation signal. The detection of the transmission preparation signal is performed by referring to the image data 123, that is, by calculating the sum of the difference between each frame captured in a certain past period for a frame (still image) at a certain point in time included in the image data. Based on the results.

  As shown in FIG. 4, the transmission preparation signal includes a light emission pattern that repeats blinking in a short period. Therefore, in the pixel corresponding to the light emission point of the transmission preparation signal, the sum of the inter-frame differences in the past certain period is the other pixels. It becomes a big value compared with. For this reason, it is possible to determine whether or not the transmission preparation signal is transmitted at a position corresponding to the pixel by determining the threshold value of the sum of the inter-frame differences in the past certain period.

  D (x, y, t), which is the sum of inter-frame differences up to N frames before the pixel at the coordinate (x, y) at time t, can be calculated by the following equation (1).

  Here, P (x, y, t) is an average value of the pixel values of the RGB pixels of the pixel at the coordinate (x, y) at time t. Note that P (x, y, t) may be the maximum pixel value of each RGB pixel of the pixel at the coordinate (x, y) at time t.

  When the decoding unit 133 detects that the transmission preparation signal is transmitted and the light emission point in this way, the decoding unit 133 monitors the image acquired by the image acquisition unit 132 and detects the transmission start signal at the light emission point. Wait for When the transmission start signal is detected, the decoding unit 133 regards the light emission pattern of the light emission points after the end of the transmission start signal as an encoded mail address, and detects the transmission end signal at the light emission point. Decrypt until.

  The face area specifying unit 134 executes the image processing method according to the present embodiment, and specifies the face area of the sender of the transmission start signal detected by the decoding unit 133. The face area of the sender is specified by the face area specifying unit 134 based on the processing results of the face area detecting unit 134a, the head region detecting unit 134b, the head region selecting unit 134c, and the position converting unit 134d.

  Note that the target image data of the sender's face area is preferably image data taken when the transmission preparation signal is detected by the decoding unit 133, but optical communication is performed after detection of the transmission preparation signal. It may be image data taken at any time until the end.

  The face area detection unit 134a detects the face area of each person included in the image data from the image data to be specified of the sender's face area. Detection of a face area from image data can be realized by using a known face recognition technique.

  The head region detection unit 134b detects the head region (person) of each person included in the image data from the image data captured by the imaging device 20c when the target image data of the sender's face region is captured. Area). The head region is a region of a person's head part. Here, the detected head region may be a portion in which a portion estimated to be a human head is captured at any angle, and does not need to include a face. The detection of the head region from the image data is realized, for example, by applying a known face recognition technique.

  The head region selection unit 134c has the following two relations regarding the positional relationship between the transmitter, the light emitting point, and the imaging device that is the communication destination device by optical communication from the head region detected by the head region detection unit 134b. Those satisfying the criteria are selected (identified) for each imaging device 20 used for optical communication reception, assuming that there is a possibility of being the head region of the sender of the mail address.

  The first criterion for selection by the head region selection unit 134c is that the distance from the light emitting point of optical communication is smaller than the threshold value. The threshold value used in the first criterion is determined based on the maximum distance that can be taken by the sender and the portable terminal device 40 when optical communication is performed. For example, the length of a general human arm is used as the maximum distance that the sender and the portable terminal device 40 can take when optical communication is performed.

  A second criterion for selection by the head region selection unit 134c is that the head region selection unit 134c is located far from the light emitting point of optical communication when viewed from the imaging device 20 that is a communication destination device used for reception of optical communication. . When the sender performs optical communication, the face should be directed in the direction of the destination imaging device 20, the portable terminal device 40 should be held in front of the face, and the light emitting point should be located behind the head The reason for the second standard is that it is not considered to be.

  The position converting unit 134d converts the position of each head region selected by the head region selecting unit 134c into a position in the image data captured by the imaging devices 20a and 20b. The position conversion method is such that at least one of the X-axis direction and the Y-axis direction of the position of an object that is commonly present in the images before and after the conversion is based on the positional relationship of the imaging device 20, the resolution of the image, and the like. Is determined in advance.

  The face area specifying unit 134 compares the position of the head area converted by the position converting unit 134d with the position of the face area detected by the face area detecting unit 134a, and the difference in position is a predetermined error. If there is a combination within the range, it is determined that the head region and the face region of the combination exist at the same position, and the face region is specified as the sender's face region. When there are a plurality of combinations of error ranges, the combination closest to the light emission position is determined as a combination including the sender's face area. Here, when finding the combination closest to the light emitting position, either the distance between the face area and the light emitting point, or the distance between the head area and the light emitting point may be evaluated heavily, or both may be evaluated equally. May be.

  In this way, the head region estimated to be that of the sender is acquired from an image captured from a direction different from that of the imaging device used for reception of optical communication, and the sender's location is determined based on the position of the head region. By specifying the face area, the sender's face area can be detected with high accuracy.

  The tracking unit 135 tracks the movement of at least one of the face region and the head region identified as those of the sender by the face region identifying unit 134. The tracking by the tracking unit 135 continues until it is determined that the tracking target has left the space in the elevator. Tracking of the head region and the like can be realized using a known tracking technique.

  The floor number acquisition unit 136 acquires the current floor number of the elevator. The number of floors can be acquired, for example, from an elevator operation monitoring device.

  The attribute analysis unit 137 analyzes the face area determined to be the sender by the face area specifying unit 134 to acquire attribute information such as age and gender, and the viewer along with the mail address decoded by the decoding unit 133 Record in attribute data 126. The process of acquiring attribute information such as age and sex from the face area in the image data can be realized using a known image analysis technique.

  Further, when the tracking unit 135 determines that the tracking target has left the space in the elevator, the attribute analysis unit 137 acquires the floor number of the elevator at that time from the floor acquisition unit 136. The attribute analysis unit 137 records the acquired rank in the viewer attribute data 126 in association with the mail address decrypted by the decryption unit 133.

  The information distribution unit 138 acquires, from the distribution information 124, corresponding to the advertisement displayed when the transmission preparation signal is detected by the decoding unit 133 and corresponding to the attribute information acquired by the attribute analysis unit 137. . Here, for the advertisement ID of the advertisement displayed when the transmission preparation signal is detected by the decryption unit 133, the line in which the display time includes the time when the transmission preparation signal is detected by the decryption unit 133 is displayed in the advertisement display schedule 122. Can be obtained by searching from.

  Then, the information distribution unit 138 instructs the mail transmission unit 139 to transmit the acquired information by mail. The mail transmission unit 139 transmits the information instructed to be transmitted to the mail address decrypted by the decryption unit 133 by mail.

  Next, the operation of the advertisement management apparatus 10 will be described. FIG. 9 is a flowchart illustrating a processing procedure of optical communication decoding processing. As shown in FIG. 9, the decoding unit 133 acquires image data from the image acquisition unit 132 (step S101). And the decoding part 133 acquires the image data of the past fixed period image | photographed with the same imaging device 20 from the image data 123, and calculates the sum of the difference between frames for every pixel (step S102).

  Here, when there is no pixel whose sum of interframe differences exceeds the threshold, that is, when a transmission preparation signal is not transmitted (No at Step S103), the decoding unit 133 restarts the processing procedure from Step S101.

  On the other hand, when there is a pixel whose sum of interframe differences exceeds the threshold (Yes at Step S103), the decoding unit 133 selects a pixel having the maximum sum of interframe differences as a light emitting point (Step S104). And the decoding part 133 acquires the light emission pattern of the past fixed time of the selected pixel (step S105).

  If the end of the transmission start signal is detected (Yes at Step S106), the decryption unit 133 decrypts the mail address until the transmission end signal is detected (Step S107). On the other hand, when the end of the transmission start signal is not detected (No at Step S106), the decoding unit 133 restarts the processing procedure from Step S105.

  Note that the decoding process is repeatedly performed for each imaging device 20. For this reason, when a transmission preparation signal is detected in a decoding process corresponding to an imaging device 20, the transmission preparation signal may not be detected for a predetermined period in a decoding process corresponding to another imaging device 20. By doing so, it is possible to prevent the transmitted e-mail address from being decoded twice in the decoding process corresponding to each imaging device 20.

  Further, when a transmission preparation signal is detected within a predetermined period in the decoding process corresponding to each imaging device 20, the mail address may be synthesized by combining the light emission patterns obtained in the respective decoding processes. By synthesizing the light emission pattern in this way, even when light is blocked in a certain direction during optical communication, the influence can be reduced.

  Next, the processing procedure of the face area specifying process will be described with reference to the image data examples shown in FIGS.

  FIG. 10 is a diagram illustrating an example of image data captured by the ceiling-side imaging device 20c when the transmission preparation signal is detected. In the image data 81 shown in FIG. 10, the lower side in the figure corresponds to the installation position of the imaging device 20a, and the upper side in the figure corresponds to the installation position of the imaging device 20b. The coordinates of the image data 81 are set so that the value of the X coordinate increases as it goes to the right in the figure, and the value of the Y coordinate increases as it goes down in the figure, with the upper left in the figure as the origin. FIG. 10 shows the heads of the persons P1 to P3 and the light emitting point 81d for optical communication.

  FIG. 11 is a diagram illustrating an example of image data captured by the wall-side imaging device 20a. Image data 82 shown in FIG. 11 is image data taken at the same timing as the image data 81. The coordinates of the image data 82 are set such that the upper left corner in the figure is the origin, the X coordinate value increases toward the right side in the figure, and the Y coordinate value increases toward the lower side in the figure. FIG. 11 shows a person P1 standing rearward, persons P2 and P3 standing frontward, and a light emitting point 82c for optical communication.

  FIG. 12 is a diagram illustrating an example of image data captured by the wall-side imaging device 20b. Image data 83 shown in FIG. 12 is image data taken at the same timing as the image data 81. The coordinates of the image data 83 are set so that the value of the X coordinate increases as it goes to the right in the figure, and the value of the Y coordinate increases as it goes down in the figure, with the upper left in the figure as the origin. In FIG. 12, a person P1 standing facing the front, persons P2 and P3 standing rearward, and a light emitting point 83b of optical communication are shown.

  FIG. 13 is a flowchart showing the processing procedure of the face area specifying process. This processing procedure is executed by the face area specifying unit 134 every time a transmission preparation signal is detected by the decoding unit 133.

  As illustrated in FIG. 13, the face area specifying unit 134 acquires information regarding the light emission position of the detected transmission preparation signal from the decoding unit 133 (step S <b> 201). An example of the acquired light emission position information is shown in FIG. As shown in FIG. 14, coordinates indicating the light emission position are acquired for each imaging device 20. These coordinates correspond to the coordinates of the light emitting points in the image data 81-83.

  Subsequently, the face area specifying unit 134 acquires image data photographed by each imaging device 20 when the transmission preparation signal is detected (step S202). Here, it is assumed that the image data 81 to 83 are acquired.

  Subsequently, the face area specifying unit 134 causes the face area detecting unit 134a to execute processing for extracting the face area from each image data photographed from the wall side (step S203). In step S203, the face area 82a of the person P2 facing the front and the face area 82b of the person P3 facing the front are detected from the image data 82. Further, the face area 83a of the person P1 facing the front is detected from the image data 83. As shown in FIG. 16, the combination of the detected upper left coordinate, width, and height of each face area is assigned with a face area ID, which is a unique identification number, and is managed in association with the imaged imaging device. The

  Subsequently, the face area specifying unit 134 causes the head area detecting unit 134b to execute a process of extracting the head area from the image data photographed from the ceiling side (step S204). In step S204, a head region 81a of the person P1, a head region 81b of the person P2, and a head region 81c of the person P3 are detected from the image data 81. As shown in FIG. 15, the combination of the detected upper left coordinates, width, and height of each head region is managed by assigning a head region ID that is a unique identification number.

  Subsequently, the face area specifying unit 134 performs a process of selecting a head area of a person who may be performing optical communication with the imaging apparatus 20 for each imaging apparatus 20 on the wall side. 134c is executed. Then, the face area specifying unit 134 stores the selected head area in association with the imaging device 20 that is assumed to be the transmission destination of optical communication (step S205).

  In step S205, head regions 81a and 81b are selected from the image data 81 as head regions of a person who may be performing optical communication with the imaging device 20a. Since the distance between the center of these head regions and the light emitting point 81d is smaller than the threshold Th1, the first criterion is satisfied, and the Y coordinate of the center of these head regions is smaller than Y1 which is the Y coordinate of the light emitting point 81d. This is because the second criterion is also satisfied.

  Further, from the image data 81, a head region 81c is selected as the head region of a person who may be performing optical communication with the imaging device 20b. Since the distance between the center of the head region 81c and the light emitting point 81d is smaller than the threshold Th1, the first criterion is satisfied, and the Y coordinate of the center of the head region 81c is larger than Y1 which is the Y coordinate of the light emitting point 81d. This is because the second criterion is also satisfied.

  As shown in FIG. 17, the head region selected in this way is stored in association with a video device assumed to be a transmission destination of optical communication. Specifically, “HEAD01” that is the head region ID of the head region 81a and “HEAD02” that is the head region ID of the head region 81b are associated with the imaging device 20a. The head area ID “HEAD03” of the head area 81c is associated with the imaging device 20b.

  Subsequently, the face area specifying unit 134 acquires one of the selected head areas that has not been acquired (step S206). When an unacquired head region can be acquired (No at Step S207), the face region specifying unit 134 converts the X coordinate of the center of the head region into the coordinates of each image data photographed from the wall side. The position conversion unit 134d is caused to execute the processing to be performed (step S208).

  If the head region 81b is selected in step S206, X1 which is the X coordinate of the center is converted as follows. When X1 is converted to the coordinates of the image data 82, the X-coordinate direction of the image data 82 is the same as that of the image data 81 including the head region 81b. Therefore, the conversion is performed by multiplying the coefficient according to the difference in resolution. Complete. On the other hand, when X1 is converted into the coordinates of the image data 83, the direction of the X coordinate of the image data 83 is opposite to that of the image data 81. Will be multiplied by a coefficient corresponding to.

  Subsequently, the face area specifying unit 134 captures the face area in which the distance between the center coordinates and the converted coordinates is smaller than the threshold by the imaging device stored in association with the head area selected in step S206. A face area detected from the detected image data is selected (step S210). The threshold value used here is for absorbing errors.

  Consider the case where the head region 81a is selected in step S206. The imaging device 20a is stored in step S205 in association with the head region 81a, and the face regions 82a and 82b are detected from the image data 82 taken by the imaging device 20a. However, the person P1 standing backward is located at the position where the X coordinate of the center of the head region 81a is converted into the X coordinate of the image data 82, and the face region is detected in the vicinity of the converted X coordinate. Absent. For this reason, the face region corresponding to the head region 81a is not selected.

  Further, consider the case where the head region 81c is selected in step S206. The imaging device 20b is stored in step S205 in association with the head region 81c, and the face region 83a is detected from the image data 83 taken by the imaging device 20b. However, the person P3 standing backwards is at the position where the X coordinate of the center of the head region 81c is converted to the X coordinate of the image data 83, and the face region is detected in the vicinity of the converted X coordinate. Absent. For this reason, the face region corresponding to the head region 81c is not selected.

  Consider the case where the head region 81b is selected in step S206. The imaging device 20a is stored in step S205 in association with the head region 81b, and the face regions 82a and 82b are detected from the image data 82 taken by the imaging device 20a. Then, there is a person P2 standing facing the front at the position where the X coordinate of the center of the head region 81a is converted into the X coordinate of the image data 82, and the value of the converted X coordinate is the center of the face region 82a. It almost coincides with the X coordinate. For this reason, the face area 82a is selected as the face area corresponding to the head area 81b.

  When the face area can be selected in step S209 (Yes in step S210), the face area specifying unit 134 stores the selected face area as a candidate for the sender area in association with the head area selected in step S206. (Step S211), and the processing procedure is restarted from Step S206. On the other hand, when the face area cannot be selected in step S209 (No in step S210), the face area specifying unit 134 restarts the processing procedure from step S206.

  In this way, the face area specifying unit 134 collates the head area with the face area until there is no unacquired head area. Then, all the head regions selected in step S205 are acquired. If there are no unacquired regions (Yes in step S207), the comparison between the head region and the face region is ended.

  Subsequently, when there are a plurality of combinations of the head area and the face area stored as candidates for the sender's area (Yes in step S212), the face area specifying unit 134 selects the combination having the smallest distance from the light emission position. A candidate is selected (step S213).

  Subsequently, if there is a combination of the head area and the face area stored as the sender area candidate (Yes in step S214), the face area specifying unit 134 transmits the head area and the face area included in the candidate. (Step S215). On the other hand, if there is no combination of the head area and the face area stored as the sender area candidate (No in step S214), the face area specifying unit 134 specifies the sender head area and the face area. It is determined that it cannot be performed (step S216).

  In the example shown in FIGS. 10 to 12, since only the combination of the head region 81b and the face region 82a is selected as a candidate, the head region 81b and the face region 82a are identified as those of the sender. This specific result is reasonable.

  FIG. 18 is a flowchart showing the processing procedure of the viewer tracking process. The viewer tracking process is executed by the tracking unit 135 and the like each time the face region specifying unit 134 specifies the sender's face region and head region.

  As shown in FIG. 18, the tracking unit 135 tracks the movement of the head region identified as that of the sender, that is, the viewer who has transmitted the e-mail address with an interest in the advertisement (step S301). This tracking is performed until it is determined that the head region has gone out of the space in the elevator.

  Then, if it is determined that the head region has gone out of the space in the elevator, for example, the head region has disappeared from the outer peripheral portion of the image data (Yes in step S302), the floor acquisition unit 136 The floor number is acquired (step S303). Then, the attribute analysis unit 137 stores the acquired rank in the viewer attribute data 126 in association with the decrypted mail address (step S304).

  The configuration of the advertisement management system 1 shown in FIG. 1 can be variously changed without departing from the gist. For example, the advertisement management system 1 can be modified not to manage advertisements displayed in an elevator, but to manage advertisements displayed in a train or on the street.

  Further, the advertisement management system 1 uses a total of three imaging devices, one on the ceiling side and two on the wall side. However, if a plurality of imaging devices that shoot the space to be photographed from different angles are installed, the imaging is performed. The number of devices and the installation position are not limited.

  In the above description, an example in which a mail address is transmitted from the mobile terminal device 40 to the imaging device 20 by optical communication has been described as an example. However, a transmission preparation signal is detected from the image 81 taken by the ceiling imaging device 30c. You may be able to.

  Therefore, as described above, it is not always necessary to urge the mobile terminal to send an e-mail address to the imaging device in the advertising content.

  For example, in an advertisement content, even when the user is prompted to send an optical signal toward a predetermined reference position other than the imaging device (for example, the position of a poster attached to an arbitrary wall surface), The owner of the terminal can be identified based on the positional relationship between the light emitting point and the predetermined reference position.

  That is, in this case, the second reference by the head region selection unit 134c is that the head region is located farther from the light emitting point of optical communication when viewed from a predetermined reference position (poster position). What you are doing.

  Furthermore, if the ceiling imaging device 30c also recognizes the characteristics of a person such as clothes and hairstyles, the system configuration without the wall-side imaging device is very important for the owner of the portable terminal device. Information such as advertisements according to the characteristics can be transmitted.

  All or part of the functions of the advertisement management apparatus 10 can be implemented as software, and an equivalent function can be realized by executing the software on a computer. Hereinafter, an example of a computer that executes an image processing program 1071 in which a function related to face area specification of the advertisement management apparatus 10 is implemented as software will be described.

  FIG. 19 is a functional block diagram illustrating a computer 1000 that executes the image processing program 1071. A central processing unit (CPU) 1010 that executes various arithmetic processes of the computer 1000, an input device 1020 that receives input of data from a user, a monitor 1030 that displays various information, and a medium reading that reads a program from a recording medium A device 1040, a network interface device 1050 that exchanges data with another computer via a network, a RAM (Random Access Memory) 1060 that temporarily stores various information, and a hard disk device 1070 are connected via a bus 1080. It is connected.

  The hard disk device 1070 stores an image processing program 1071 having the same function as the face area specifying unit 134 shown in FIG. 1 and image processing data 1072 corresponding to the image data 123 shown in FIG. The Note that the image processing data 1072 can be appropriately distributed and stored in another computer connected via a network.

  Then, the CPU 1010 reads the image processing program 1071 from the hard disk device 1070 and develops it in the RAM 1060, whereby the image processing program 1071 functions as the image processing process 1061. Then, the image processing process 1061 expands information read from the image processing data 1072 and the like in an area allocated to itself on the RAM 1060 as appropriate, and executes various data processing based on the expanded data and the like.

  The image processing program 1071 is not necessarily stored in the hard disk device 1070. The computer 1000 may read and execute the program stored in a storage medium such as a CD-ROM. . The computer 1000 stores the program in another computer (or server) connected to the computer 1000 via a public line, the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), or the like. You may make it read and run a program from these.

  As described above, in the present embodiment, the face area candidates are narrowed down based on information obtained by photographing from different directions, and thus, from among images photographed by an imaging device installed on the wall side. The face area of the sender of optical communication can be specified with high accuracy.

  The following supplementary notes are further disclosed with respect to the embodiments including the above examples.

(Appendix 1) In an advertisement information providing system for transmitting appropriate information according to an individual attribute to the individual optical communication device, an image identifying the owner of the optical communication device by analyzing the image captured by the imaging device A processing program,
Analyzing the image captured by the imaging device that captures a predetermined space where the optical communication device and its owner are present, the optical signal transmitted by the optical communication device, and the owner of the optical communication device A detection procedure for detecting a plurality of human regions that can be candidates;
Based on the positional relationship between each person area and a light emitting position of the optical signal in the image, a predetermined reference position determined in advance, among the plurality of person areas detected in the image in the detection procedure An image processing program for causing a computer to execute a person area specifying procedure for specifying a person area of a holder of an optical communication apparatus.

(Additional remark 2) The person area specifying procedure includes a person area located farther from the light emission position as viewed from the predetermined position and having a distance from the light emission position smaller than a threshold value. The image processing program according to appendix 1, wherein the image processing program is specified as a person area of a holder.

(Additional remark 3) The face area detection procedure which detects the face area of the person included in the 2nd picture photoed by the 2nd imaging device which picturizes the predetermined space from the direction different from the imaging device,
Of the face areas detected in the second image in the face area detection procedure, those determined to be present at the same position as the person area specified in the image in the person area selection procedure are The image processing program according to appendix 1 or 2, further causing a computer to execute a face area specifying procedure for specifying a face area of a person who has transmitted an optical signal using a communication device.

(Additional remark 4) Analyzing the face area | region identified by the said face area | region identification procedure, and making a computer perform further the attribute analysis procedure which acquires the attribute information of the person who transmitted the optical signal using the said optical communication apparatus. The image processing program according to appendix 3, which is characterized.

(Supplementary Note 5) Based on the identification information of the optical communication device included in the information decoded by the decoding procedure, information corresponding to the attribute information analyzed by the attribute analysis procedure is transmitted to the optical communication device The image processing program according to appendix 4, further causing a computer to execute a transmission procedure.

(Appendix 6) In an advertisement information providing system for transmitting appropriate information according to an individual attribute to the individual optical communication device, an image identifying the owner of the optical communication device by analyzing the image captured by the imaging device A processing device comprising:
Analyzing the image captured by the imaging device that captures a predetermined space where the optical communication device and its owner are present, the optical signal transmitted by the optical communication device, and the owner of the optical communication device A detection unit for detecting a plurality of human regions that can be candidates;
Based on the positional relationship between each person area and a light emitting position of the optical signal in the image, a predetermined reference position determined in advance, among the plurality of person areas detected in the image by the detection unit. An image processing apparatus comprising: a person area specifying unit that specifies a person area of a holder of the optical communication apparatus.

(Additional remark 7) The said person area specific | specification part is located far from the said light emission position seeing from the said predetermined position, and the person area whose distance with the said light emission position is smaller than a threshold value is the optical communication apparatus. The image processing apparatus according to appendix 6, wherein the image processing apparatus is specified as a person area of the owner.

(Supplementary Note 8) A face area detection unit that detects a face area of a person included in a second image captured by a second imaging apparatus that captures the predetermined space from a direction different from the imaging apparatus;
Of the face areas detected in the second image by the face area detection unit, those determined to be present at the same position as the person area specified in the image by the person area selection unit The image processing apparatus according to appendix 6 or 7, further comprising: a face area specifying unit that specifies the face area of the person who transmitted the optical signal using the communication apparatus.

(Additional remark 9) It further provided with the attribute analysis part which analyzes the face area specified by the said face area specific | specification part, and acquires the attribute information of the person who transmitted the optical signal using the said optical communication apparatus, It is characterized by the above-mentioned. The image processing apparatus according to appendix 8.

(Supplementary note 10) Based on the identification information of the optical communication device included in the information decoded by the decoding unit, information corresponding to the attribute information analyzed by the attribute analysis unit is transmitted to the optical communication device The image processing apparatus according to appendix 9, further comprising: a transmitting unit that performs the processing.

(Additional remark 11) Image which identifies the holder of an optical communication apparatus by analyzing the image image | photographed with the imaging device in the advertisement information provision system which transmits appropriate information according to an individual attribute to the personal optical communication apparatus A processing method,
Analyzing the image captured by the imaging device that captures a predetermined space where the optical communication device and its owner are present, the optical signal transmitted by the optical communication device, and the owner of the optical communication device A detection step of detecting a plurality of human regions that can be candidates;
Among the plurality of person areas detected in the image in the detection step, the light signal emission position in the image, a predetermined reference position determined in advance, and the positional relationship between each person area, An image processing method comprising: a person area specifying step of specifying a person area of a holder of the optical communication apparatus.

(Supplementary Note 12) In the person area specifying step, a person area that is located farther from the light emitting position than the predetermined light position and whose distance from the light emitting position is smaller than a threshold is determined by the optical communication apparatus. The image processing method according to appendix 11, wherein the image processing method is specified as a person area of a holder.

(Additional remark 13) The face area detection process which detects the face area of the person included in the 2nd picture photoed by the 2nd image pick-up device which picturizes the predetermined space from the direction different from the image pick-up device,
Of the face areas detected in the second image in the face area detection step, those determined to be present at the same position as the person area specified in the image in the person area selection step are The image processing method according to claim 11 or 12, further comprising: a face area specifying step of specifying as a face area of a person who transmitted an optical signal using a communication device.

(Supplementary note 14) The method further includes an attribute analysis step of analyzing the face region specified by the face region specifying step and acquiring attribute information of a person who has transmitted an optical signal using the optical communication device. The image processing method according to appendix 13.

(Supplementary Note 15) Based on the identification information of the optical communication device included in the information decoded by the decoding step, information corresponding to the attribute information analyzed by the attribute analysis step is transmitted to the optical communication device 15. The image processing method according to appendix 14, further comprising a transmitting step.

DESCRIPTION OF SYMBOLS 1 Advertisement management system 10 Advertisement management apparatus 110 Network interface part 120 Storage part 121 Advertisement data 122 Advertisement display schedule 123 Image data 124 Delivery information 125 Delivery information selection condition 126 Viewer attribute data 130 Control part 131 Advertisement display control part 132 Image acquisition part 133 Decoding unit 134 Face region specifying unit 134a Face region detection unit 134b Head region detection unit 134c Head region selection unit 134d Position conversion unit 135 Tracking unit 136 Rank acquisition unit 137 Attribute analysis unit 138 Information distribution unit 139 Mail transmission unit 20a 20c imaging device 30a-30b display device 40 mobile terminal device 401 user interface unit 402 mail address transmission unit 403 encoding unit 404 light emission control unit 405 light emission unit 406 wireless communication unit 407 mail Transmission / reception unit 408 WEB browser 50 Network 60 Internet 70 Wireless base station 1000 Computer 1010 CPU
1020 Input device 1030 Monitor 1040 Medium reader 1050 Network interface device 1060 RAM
1061 Image processing process 1070 Hard disk device 1071 Image processing program 1072 Image processing data 1080 Bus

Claims (5)

In an advertisement information providing system that transmits appropriate information according to an individual attribute to an optical communication device of the individual, an image processing program that analyzes an image captured by the imaging device and identifies the owner of the optical communication device. And
Analyzing the image captured by the imaging device that captures a predetermined space where the optical communication device and its owner are present, the optical signal transmitted by the optical communication device, and the owner of the optical communication device A detection procedure for detecting a plurality of human regions that can be candidates;
The plurality of person regions detected in the image in the detection procedure are located farther from the light emission position of the optical signal in the image when viewed from a predetermined reference position, and the light emission An image processing program for causing a computer to execute a person area specifying procedure for specifying a person area whose distance from a position is smaller than a threshold as a person area of a holder of the optical communication apparatus. A face area detection procedure for detecting a face area of a person included in a second image captured by a second image capturing apparatus that captures the predetermined space from a direction different from the image capturing apparatus;
Of the face areas detected in the second image in the face area detection procedure, those determined to be present at the same position as the person area specified in the image in the person area selection procedure are the image processing program of the serial placement in claim 1, characterized in that to execute the further computer and the face area specifying step of specifying a human face region that has transmitted the optical signal by using a communication device. The computer further executes an attribute analysis procedure for analyzing the face area specified by the face area specifying procedure and acquiring attribute information of a person who has transmitted an optical signal using the optical communication device. Item 3. The image processing program according to Item 2 . A transmission procedure for transmitting information corresponding to the attribute information analyzed by the attribute analysis procedure to the optical communication device based on the identification information of the optical communication device included in the information decoded by the decoding procedure; The image processing program according to claim 3 , further causing a computer to execute the image processing program. In an advertisement information providing system that transmits appropriate information according to an individual attribute to the optical communication device of the individual, the image processing device analyzes an image captured by the imaging device and identifies the owner of the optical communication device. And
Analyzing the image captured by the imaging device that captures a predetermined space where the optical communication device and its owner are present, the optical signal transmitted by the optical communication device, and the owner of the optical communication device A detection unit for detecting a plurality of human regions that can be candidates;
The plurality of person regions detected in the image by the detection unit are located farther from the light emission position of the optical signal in the image when viewed from a predetermined reference position, and the light emission An image processing apparatus comprising: a person area specifying unit that specifies a person area whose distance from the position is smaller than a threshold as a person area of a holder of the optical communication apparatus.

Images