JP5448468B2 - Transaction monitoring device - Google Patents

Description

  The present invention relates to a transaction monitoring apparatus, and more particularly to a transaction monitoring apparatus that determines whether a user is taking a specific action from an image obtained by photographing a user of a transaction terminal of an automatic transaction system.

Conventionally, in money transactions between multiple users, remittance processing such as transfer procedures using a transaction terminal installed at a financial institution or convenience store, for example, an automated teller machine (ATM) is performed. Widely used.
However, in recent years, there has been an increase in fraudulent acts that use the remittance processing by this transaction terminal to pretend to be relatives such as family members or relatives, and to remit cash to users of transaction terminals. It has become.
Similarly, disguise as if the money you overpaid is reimbursed, such as reimbursement of medical expenses or taxes, pretending to be a member of a public organization, and let the user skillfully operate the transaction terminal The fraudulent act of transferring money into another person's (frauder's) account before the user himself / herself notices is also increasing rapidly.
In response to such a problem, a system has been proposed that attempts to prevent fraudulent damage by determining the legitimacy of a money recipient when a user performs remittance processing (for example, (See Patent Document 1).

  The automatic transaction system disclosed in Patent Document 1 includes a handset that makes a call between a transfer person who performs remittance processing at a transaction apparatus and a recipient who receives the remittance. And this automatic transaction system collates the voice print data of the recipient acquired through this handset with the voice print data of the relative of the transfer person stored in the customer database in advance, and whether or not the transfer transaction is possible based on the check result. By judging, fraud damage can be prevented in advance.

JP 2007-80123 A

  However, in order to prevent fraud damage, the automatic transaction system disclosed in Patent Document 1 must always perform a voiceprint collation of the recipient during the transfer procedure. For this reason, this automatic transaction system obtains voice print data of all persons who can be considered as the remittance destination for all users who perform remittance processing with the transaction device, and registers them in the customer database in advance. There must be. If the voiceprint data of the person to whom the remittance is not registered in the customer database, the user cannot perform remittance processing for that person's account using this automatic transaction system. There is a risk of hindering transactions.

  Therefore, the present invention provides a transaction monitoring device that prevents fraudulent acts that abuse the remittance processing by a transaction terminal of the system without hindering smooth money transactions between legitimate users of the automatic transaction system. Objective.

  In general, in fraudulent acts in which a relative such as a family or a relative is dressed and demands money transfer, elderly people with weak hearing are often targeted as victims. As a technique for the fraud, the fraudster induces the victim to call the fraudster himself after the victim has gone to the front of the transaction terminal in the guise of the victim. If the victim trusts the fraudster, the victim calls the fraudster using the mobile phone in front of the transaction terminal as directed by the fraudster. Then, the victim operates the transaction terminal according to a specific instruction on the operation method of the transaction terminal through the telephone from the fraudster, and remits money to the fraudster's account without noticing it as a fraud.

  The same is true for fraudulent acts that appear as if there is a refund, and it is often the elderly who are unfamiliar with the operation of transaction terminals that are targeted as victims. As a means of fraud in this case, fraudsters will be charged with overpaid money, such as medical insurance or insurance money, or tax refunds, pretending to be a social insurance office or local government employee or tax officer. The victim is falsely guided to the transaction terminal as if it is returned by operating the terminal, and the fraudulent himself is called after the victim goes to the front of the transaction terminal. If the victim trusts the fraudster, similar to the type of fraud described above, the victim calls the fraudster using the mobile phone in front of the transaction terminal as directed by the fraudster. End up. And when the fraudster gets a call from the victim who goes to the transaction terminal, he / she makes the victim operate the transaction terminal skillfully, such as "Please operate the ATM as I said". In the amount entry screen, ask for a false transfer amount, such as “Enter your PIN here”. Therefore, even if the victim operates the transaction terminal while thinking that money will be refunded, the victim actually sends money to the account of the other person (scammers) without realizing the person himself / herself.

  In the transaction monitoring apparatus according to the present invention, the fraudulent act that misuses the remittance processing of the transaction terminal described above is performed by the victim operating the transaction terminal in front of the transaction terminal according to instructions from the fraudster through the mobile phone. It pays attention to the point.

  This invention for solving this subject provides the transaction monitoring apparatus which monitors the action of the user of the automatic transaction terminal which can perform remittance processing. The transaction monitoring apparatus according to the present invention is a face detection that extracts a face region corresponding to a user's face from image data acquired from a monitoring camera installed so as to photograph the front direction of the user facing the automatic transaction terminal. And a head detection unit that extracts a head region when the user's head is viewed from the top of the image data, and a face region detected by the face detection unit, the face region is set as a reference. When the user's hand is detected from the first search area and the head area is detected by the head detection unit, the user's hand is detected from the second search area set based on the head area. When the hand area detection unit detects the user's hand from each of the hand area detection unit and the image data sequentially acquired during the predetermined period, it is determined that the user is in a call using the mobile phone. The determination unit and the determination unit And an abnormality informing unit for informing an alarm is determined to be in the call using the phone.

  The hand region detection unit sets the distance from the upper end of the head region to the upper end of the second search region to be smaller than the distance from the upper end of the face region to the upper end of the first search region. Is preferred.

  The transaction monitoring apparatus further includes a shoulder detection unit that determines whether the contour of the user's shoulder is hidden by the object from the image data, and the hand region detection unit is a determination condition for detecting the user's hand As described above, when the face area is detected by the face detection unit, it is determined that the contour of the shoulder is hidden by the object by the shoulder detection unit, and the head area is detected by the head detection unit It is preferable that the determination condition does not include that the shoulder detection unit determines that the shoulder outline is hidden by the object.

  In this case, the shoulder detection unit extracts an edge line in which the edge pixel extracted from the image data is continuous in the face area direction or the horizontal direction, and the length of the edge line is less than a predetermined length, or It is preferable to determine that the contour of the shoulder is hidden by the object when the edge of the edge line is in contact with the edge line that is continuous in the vertical direction.

  Further, when the abnormality notifying unit detects that the telephone device provided in the automatic transaction terminal is used, even if it is determined that the user is talking using the mobile phone, the abnormality notifying unit may not notify the alarm. preferable.

  The transaction monitoring device according to the present invention is capable of preventing fraudulent acts that abuse the remittance processing by the transaction terminal of the system without hindering smooth money transactions between legitimate users of the automatic transaction system. Play.

It is a schematic block diagram of the automatic transaction system containing the transaction monitoring apparatus which concerns on one Embodiment of this invention. It is a schematic block diagram of the transaction monitoring apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of the search area | region of the outline of a shoulder. (A) is a figure which shows an example of the search area | region of a hand set when the face area is detected, (b) is a hand set up when the face area is not detected. It is a figure which shows an example of a search area | region. It is a flowchart which shows the operation | movement of the telephone call detection process controlled by the computer program run on the control part of the transaction monitoring apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows operation | movement of the whole process of a transaction monitoring apparatus controlled by the computer program run on the control part of the transaction monitoring apparatus which concerns on one Embodiment of this invention.

Hereinafter, a transaction monitoring apparatus according to one embodiment will be described with reference to the drawings.
This transaction monitoring apparatus is installed in a transaction terminal of an automatic transaction system that performs money transactions such as ATM. The transaction monitoring apparatus detects the user's face area from an image of the user of the transaction terminal, and checks whether or not a hand is present in the vicinity of the face area. Determine if you are using and talking. When the transaction monitoring apparatus determines that the user is making a call using the mobile phone, the transaction monitoring apparatus issues a warning to the user that there is a possibility of fraud. In particular, this transaction monitoring device is an area similar to the human head shape when the user's face cannot be detected from the image, such as when the user is scolding to operate the operation screen of the transaction terminal. If it is detected that the user has a hand in the vicinity of the area, it can be determined whether the user is making a call using the mobile phone, thereby determining whether there is a call using the mobile phone with high accuracy. To do.

FIG. 1 is a schematic configuration diagram of an automatic transaction system including a transaction monitoring apparatus according to one embodiment. As shown in FIG. 1, a transaction terminal 100 installed in a store of a financial institution includes a housing 110, a touch panel display 120, accepting ports 111 and 112 such as a passbook or cash card, and accepting / withdrawing money. 113.
Receiving ports 111 and 112 such as passbooks or cash cards are formed above the front surface of the housing 110, and the transaction terminal 100 can be entered into the passbook inserted into the receiving port 111 or inserted into the receiving port 112. It is possible to obtain predetermined information from the cash card. The money accepting / dispensing port 113 is formed in a portion formed in a table shape in front of the housing 110, and the user can deposit and pay out money through the accepting / dispensing port 113. It is possible.

  The touch panel display 120 is installed in a portion formed in a table shape in front of the housing 110 so that information is displayed upward. And the touch panel display 120 displays the operation button and operation guidance according to various processes, such as transfer, payment | withdrawal, and withdrawal, with respect to the user of the transaction terminal 100. When the user presses an operation button displayed on the touch panel display 120, the touch panel display 120 acquires an operation signal corresponding to the pressed operation button, or information indicating a password, a transfer destination, an amount of money, and the like. To do. Transaction terminal 100 then transmits the operation signal or information acquired from touch panel display 120 to a server (not shown) connected via a communication line (not shown), and the user according to a response signal from the server. The process corresponding to the operation selected by is executed.

A monitoring camera 130, a speaker 140, and a sensor 160 are installed in or near the transaction terminal 100, and these devices are connected to the transaction monitoring apparatus 1.
Monitoring camera 130 is installed above the front of housing 110 so that an area from the front to the shoulder of the user of transaction terminal 100 can be photographed. In addition, if the surveillance camera 130 can image | photograph the area | region from a user's head to a shoulder from the front, it will be installed in the wall surface adjacent to the back surface of the transaction terminal 100 separately from the transaction terminal 100, for example. Also good. The surveillance camera 130 includes a two-dimensional detector composed of a photoelectric converter having sensitivity to visible light, such as CCD or C-MOS, and imaging optics for forming an image of the monitoring region on the two-dimensional detector. System. The monitoring camera 130 captures images at regular time intervals (for example, 1/5 second), and sequentially outputs images captured of the user to the transaction monitoring apparatus 1.

  The speaker 140 is installed above the housing 110. The speaker 140 is connected to the transaction monitoring device 1 and emits a sound corresponding to the alarm sound signal received from the transaction monitoring device 1.

  Furthermore, the sensor 160 for detecting the user who operates the transaction terminal 100 is installed in the vicinity of the transaction terminal 100, for example, the wall surface of the booth where the transaction terminal 100 is installed. The sensor 160 can be, for example, a human sensor that receives infrared rays emitted from a detection region set in advance in a predetermined space in front of the transaction terminal 100 and detects a user based on a change in the amount of received light. . Alternatively, the sensor 160 may be an image sensor that detects a user from a change area obtained by a difference between a background image acquired in advance and an image captured by a camera, a mat sensor, a proximity sensor, or the like. The sensor 160 is connected to the transaction monitoring device 1 and outputs a human body detection signal indicating that to the transaction monitoring device 1 while detecting the user. Stop output.

Moreover, the autophone 150 is installed in the vicinity of the transaction terminal 100, for example, on the wall surface of the booth where the transaction terminal 100 is installed. The autophone 150 is connected to an autophone control device 151 having a processor, a semiconductor memory, a communication interface, and its peripheral circuits. When the handset of the autophone 150 is disconnected, the autophone control device 151 automatically transmits to the ATM center device 180 installed in the remote ATM center connected to the autophone control device 151 via the communication line 170. The process is executed, and the user can talk to the operator of the ATM center apparatus 180 via the autophone 150.
Further, the autophone control device 151 is also connected to the transaction monitoring device 1, and when the user is using the autophone 150, outputs an autophone use signal indicating that to the transaction monitoring device 1. Specifically, the autophone control device 151 outputs an autophone use signal to the transaction monitoring device 1 while receiving an off-hook signal indicating that the handset is disconnected from the autophone 150.
In addition, the autophone 150 and the transaction monitoring apparatus 1 may be directly connected, and the transaction monitoring apparatus 1 may be able to confirm that the autophone 150 is in use by directly receiving an off-hook signal from the autophone 150.

Next, the transaction monitoring apparatus 1 according to an embodiment of the present invention will be described.
FIG. 2 is a schematic configuration diagram of the transaction monitoring apparatus 1. As shown in FIG. 2, the transaction monitoring apparatus 1 includes an image acquisition unit 2, an audio output unit 3, a communication unit 4, a storage unit 5, and a control unit 6.

  The image acquisition unit 2 is an image input interface and includes, for example, a video input terminal. Then, the image acquisition unit 2 acquires the image data input from the monitoring camera 130 and passes it to the control unit 6.

  The audio output unit 3 is an audio output interface and includes, for example, an audio output terminal. Then, the audio output unit 3 outputs the alarm audio signal received from the control unit 6 to the speaker 140.

The communication unit 4 is a communication interface, and includes, for example, an interface and a control circuit according to a communication standard such as Ethernet (registered trademark), Universal Serial Bus 2 (USB2), IEEE802.11a, IEEE1394, RS232C. Then, the communication unit 4 receives the autophone use signal from the autophone control device 151 or the off-hook signal from the autophone 150 and passes these signals to the control unit 6.
Further, the communication unit 4 passes the human body detection signal received from the sensor 160 to the control unit 6.
Moreover, the transaction monitoring apparatus 1 and the transaction terminal 100 may be connected via the communication part 4, In this case, the control part 6 of the transaction monitoring apparatus 1 sends an alarm signal etc. via the communication part 4 to the transaction terminal 100. May be sent to.

The storage unit 5 includes at least one of a nonvolatile semiconductor memory such as a flash memory, a volatile semiconductor memory, or a storage device such as a magnetic disk (HDD). Note that the storage unit 5 may be realized as a part of the storage area of the semiconductor memory included in the control unit 6. The storage unit 5 stores various programs and data used in the transaction monitoring apparatus 1. For example, the storage unit 5 stores, as such data, a human body detection flag indicating whether a user is detected by the sensor 160 and an autophone use flag indicating whether the autophone 150 is used. Further, the storage unit 5 stores search area information for setting a search range of the user's hand and shoulder, a template for determining the user's face, head, and hand, determination criterion information, and the like. Furthermore, the memory | storage part 5 memorize | stores the audio | voice data corresponding to the audio | voice message which calls attention to the fraud act to send money from the transaction terminal 100 using a mobile telephone.
The storage unit 5 is connected to the control unit 6 and outputs a predetermined program and data or stores predetermined data in response to a request from the control unit 6.

  The control unit 6 includes a processor unit, semiconductor memories such as RAM and ROM, and peripheral circuits. And the control part 6 controls the transaction monitoring apparatus 1 whole. Further, the control unit 6 determines whether or not the user is using the mobile phone based on the image data received from the monitoring camera 130, and urges attention through the speaker 140 if it is used. Therefore, the control unit 6 includes a portable use detection unit 11, an abnormality notification unit 12, and a flag processing unit 13. Each of these units included in the control unit 6 is, for example, a functional module implemented by a computer program executed on the processor unit.

  The portable use detection unit 11 checks whether there is a user's hand near the user's face or head from the image data received from the monitoring camera 130, so that the user uses the mobile phone. Determine whether or not. For this purpose, the portable use detection unit 11 includes an edge image generation unit 21, a face detection unit 22, a head detection unit 23, a shoulder detection unit 24, a hand region detection unit 25, and a determination unit 26.

The edge image generation unit 21 extracts edge pixels corresponding to the contour of each part of the user from image data (hereinafter referred to as a monitoring image) taken by the user, received from the monitoring camera 130. For this purpose, the edge image generation unit 21 performs, for example, a calculation between neighboring pixels using an edge detection filter such as a sobel filter or a prewitt filter on the monitoring image. Then, the edge image generation unit 21 sets a pixel whose absolute value of the difference value obtained as a result of the calculation between neighboring pixels is larger than a predetermined threshold as an edge pixel. Note that the predetermined threshold can be, for example, an average value of absolute differences obtained for each pixel of the monitoring image. In addition, it is preferable that the edge image generation unit 21 sets the direction of the edge detection filter to at least two directions of the vertical direction and the horizontal direction, and detects the edge in the vertical direction and the edge in the horizontal direction. Then, the edge image generation unit 21 sets the direction of the edge of the pixel to be the maximum value among the obtained absolute difference values in the respective directions.
The edge image generation unit 21 creates an edge image representing the extracted edge pixel. For example, in the edge image, the pixel value of the pixel corresponding to the extracted edge pixel is represented as 1 if it is a horizontal edge, 2 if it is a vertical edge, and the pixel values of other pixels are 0. expressed. Alternatively, the edge image generation unit 21 may create edge direction data representing the direction of each extracted edge pixel separately from the edge image.
The edge image generation unit 21 passes the created edge image to the head detection unit 23, the shoulder detection unit 24, and the hand region detection unit 25. When the face detection unit 22 detects a face area based on the edge image, the edge image generation unit 21 also passes the edge image to the face detection unit 22.

  The face detection unit 22 detects a face area in which the user's face is shown from the monitoring image. Therefore, the face detection unit 22 can use any of various techniques for detecting a face area from an image. For example, when the monitoring image is a color image, the face detection unit 22 converts the color information of each pixel of the monitoring image into HSV color system color information. Then, the face detection unit 22 extracts pixels having color information whose hue H component is included in a predetermined range as skin color pixels. Note that the predetermined range is set to 0 to 30 when the color information of the component of the hue H is represented by 0 to 255, for example. Then, the face detection unit 22 determines whether the candidate region is a face region by an Adaboost discriminator using Haar-like features extracted from the skin color pixels and the region surrounded by the skin color pixels as candidate regions. To do. The Haar-like feature is a luminance difference between a plurality of adjacent rectangular areas arbitrarily set in the input image area. The Adaboost classifier includes a plurality of weak classifiers and a strong classifier that is determined by integrating the determination results of the weak classifiers.

  Each weak classifier calculates a different Haar-like feature from the input image region, and outputs a discrimination result as to whether the input candidate region is a face region based on the calculated Haar-like feature . For example, the weak classifier outputs 1 when determining that the candidate area is a face area, and outputs -1 when determining that the candidate area is not a face area. On the other hand, the strong classifier weights the output result from each weak classifier, and obtains the weighted sum. Then, the strong classifier determines that the candidate area is a face area when the obtained weighted sum is higher than a predetermined threshold (for example, 0).

In addition, the weak classifier that uses the Haar-like feature for identification in which position in the candidate area is used, and the weight for the weak classifier is a plurality of sample images in which the human face is not captured and the human face Is determined by pre-learning using a plurality of sample images in which. The outline of the learning procedure is as follows.
(1) For all the weak classifiers that can be used, a threshold value for identifying whether or not a face is shown in the input image area is set based on the value of the Haar-like feature.
(2) The weight for each sample image is determined. The initial value of the weight is the same value for each sample image.
(3) Each sample image is input to all weak classifiers, and the weights assigned to the sample images that have failed to be identified for each weak classifier are summed up. The sum of the weights is used as the evaluation value.
(4) The weak classifier having the smallest evaluation value is selected as the weak classifier used in the Adaboost classifier. Then, the weight assigned to the output of the weak classifier selected from the evaluation value is determined.
(5) The weight of the sample image for which the selected weak classifier has failed to be identified is increased.
(6) Repeat steps (3) to (5).
For details of the Haar-like feature and the Adaboost discriminator, see, for example, Paul Viola and Michael Jones, "Rapid Object Detection using a Boosted Cascade of Simple Features", IEEE CVPR, vol.1, pp.511-518, 2001. Is disclosed.

Further, the face detection unit 22 may detect a face region by extracting facial feature points such as eye regions, nose tips, and mouth corner points characteristic of a person's face from the monitoring image. For example, the face detection unit 22 detects an elliptical edge distribution approximate to the contour shape of the face from the edge image received from the edge image generation unit 21 using pattern matching or the like. Then, the face detection unit 22 determines in advance the feature amount obtained based on the position, pattern, and the like of the edge pixel existing in the region surrounded by the elliptical edge distribution for the parts such as the eyes, the nose, and the mouth. Each facial feature point can be extracted by checking whether or not the specified condition is satisfied and specifying the position of each part. When these face feature points can be extracted, the face detection unit 22 sets a region surrounded by the elliptical edge distribution as a face region. In addition, instead of performing edge extraction processing to extract edge pixels, the face detection unit 22 performs Gabor conversion processing or wavelet conversion processing to extract pixels having large local changes in different spatial frequency bands. Also good. Furthermore, the face detection unit 22 performs template matching between the template corresponding to the face area and the monitoring image to obtain a correlation value between the template and the monitoring image, and when the correlation value is equal to or greater than a predetermined threshold, A corresponding region in the monitoring image may be detected as a face region.
When the face detection unit 22 can detect the face area from the monitoring image, information representing the face area (for example, the coordinates of the pixel at the upper left corner of the circumscribed rectangle of the face area and the coordinates of the pixel at the lower right corner) To the shoulder detection unit 24 and the hand region detection unit 25. On the other hand, if the face area cannot be detected from the monitoring image, the face detection unit 22 returns a face detection impossible signal indicating that to the portable use detection unit 11.

When the face detection unit 22 cannot detect a face area from the monitoring image, the head detection unit 23 detects a head area in which the user's head is reflected from the monitoring image. For example, as illustrated in FIG. 1, the touch panel display 120 of the transaction terminal 100 is arranged to display information upward. Therefore, the user may be crawling to operate the transaction terminal 100. When the user is scolding, most of the user's face is hidden behind the head and cannot be seen on the monitoring image. That is, the facial feature information on the monitoring image does not appear clearly compared to when the user is facing the front.
Therefore, the head detection unit 23 detects the head region without using the facial features of the user. For example, the head detection unit 23 performs template matching between at least one elliptical template that is a pattern approximate to the contour shape of the head viewed from the top direction and the edge image received from the edge image generation unit 21. To obtain a correlation value between the template and the edge image. Then, when the correlation value is equal to or greater than a predetermined threshold, the head detection unit 23 detects an area in the monitoring image corresponding to the correlation value as a head area.

Note that the head detection unit 23 assumes that the user's head is present in the monitoring image in accordance with the installation position, angle of view, and shooting direction of the monitoring camera 130 in order to prevent erroneous detection and improve processing speed. It is preferable to perform pattern matching only in a region to be processed. Note that the region where the head is supposed to exist is, for example, when the monitoring camera 130 is installed such that the space above the touch panel display 120 of the transaction terminal 100 is located at the center of the monitoring image. Set to
The head detection unit 23 passes information indicating the head region (for example, the coordinates of the pixel at the upper left end point and the coordinates of the pixel at the lower right end point of the circumscribed rectangle of the head region) to the hand region detection unit 25. On the other hand, if the head detection unit 23 cannot detect the head region from the monitoring image, the head detection unit 23 returns a head detection impossible signal indicating that to the portable use detection unit 11.

When the face detection unit 22 can detect the face area of the user, the shoulder detection unit 24 detects the contour of the user's shoulder from the edge image generated by the edge image generation unit 21. Then, the shoulder detection unit 24 determines whether or not the contour of the shoulder is hidden by an object such as a user's hand or a mobile phone. First, in order to prevent erroneous detection of the shoulder contour, the shoulder detection unit 24 sets a shoulder contour search region in the vicinity of the detected face region in consideration of the human body shape.
FIG. 3 shows an example of a shoulder contour search region. As shown in FIG. 3, when the user's face area 310 is detected in the monitoring image 300, the search area 320 for the contour of the right shoulder (the shoulder on the left side in the image) is the face area 310. The end point of the upper right corner of the search area 320 is set to overlap the end point of the lower left corner of the circumscribed rectangle. In addition, the search area 330 of the contour of the left shoulder (the shoulder on the right side in the image) is set so that the end point of the upper left corner of the search area 330 overlaps the end point of the lower right corner of the circumscribed rectangle of the face area 310. The widths and heights of the search areas 320 and 330 are set to be twice the width of the face area 310 and 1/2 the height of the face area 310, respectively (in FIG. 3, the size of the search area). Is shown in a simplified manner).

  The shoulder detection unit 24 performs thinning processing on the edge image in each search region, and extracts edge lines in which edge pixels are connected with a predetermined number of pixels (for example, one pixel). Here, the shoulder detection unit 24 performs a probabilistic Hough transform process on the edge line, and approximates the edge line to a line segment. As a result, the curved edge line is divided into a plurality of edge lines having ends at portions having a high curvature. Then, the shoulder detection unit 24 detects an edge line extending in the horizontal direction or the face direction and an edge line extending in the vertical direction among the extracted edge lines in each search region. Here, in each search region, an angle formed by a straight line that is high on the side close to the face and low on the side far from the face and a straight line extending in the horizontal direction is positive. At this time, when the angle between the edge line of interest and the horizontal axis is included in the range of 0 ° to 30 ° with respect to the horizontal axis, the shoulder detection unit 24 extends the edge line in the horizontal direction or the face direction. A line. Note that the shoulder detection unit 24 uses, for example, the average value of the inclination with respect to the horizontal axis between adjacent edge pixels on the edge line or the inclination with respect to the horizontal axis between the end points of the edge line as the angle formed by the edge line and the horizontal axis. . Alternatively, the shoulder detection unit 24 includes only the edge line in which the edge pixels corresponding to the edge changing in the vertical direction are connected, and the angle between the edge line and the horizontal axis is included in the range of 0 ° to 30 °. If the angle is included in the range of 0 ° to 30 °, the edge line may be an edge line extending in the horizontal direction or the face direction.

  Further, when the angle formed between the edge line of interest and the horizontal axis is included in the range of 60 ° to 120 ° with respect to the horizontal axis, the shoulder detection unit 24 sets the edge line as an edge line extending in the vertical direction. Alternatively, the shoulder detection unit 24 may detect an edge line in which edge pixels corresponding to edges changing in the horizontal direction are connected as an edge line extending in the vertical direction.

The shoulder detection unit 24 determines that the contour of the shoulder is hidden by the object when the edge line extending in the horizontal direction or the face direction is less than a predetermined length in each search area. In addition, when the edge line extending in the horizontal direction or the face direction is longer than a predetermined length in each search area, the shoulder detection unit 24 contacts the edge line extending in the vertical direction. If it is, it is determined that the contour of the shoulder is hidden by the object. That is, the shoulder detection unit 24 detects the shoulder when the edge line extending in the horizontal direction or the face direction is less than a predetermined length, or when the end of the edge line is in contact with the edge line continuous in the vertical direction. Is determined to be hidden by the object. This is because, if the contour line of the shoulder cannot be seen by being blocked by an object such as an arm or a mobile phone, there is an edge line extending vertically along the contour of the arm or the mobile phone on the monitoring image, The vertical edge line corresponds to the cutting of the shoulder outline. On the other hand, if the edge line extending in the horizontal direction or the face direction is longer than a predetermined length and the end of the edge line is not in contact with the edge line extending in the vertical direction, the shoulder detection unit 24 has a contour of the shoulder. Judge that it was detected. The predetermined length is set to, for example, ¼ to ３ of the width of the face area detected by the face detection unit 22.
The shoulder detection unit 24 returns a shoulder detection result signal indicating the determination result of whether or not the contour of the shoulder is hidden by the object to the portable use detection unit 11.

The hand region detection unit 25 determines whether or not the user's hand can be detected in the vicinity of the face region detected by the face detection unit 22 or the head region detected by the head detection unit 23. Therefore, first, when the user is using the mobile phone, the hand region detection unit 25 sets a hand search region in consideration of bringing the hand close to the side of the user's face.
FIG. 4A shows an example of a hand search area that is set when a face area is detected, and FIG. 4B shows a hand that is set when a face area is not detected. An example of the search area is shown. As shown in FIG. 4A, when the user's face area 410 is detected, the upper end of the hand search area 420 is set to the center of the face area 410 in the height direction, and the lower end of the search area 420 Is set so as to contact the lower side of the monitoring image 400 so that the search area 420 includes the lower end of the position where the user is likely to place his / her hand during a call. Further, the left and right ends of the search area 420 are set to include a range that is assumed to include both shoulders of the user. For example, the left and right ends of the search area 420 are set at positions that are separated from the horizontal center of the face area 410 by a distance that is two to three times the width of the face area 410 (see FIG. ) Shows the size of each region in a simplified manner). Furthermore, in a region 430 having a predetermined width below the lower end of the face region 410, there is an object from which skin color and a large number of edge pixels are extracted, such as a neck or clothing button. Therefore, this region 430 may be excluded from the search region 420 in order to prevent erroneous detection of an object from which skin color and a large number of edge pixels are extracted as a hand. In addition, when a user makes a call using a mobile phone, the user is unlikely to bring his hand under the face. Therefore, even if such a region 430 is excluded from the hand search region, It is considered unlikely that the search will fail. The width of the excluded area 430 is set to, for example, 1/2 to 1 times the width of the face area 410, and the center line of the excluded area 430 matches the center line of the face area 410. The horizontal position of the excluded area 430 is set.

On the other hand, as shown in FIG. 4B, when the user's face area is not detected and the user's head area 440 is detected instead, the upper end of the hand search area 450 is It is set at the upper end of area 440. This is because, unlike when the user is facing the front, when the user is scolding, the position of the hand may be located near the upper end of the head region. The lower end of the search area 450 is set so as to come into contact with the lower side of the monitoring image 400 so that the search area 450 includes the lower end of the position where the user is likely to place his / her hand during a call. Further, the left and right ends of the search area 450 are set so as to include a range that is assumed to include both shoulders of the user. For example, the left and right ends of the search area 450 are set at positions that are separated from the horizontal center of the head area 440 by a distance that is two to three times the width of the head area 440 (see FIG. 4). (B) shows the size of each region in a simplified manner). Also in this case, the region 460 having a predetermined width below the lower end of the head region 440 may be excluded from the search region 450.
Thus, the hand region detection unit 25 determines the position of the upper end of the search region with respect to the height of the head region when the user's head region is detected, when the user's face region is detected. It is set above the position of the upper end of the search area with respect to the height of the face area. Specifically, the height of the head region is h _c , the distance from the top of the head region to the top of the search region is d _c , the height of the face region is h _f , and the top of the face region is the top of the search region If the distance to is d _f , the search area is set so that d _c / h _c <d _f / h _f or d _c <d _f . Thereby, the user's hand during a call can be detected with high accuracy.

The hand region detection unit 25 determines whether or not a hand can be detected in the hand search region described above. Here, the hand has an inverted U-shaped outline on the image, and has a feature that a large number of edge pixels exist in the outline and a lot of skin color pixels are included in the outline. Therefore, the hand region detection unit 25 detects, for example, skin-colored pixels existing in the hand search region set in the monitoring image. As described with respect to the face detection unit 22, the skin color pixel is a color whose hue H component is included in a predetermined range such as 0 to 30 when the color information of the pixel is expressed in the HSV color system. It is a pixel having information.
The hand region detecting unit 25 extracts a skin color region including the skin color pixel as a hand candidate region by performing a labeling process on the skin color pixel. The hand region detection unit 25 may include a non-skin color pixel surrounded by a skin color region in the hand candidate region by performing a morphological closing operation or the like. Accordingly, even when a pixel corresponding to a part of the hand is not a skin color on the monitoring image due to an influence of a shadow or the like, the hand region detection unit 25 includes a non-skin color pixel corresponding to a part of the hand in the hand candidate region. Can do. The hand region detection unit 25 may extract a plurality of hand candidate regions.

When the face area is detected by the face detection unit 22, the hand area detection unit 25 detects the area S _h of the hand candidate area, that is, the number of pixels included in the hand candidate area, and the area S _{f of the} face area, A ratio S _h / S _f with the number of pixels included in the face area is obtained. In the hand region detection unit 25, the ratio S _h / S _f between the area of the hand candidate region and the area of the face region is included in a predetermined range corresponding to a general ratio of the size of the human face and hand. It is determined whether or not. For example, the predetermined range is a range from 0.1 to 1.
On the other hand, not detected face area, when the head region is detected, the hand region detection part 25, the area S _h hand candidate area, the area of the head region S _c, i.e., included in the head region A ratio S _h / S _c to the number of pixels to be obtained is obtained. The hand region detection part 25, the ratio S _h / S _c of the area of the area and head area of the hand candidate region, equivalent to the head of the human to the general ratios of the size of the area and hand as viewed from the top It is determined whether or not it falls within a predetermined range. For example, the predetermined range is a range from 0.1 to 1.

When S _h / S _f or S _h / S _c is out of a predetermined range, the hand area detection unit 25 determines that the hand candidate area is not a user's hand. On the other hand, when S _h / S _f or S _h / S _c is included in the predetermined range, the hand region detection unit 25 includes the number E _{h of} edge pixels included in the region on the edge image corresponding to the hand candidate region. Count. The hand region detection part 25, a predetermined the ratio E _h / S _h number E _h of the edge pixels to the area S _h hand candidate region corresponds to the minimum value of the common ratio of the edge pixels contained in the hand region It is determined whether or not it is equal to or greater than the threshold value. For example, the predetermined threshold is set to 0.1.
When the ratio E _h / S _h of the number E _h of edge pixels to the area _Sh of the hand candidate area is equal to or less than a predetermined threshold, the hand area detection unit 25 determines that the hand candidate area is not the user's hand. . On the other hand, when the ratio E _h / S _h is larger than a predetermined threshold, the hand region detection unit 25 determines that the hand candidate region is a hand region depending on whether or not a face region is detected. Change the criteria. That is, when the user is facing the transaction terminal 100 and a face is shown on the monitoring image, the user makes a call using a mobile phone as shown in FIG. If so, the hand holding the mobile phone will hide the outline of one shoulder. On the other hand, if the user is talking to operate the transaction terminal 100 or view the touch panel 120, even if the user is making a call using the mobile phone, the hand holding the mobile phone will show the shoulder contour on the monitoring image. It is highly possible that the hand does not hide the contour of the shoulder.

Therefore, when the face area is detected, the hand area detection unit 25 indicates that the shoulder detection result signal output by the shoulder detection unit 24 indicates that the contour of one shoulder is hidden by the object. The hand candidate area is determined to be the user's hand. On the other hand, if the shoulder detection result signal indicates that both shoulders are detected, the hand region detection unit 25 determines that the hand candidate region is not the user's hand.
The hand region detection unit 25 may determine that the hand candidate region is the user's hand only when the contour of the shoulder where the hand candidate region exists is hidden by the object.

On the other hand, when the face area is not detected, the hand area detection unit 25 uses a hand candidate area in which S _h / S _c is included in a predetermined range and E _h / S _h is larger than a predetermined threshold. It is determined that it is a person's hand.
The hand region detection unit 25 returns a hand region detection result signal indicating the determination result of whether or not the user's hand has been detected to the portable use detection unit 11.

  Based on the detection result of the user's hand, the determination unit 26 determines whether the user is in a call using the mobile phone. Specifically, when the determination unit 26 returns a determination result indicating that the user's hand has not been detected with respect to the monitoring image acquired at any time point, the control unit The counter value stored in the memory of 6 is reset to 0. On the other hand, when the determination unit 26 returns a determination result indicating that the user's hand has been detected with respect to the monitoring image acquired at any time, the value of the counter is incremented by one. To do. Therefore, when the user's hand is continuously detected for the sequentially acquired monitoring images, the value of the counter increases. When the value of the counter reaches a predetermined value, that is, when the user's hand is detected in the vicinity of the user's face or head for a certain period of time, the determination unit 26 indicates that the user Is used to determine that a call is in progress. The predetermined value is preferably set to, for example, the number of monitoring images acquired during a period (for example, 1 minute) that is considered necessary for the user to listen to an instruction from a fraudulent person.

  When it is determined that the user is in a call using the mobile phone, the determination unit 26 returns a call detection signal indicating that the user is in a call to the portable use detection unit 11. Then, the portable use detection unit 11 notifies the control unit 6 of the call detection signal. On the other hand, if the value of the counter is less than the predetermined value, the determination unit 26 does not know whether the user is busy or not, and therefore does not notify the portable use detection unit 11 of the determination result.

  With reference to the flowchart shown in FIG. 5, the operation procedure of the call detection process by the portable use detection unit 11 of the control unit 6 will be described. This operation procedure is controlled by a computer program executed in the control unit 6.

First, when a monitoring image is acquired, the edge image generation unit 21 creates an edge image obtained by extracting edge pixels corresponding to the contour of each part of the user from the monitoring image (step S101). The edge image generation unit 21 passes the created edge image to the head detection unit 23, the shoulder detection unit 24, and the hand region detection unit 25. When the face detection unit 22 detects a face area based on the edge image, the edge image generation unit 21 also passes the edge image to the face detection unit 22.
Next, the face detection unit 22 executes processing for detecting the face area of the user facing the front with respect to the transaction terminal 100, and determines whether or not the face area has been detected (step S102). If the face area cannot be detected in step S102, the face detection unit 22 returns a face detection impossible signal to the portable use detection unit 11. Then, the head detection unit 23 executes a process of detecting the head region from the monitoring image, and determines whether or not the head region has been detected (step S103). If the head region cannot be detected in step S103, a head detection impossible signal indicating that is returned to the portable use detection unit 11. And the portable use detection part 11 transfers control to step S107. On the other hand, if a head region is detected in step S103, the hand region detection unit 25 can detect the hand region by executing a process of detecting the hand region in the search region set in the vicinity of the head region. It is determined whether or not (step S104). If the hand region cannot be detected in step S104, the hand region detection unit 25 returns a hand region detection result signal indicating that the user's hand has not been detected to the portable use detection unit 11. And the portable use detection part 11 transfers control to step S107. On the other hand, when the hand region can be detected in step S104, the hand region detection unit 25 returns a hand region detection result signal indicating that the user's hand has been detected to the portable use detection unit 11. And the portable use detection part 11 transfers control to step S109.

If a face area is detected in step S102, the face detection unit 22 passes information indicating the face area to the shoulder detection unit 24 and the hand area detection unit 25. Then, the shoulder detection unit 24 detects the shoulder contour in the search region set in the vicinity of the face region, and determines whether or not the shoulder contour is hidden by the object (step S105). Then, the shoulder detection unit 24 returns a shoulder detection result signal indicating the determination result to the portable use detection unit 11. Next, the hand area detection unit 25 executes a process of detecting a hand area within the search area set in the vicinity of the face area, and determines whether or not the hand area has been detected (step S106). At that time, as described above, the hand region detection unit 25 refers to the determination result as to whether or not the contour of the shoulder is hidden by the object. In step S <b> 106, when the hand region cannot be detected, the hand region detection unit 25 returns a hand region detection result signal indicating that the user's hand has not been detected to the portable use detection unit 11. And the portable use detection part 11 transfers control to step S107.
In step S107, the determination unit 26 resets the counter to zero. And the portable use detection part 11 determines whether the human body detection flag memorize | stored in the memory | storage part 5 is ON (step S108). In step S108, if the human body detection flag is OFF, the portable use detection unit 11 ends the call detection process. On the other hand, if the human body detection flag is ON in step S108, the portable use detection unit 11 returns the control to step S101, and executes call detection processing on the newly acquired monitoring image.

On the other hand, when the hand region can be detected in step S106, the determination unit 26 increments the counter by 1 (step S109). Then, the determination unit 26 determines whether or not the counter value has reached a predetermined value (step S110). In step S110, when the value of the counter has not reached the predetermined value, the portable use detection unit 11 returns the control to step S101, and executes a call detection process on the newly acquired monitoring image.
On the other hand, if the value of the counter has reached the predetermined value in step S110, the determination unit 26 determines that the user is in a call using the mobile phone (step S111). Then, the determination unit 26 returns a call detection signal indicating that it is detected that a call is in progress to the portable use detection unit 11, and the portable use detection unit 11 notifies the control unit 6 of the call detection signal. And the determination part 26 resets a counter to 0 (step S112), and the portable use detection part 11 complete | finishes a telephone call detection process after that.
Note that the details of the processing in each step have been described in the description of each unit of the portable use detection unit 11, and thus the description thereof is omitted here.

If the later-described autophone use flag is set to OFF, the abnormality notification unit 12 checks whether a call detection signal is notified from the portable use detection unit 11. Then, the abnormality notification unit 12 transfers money from the transaction terminal 100 when the call detection signal is notified from the portable use detection unit 11, that is, when it is determined that the user is calling using the mobile phone. The alarm voice data for calling attention to the fraudulent act is read from the storage unit 5. Then, the abnormality notifying unit 12 outputs alarm sound data to the speaker 140 via the sound output unit 3, and issues an alarm to the user. The abnormality notifying unit 12 ends the process of notifying the alarm sound after repeatedly outputting the alarm sound data a predetermined number of times. Alternatively, the abnormality notification unit 12 may repeatedly output the alarm sound data to the speaker 140 while the call detection signal is notified from the portable use detection unit 11. In addition, the process which alert | reports the alarm which the abnormality alerting | reporting part 12 performs is not restricted to the output of alarm sound data, for example, the abnormality alerting | reporting part 12 outputs the abnormality signal to the autophone control apparatus 151, and the autophone control apparatus which received this 151 may be a process of notifying the ATM center device 180 via the communication line 170 that the user is likely to be fraudulent.
On the other hand, if the autophone use flag is set to ON, the user is not erroneously recognizing that he / she is talking to the operator using the autophone 150 as a call using the mobile phone. The abnormality notification unit 12 does not output alarm sound data to the speaker 140.

The flag processing unit 13 includes a human body detection flag that serves as a trigger for the portable use detection unit 11 to start the call detection process, and an autophone use flag that serves as a determination criterion as to whether or not the abnormality notification unit 12 outputs alarm sound data. Set the value of.
Specifically, the flag processing unit 13 detects the human body stored in the storage unit 5 while receiving from the sensor 160 a human body detection signal indicating that a user operating the transaction terminal 100 has been detected. Set the flag to ON. On the other hand, when the flag processing unit 13 stops receiving the human body detection signal, it rewrites the human body detection flag to OFF.
Further, the flag processing unit 13 stores the storage unit 5 while receiving an autophone use signal indicating that the autophone 150 is being used from the autophone control device 151 or receiving an off-hook signal from the autophone 150. Is set to ON. On the other hand, when the flag processing unit 13 stops receiving the autophone use signal and the off-hook signal, it rewrites the autophone use flag to OFF.

With reference to the flowchart shown in FIG. 6, the operation procedure of the entire process performed by the transaction monitoring apparatus 1 will be described. This operation procedure is controlled by a computer program executed in the control unit 6.
When transaction terminal 100 is activated, transaction monitoring device 1 is also activated and processing is started. And the control part 6 determines whether the human body detection flag memorize | stored in the memory | storage part 5 is set to ON (step S201). If the human body detection flag is OFF in step S201 (step S201-No), the control unit 6 advances the control to step S206. On the other hand, if the human body detection flag is ON in step S201 (step S201-Yes), the portable use detection unit 11 of the control unit 6 executes a call detection process (step S202). The detailed procedure of the call detection process is as described above with reference to FIG. Then, the control unit 6 refers to the call detection signal from the mobile use detection unit 11 and determines whether or not the user is making a call using the mobile phone (step S203).

  When it is determined in step S203 that the user is talking using a mobile phone (step S203—Yes), the abnormality notifying unit 12 of the control unit 6 starts outputting alarm sound data to the speaker 140. (Step S204). After starting the output of the audio data, the control unit 6 determines whether or not the human body detection flag stored in the storage unit 5 is set to ON (step S205). If it is determined that the human body detection flag is ON, the control unit 6 loops the process of step S205 until the human body detection flag is turned OFF. This is to prevent the warning voice data from being output many times to the same user. On the other hand, if it is not determined in step S203 that the user is making a call using a mobile phone (step S203—No), or after it is determined No in step S205, the control unit 6 performs the transaction. It is determined whether the terminal 100 has finished the service (step S206). Note that the control unit 6 can determine that the transaction terminal 100 has ended the service by receiving the notification that the transaction terminal 100 has reached the end of business hours or that the transaction terminal 100 is shutting down. In step S206, when transaction terminal 100 has not ended the service, control unit 6 returns control to step S201, and repeats the processing of steps S201 to S206. On the other hand, when transaction terminal 100 terminates the service in step S206, the process of transaction monitoring device 1 is terminated and transaction monitoring device 1 is turned off.

  As described above, the transaction monitoring apparatus according to an embodiment of the present invention is installed in a transaction terminal that performs money transactions such as ATM. The transaction monitoring apparatus detects the user's face area from an image of the user of the transaction terminal, and checks whether or not a hand is present in the vicinity of the face area. Determine whether it is in use. When this transaction monitoring apparatus determines that the user is using the mobile phone, the transaction monitoring apparatus issues a warning to the user that there is a possibility of fraud. In particular, when the transaction monitoring apparatus cannot detect the user's face from the image, the user can detect that there is a hand in the vicinity of the area by detecting an area similar to the human head shape. Determine that you are using. Thus, the transaction monitoring apparatus can determine whether or not the user is using the mobile phone with high accuracy even when the user is calling to operate the operation screen of the transaction terminal.

The preferred embodiments of the present invention have been described above, but the present invention is not limited to these embodiments. For example, the portable use detection unit 11 may interrupt the execution of the call detection process while the autophone use flag is set to ON.
In the operation procedure of the call detection process shown in FIG. 5, the portable use detection unit 11 may exchange the order of the process of step S105 and the process of step S106. In this case, when the hand region detection unit 25 detects the hand region in either the left or right of the face region, the hand region detection unit 25 notifies the shoulder detection unit 24 of whether the hand region is detected in the left or right side of the face region. And the shoulder detection part 24 should just determine whether the outline of a shoulder is hidden by the object only in the search area | region of the notified side. In this case, the portable use detection unit 11 has the hand region detected by the hand region detection unit 25 and the shoulder contour is hidden by the object in the search region on the side where the hand region is detected by the shoulder detection unit 24. If it is determined that there is, control proceeds to step S109. On the other hand, in the portable use detection unit 11, the hand region is not detected by the hand region detection unit 25 or the contour of the shoulder is hidden by an object in the search region on the side where the hand region is detected by the shoulder detection unit 24. If it is determined that it is not, control proceeds to step S107.
Furthermore, the shoulder detection unit 24 may be omitted according to the required detection accuracy of the hand region. In this case, even when the face area is detected, the hand area detection unit 25 determines whether or not the hand area is detected in step S106 without referring to the determination result regarding the shoulder contour. Moreover, the portable use detection part 11 may interchange the order of the process of step S102, S105, and S106, and the process of step S103 and S104.
As described above, those skilled in the art can make various modifications in accordance with the embodiment to be implemented within the scope of the present invention.

DESCRIPTION OF SYMBOLS 100 Transaction terminal 110 Case 120 Touch panel display 130 Surveillance camera 140 Speaker 150 Autophone 151 Autophone control apparatus 1 Transaction monitoring apparatus 2 Image acquisition part 3 Audio | voice output part 4 Communication part 5 Memory | storage part 6 Control part 11 Portable use detection part 12 Abnormality notification part DESCRIPTION OF SYMBOLS 13 Flag processing part 21 Edge image generation part 22 Face detection part 23 Head detection part 24 Shoulder detection part 25 Hand area detection part 26 Determination part

Claims (4)

A transaction monitoring device for monitoring the behavior of a user of an automatic transaction terminal capable of remittance processing,
A face detection unit that extracts a face area corresponding to a user's face from image data acquired from a monitoring camera installed so as to photograph the front direction of the user facing the automatic transaction terminal;
A head detection unit that extracts, as a head region, a region that approximates the contour shape of the head viewed from the top of the user's head from the image data;
When the face area is detected by the face detection unit, the user's hand is detected from the first search area set based on the face area , and the face area is not detected , and the head detection unit When the head region is detected, a hand region detection unit that detects a user's hand from the second search region set with reference to the head region;
From each of the image data sequentially acquired during a predetermined period, when the hand region detection unit detects a user's hand, a determination unit that determines that the user is in a call using a mobile phone;
When the determination unit determines that the user is busy using a mobile phone, an abnormality notification unit that notifies an alarm;
A transaction monitoring apparatus comprising:   The hand region detection unit is configured such that the distance from the upper end of the head region to the upper end of the second search region is smaller than the distance from the upper end of the face region to the upper end of the first search region. The transaction monitoring apparatus according to claim 1, wherein the transaction monitoring apparatus is set. A shoulder detection unit for determining whether the contour of the user's shoulder is hidden by the object from the image data;
The hand region detection unit, as a determination condition for detecting a user's hand, if the face region is detected by the face detection unit, the shoulder detection unit is hidden by the shoulder contour If the head region is detected by the head detector, the determination condition includes that the shoulder detector determines that the shoulder outline is hidden by the object. The transaction monitoring apparatus according to claim 1, wherein there is no transaction.   The shoulder detection unit extracts edge lines in which the edge pixels extracted from the image data are continuous in the face area direction or the horizontal direction, and the length of the edge lines is less than a predetermined length, or The transaction monitoring apparatus according to claim 3, wherein when the edge portion of the edge line is in contact with the edge line that is continuous in the vertical direction, it is determined that the contour of the shoulder is hidden by the object.

Images