WO2013157466A1

WO2013157466A1 - Smoking detection device, method and program

Info

Publication number: WO2013157466A1
Application number: PCT/JP2013/060859
Authority: WO
Inventors: 賢治小原
Original assignee: 株式会社デンソー
Priority date: 2012-04-20
Filing date: 2013-04-10
Publication date: 2013-10-24
Also published as: JP2013225205A

Abstract

This smoking detection device detects a smoking state of persons on the basis of the image information from an imaging unit which images persons. This smoking detection device is provided with: a smoking region setting means which, on the basis of a face image taken of a person's face, sets a smoking region where a cigarette being smoked is predicted to be; a cigarette detection means which detects cigarettes from images in said smoking region; and a smoking inference means which infers a smoking state if there is a cigarette in the smoking region.

Description

Smoking detection device, method and program

The present invention relates to a smoking detection apparatus, method, and program that can capture an image of a driver's face (face image) of a car, for example, and can grasp the smoking state of the driver from the face image.

Conventionally, in order to detect a driver's state that is not preferable in driving operation such as a look-aside or doze, or to perform personal authentication of the driver, a near infrared light is irradiated from the projector toward the driver's face, 2. Description of the Related Art There is known a driver monitor system that captures a driver's face with a camera, obtains a face image, and analyzes the face image to detect a driver's state or perform personal authentication.

In recent driver monitor systems, the driver's smoking state is detected by detecting the high-temperature combustion part at the tip of the cigarette with an infrared camera (temperature sensor), and air conditioning control is performed to activate the air conditioner. Techniques for detecting littering of tobacco have been proposed (see Patent Documents 1 to 3 below).

JP 2010-218494 A JP 2009-15519 A JP 2009-23400 A

However, in the above-described prior art, since an smoking state is detected using an infrared camera, an infrared camera is required in addition to a driver monitor system using a normal camera. Therefore, there is a problem that the apparatus becomes complicated and the cost increases.

Also, there is a problem that the accuracy of detecting the smoking state is low when an infrared camera is not used to simplify the device.

One embodiment provides a smoking detection apparatus, method, and program capable of accurately detecting a driver's smoking state with a simple configuration.

As one aspect of the embodiment, there is provided a smoking detection device that detects a smoking state of the person based on image information from an imaging unit that photographs the person. A smoking detection device detects a cigarette from a smoking area setting means for setting a smoking area where a smoking cigarette is expected to exist based on a face image obtained by photographing the person's face, and an image in the smoking area And a smoking estimation means for estimating that the tobacco is in the smoking state when the tobacco is present in the smoking area.

It is explanatory drawing which shows the driver monitor system structure which has a function of the smoking detection apparatus of one Embodiment. It is explanatory drawing which shows the positional relationship of a camera, a light projector, and a driver | operator. It is a block diagram which shows the electric constitution of a smoking detection apparatus. (A) is explanatory drawing which shows the mask formed of AAM, (b) is explanatory drawing which shows the area | region of the mouth and nose of a mask. (A) is explanatory drawing which shows the registered luminance distribution in a smoking area | region, (b) is explanatory drawing which shows the luminance distribution in the smoking area | region of the present flame | frame, (c) is the difference of the image of (a), and the image of (b). It is the difference image which shows. It is explanatory drawing which divides a smoking area | region into small areas and shows the arrangement | positioning (luminance distribution) of the brightness | luminance in each small area. It is explanatory drawing which shows the front-end | tip part and other part of a tobacco. It is a flowchart which shows the luminance distribution registration routine which is the processing content of a smoking detection apparatus. It is a flowchart which shows the smoking detection routine which is the processing content of a smoking detection apparatus.

Hereinafter, embodiments of the smoking detection apparatus of the present invention will be described with reference to the drawings.

A) First, the system configuration of a vehicle equipped with the smoking detection device of this embodiment will be described with reference to FIGS.

As shown in FIG. 1, in order to detect the driver's (driver's) snooze, look aside, or perform personal authentication, the vehicle (automobile) captures the driver's face image, A driver monitor system 1 for analyzing images is installed. In this embodiment, this driver monitor system 1 is used as a smoking detection device.

The driver monitor system 1 includes, for example, an imaging unit 5 disposed in the vicinity of a dash mode 3 meter (not shown) and a control unit 7 that controls the operation of the imaging unit 5 and the like.

As shown in FIG. 2, the imaging unit 5 includes a camera 9 that images the driver's face and a pair of

imaging projectors

11a and 11b (collectively referred to as 11) for irradiating the driver's face and the like with light. It has.

The camera 9 is, for example, a CCD camera that can take an image with near-infrared light (that is, has a constant sensitivity to the near-infrared light), and for example, takes an image of the driver's face obliquely from the front and below. Is arranged to be possible.

The projector 11 is made of, for example, a near-infrared LED, and is arranged so as to be substantially coaxial with the camera 9 so as to irradiate near-infrared light toward the driver's face. The irradiation area is substantially conical around the driver's face.

b) Next, the electrical configuration of the driver monitor system 1 will be described with reference to FIG.
As shown in FIG. 3, the driver monitor system 1 includes an imaging unit 5 having the camera 9 and the projector 11 described above, and a control unit 7 that controls the imaging unit 5 and the like. An air conditioner 13, a speaker 15, a power window 17, an air purifier 19, and the like are connected to the control unit 7 as devices controlled by a control signal from the control unit 7.

Among these, the control unit 7 is an electronic control device including a known microcomputer. The control unit 7 performs image processing and the like based on the image signal from the camera 9 and performs various controls according to the detection of the driver's smoking state and the detection result as described later.

c) Next, the principle of processing performed as a smoking detection device in the driver monitor system 1 will be described with reference to FIGS.
In the smoking detection apparatus of the present embodiment, first, a driver's face is photographed by a camera, and a face image indicating the driver's face is acquired from the photographed image.

As a method for acquiring the face image, for example, a well-known method as described in Japanese Patent Application Laid-Open No. 2008-276328 can be employed. In this method, as described in paragraphs [0030] to [0032] of the publication, based on the difference between the luminance of the driver's face and the luminance of the driver's background image in the image, A face image (face image) is extracted.

Next, with respect to the face image obtained as described above, as shown in FIG. 4A, for example, from a line segment (wire frame) that divides parts such as eyes, mouth and nose of the face. A mask having a mask pattern is formed.
This mask is represented by a figure (face patch drawing) in which triangles (triangular patches) connecting feature points of the face image are combined.

The feature points in the face image indicate points on the face having features that are clearly different from the surroundings on the image, such as the edges of the eyes and the mouth (especially useful for face discrimination). Furthermore, secondary points (for example, points at specific positions between feature points) obtained from the feature points may be included in the feature points. Note that in any face image, the feature points can be obtained by, for example, a well-known edge detection process.

As a method for forming such a mask pattern, for example, a well-known AAM (Active Appearance Models) method can be used (for example, Edwards, G., Taylor, C. J. and Cootes, T. F .: Interpreting Face). Images using Active Appearance Models, In IEEE Conf. On Automatic Face and Gesture Recognition 1998, pp.300-305, Japan, (1998)).

This AAM is a technique mainly used for facial expression, tracking, face recognition, etc., and it is a two-dimensional space that expresses the change in appearance from the basic shape with the arrangement of a certain feature point group as the basic shape. Is a known face image model. The AAM method will be briefly described below.

(1) For the learning face image, set its feature points (eye ring, nasal head, mouth edge, etc.) and create a model. At this point, it can be seen which feature point indicates which part.

(2) Next, register brightness information for learning.

(3) Next, each feature point is changed based on the principal component vector calculated by the principal component analysis.

(4) Next, the luminance information of the registered face is changed based on (3), and the luminance value (registered luminance value) of the registered luminance information matches the input image in the current frame. The process (3) is repeated.

(5) Next, the position where the luminance value is closest is set as the model fitting position (that is, the position where the mask is estimated to fit the face image in the current frame).

Since the (1) in advance which feature points which sites are known, after the processing of (5), by drawing a line for each site, a mask as shown in FIG. 4 (a) forming it can.

(6) Next, when a mask can be formed suitably as described above, a mask valid flag indicating that is set.

Specifically, the setting of the mask valid flag is determined by threshold processing of the difference between the registered luminance value at the time of fitting and the luminance value of the input image, for example. That is, the mask valid flag is a flag indicating whether or not the mask has been accurately fitted to the face image in the current frame.

Next, as shown in FIG. 4B, in the face image, the region of the mouth and nose (gray portion in the same figure) is divided in order to determine the region where the driver is smoking. . The mouth and nose region is a region connecting feature points indicating the outer periphery of the mouth and nose.

Next, a smoking area where smoking tobacco is expected to exist is set so as to include the mouth area and the nose area.
Specifically, for example, a rectangular smoking area as shown in FIG. That is, the upper end of the nose region is the upper end of the smoking region, the lower end of the mouth region is a predetermined value (a predetermined number of pixels) the lower end of the smoking region, the left end of the mouth region is the left end of the smoking region, and the mouth The right end of the smoking area is the right end of the smoking area. Thereby, a rectangular smoking area is set.

Note that the size of the face image changes depending on the positional relationship between the camera 9 and the face. Therefore, the smoking area is appropriately enlarged or enlarged so that the size of the smoking area becomes constant according to the change in the size of the face image. to shrink.

In addition, this smoking area is an area where it is determined that there is a high possibility of smoking cigarettes based on data obtained through experiments, etc. Can be reduced. Separately from this, for example, a smoking region can be set as appropriate, such as a polygonal or circular region centered on the center of the mouth (for example, the center of the area).

Next, for example, as shown in FIG. 6, the smoking area is divided into a large number of small areas (array elements) represented by a matrix of vertical M × horizontal N, and the luminance value in the area is stored. For example, the average value of the luminance values of each pixel in the small area is stored. Thereby, the smoking area can be expressed as a luminance distribution (luminance distribution) as shown in FIG. Note that the luminance distribution may be created by obtaining the luminance value for each pixel.

In addition, since data reliability is low only with an image in a single frame, data of array elements in a plurality of frames is stored here.
For example, the luminance distribution of FIG. 6 obtained for each frame is added for each array element, and a necessary number of data is accumulated. For example, when data for 30 frames is accumulated, it is assumed that a sufficient number of data is obtained.

And the image of the smoking area | region when not smoking as shown to the said Fig.5 (a) is memorize | stored as registration brightness distribution (basic image), The present frame which actually detects smoking with the basic image The image of the smoking area in (see FIG. 5B) is compared.

Specifically, the difference between the registered luminance distribution and the luminance distribution of the current frame is obtained. Accordingly, when the cigarette is added, the image of the cigarette portion is different, and thus an image (difference image) as shown in FIG. 5C is obtained, for example.
Therefore, it can be determined from this difference image whether or not tobacco is added. That is, the smoking state can be estimated.

Further, as shown in FIG. 7, the difference in luminance between the leading end portion of the tobacco (one end portion in the axial direction of the tobacco) and the other portion (base end portion) is obtained. For example, a difference in average luminance between the front end portion and the base end portion is obtained, and when the difference in luminance is equal to or greater than a predetermined value, it can be determined that the front end of the cigarette is lit.

In other words, when the cigarette is lit, the brightness of the tip of the cigarette is greatly increased compared to the brightness of the other parts, so the presence or absence of smoking can be determined from the difference in brightness.

Alternatively, when the cigarette is lit, its brightness is very high, so the presence or absence of smoking may be determined simply from the brightness of the tip.

In addition, what is necessary is just to classify one end part in the axial direction of a tobacco by the predetermined distance from a front end for the detection of the front-end | tip part of a tobacco. For this distance, an area where the brightness increases due to fire may be obtained by experiments or the like.

Here, it is possible to determine which side of the axial direction is the tip portion, whichever is the tip portion, but smoking can be determined. For example, it is desirable to set the tip portion farther from the center of the mouth.

D) Next, the processing method implemented by the smoking detection apparatus will be described with reference to FIGS.

<Brightness distribution registration routine>
This routine shows a process for registering the luminance distribution of the smoking area as shown in FIG. 6 on the basis of the face image in a state where the user does not smoke in advance.

As shown in FIG. 8, first, in step (S) 100, a face image is acquired as described above based on the image photographed by the camera 9.

In the following step 110, feature points are obtained from the face image by the AAM technique, and a mask as shown in FIG.

In the following step 120, it is determined whether or not the mask valid flag is set (is set). If an affirmative determination is made here, the process proceeds to step 130. If a negative determination is made, the process returns to step 100, and the same processing is repeated.

In step 130, since the mask is suitably formed, a smoking area including the mouth and nose in the mask is set, and a luminance distribution of the smoking area as shown in FIG. 6 is created.

In the subsequent step 140, the produced luminance distribution is accumulated. That is, the luminance distribution of the frame is added for each array element.

In the following step 150, it is determined whether or not the necessary data (for example, 30 frames) has been accumulated. If an affirmative determination is made here, the process proceeds to step 160, whereas if a negative determination is made, the process returns to step 100 and the same processing is repeated.

In step 160, since the necessary amount of data has been collected, it is registered as a luminance distribution in the smoking area including the mouth and nose at normal times (when not smoking).

In the subsequent step 170, the process proceeds to a smoking detection routine described later, and the present process is temporarily terminated.

<Smoking detection routine>
This routine shows a process of detecting a smoking state based on a photographed driver's face image.

When actually detecting the smoking state of the driver, as shown in step 200 of FIG. 9, first, the driver's face is photographed and the face image is acquired.

In the subsequent step 210, feature points are obtained from the face image by the AAM technique, and a mask as shown in FIG. 4A is produced.

In the following step 220, it is determined whether or not the mask valid flag is set. If an affirmative determination is made here, the process proceeds to step 230. On the other hand, if a negative determination is made, the process returns to step 200 and the same processing is repeated.

In step 230, a smoking area including the mouth and nose in the mask is set, and a luminance distribution of the smoking area is created.

In the following step 240, the difference between the luminance distribution in the current frame thus created and the registered luminance distribution created in the luminance distribution production routine is calculated.

In this case, in order to reliably detect cigarettes, an area having a luminance difference of a certain level or more is extracted as a cigarette image.

In the subsequent step 250, it is determined whether or not the shape of the cigarette as shown in FIG. For example, it is determined whether or not a long image (a long rectangular image) having a predetermined length and width corresponding to an actual cigarette has been extracted.

If an affirmative determination is made here, the process proceeds to step 260 as “possible smoking state”, whereas if a negative determination is made, the process returns to step 200 as “not smoking state”, and the same processing is repeated.

In step 260, since there is a high possibility of being in a smoking state, the tip of the cigarette (which is likely to be on fire) is detected for confirmation. That is, the brightness of an area within a predetermined range from the tip of the cigarette (for example, the average value of the luminance of pixels within the range) is detected.

In the following step 270, it is determined whether or not the tip of the cigarette is shining strongly. For example, when the brightness of the tip of the cigarette is greater than or equal to a predetermined determination value, or when the difference in brightness (average value) between the tip of the cigarette and a portion other than the tip is greater than or equal to a predetermined value , the tip of the cigarette Is determined to be shining strongly.

If an affirmative determination is made here, the process proceeds to step 280 as “smoking state”, whereas if a negative determination is made, the process returns to step 200 to confirm again, and the same processing is repeated.

In step 280, it is determined that the person is in a smoking state, and for example, a smoking confirmation flag is set.

In subsequent steps 290, 300, and 310, each process based on the determination that the person is in a smoking state is performed, and the processes described above are repeated.

In addition, what is necessary is just to perform at least 1 sort for the process of each step 290, 300, 310, The order can also be selected suitably.

Specifically, in step 290, since smoking is in progress, the air purifier 19 is activated to purify the air in the vehicle.

In step 300, the speaker 15 is driven to output an alarm such as “Let's be careful about health”.

In step 310, in order to release smoke outside the vehicle, the power window 17 is driven to control the window to be opened a little.

In addition to this, for example, the air conditioner 13 may be controlled to improve air circulation and promote air purification.

e) As described above, in the present embodiment, a human face image can be taken using a device such as the camera 9 used in the normal driver monitor system 1 and cigarettes can be detected from the face image. Such a special device is not necessary, and the device can be simplified and the cost can be reduced.

In addition, if a smoking area is set in the vicinity of the mouth or nose, for example, where it is expected that there is a cigarette currently being smoked, and if tobacco can be detected in that area, it can be estimated as a smoking state, so the determination accuracy is high .

Furthermore, in the case of a smoking state, it can be estimated that cigarettes exist around the mouth and nose, but in this embodiment, a smoking area around the mouth and nose where there is a high possibility of smoking cigarettes. Thus, it is possible to make a highly accurate smoking determination.

Also, when comparing an image of a person smoking with an image of no smoking, the image of smoking should have a cigarette image. Therefore, in the present embodiment, an image that is not smoked is obtained in advance, and an image of the cigarette is extracted (if smoked) by comparing the image with an image for which smoking is determined. it can. Thereby, a cigarette can be detected with high accuracy.

In particular, in the present embodiment, when photographing is performed by irradiating near-infrared light, an image in which the white portion of the cigarette is bright is obtained. Therefore, extraction of tobacco can be performed easily.

Furthermore, when the cigarette is lit, the tip of the cigarette shines brightly. Therefore, in this embodiment, the brightness | luminance of the front-end | tip part of this cigarette can be detected, and a smoking state can be determined more correctly based on the brightness | luminance.

The smoking area setting unit (means) corresponds to step 230, the cigarette detection unit (means) corresponds to step 240, and the smoking estimation unit (means) corresponds to steps 250 to 270, respectively.

The present invention is not limited to the above-described embodiment, and can be implemented in various modes without departing from the gist of the present invention.

For example, a computer program that performs each process as shown in FIGS. 8 and 9 of the above-described smoking detection apparatus is also within the scope of the present invention. That is, the function of the smoking detection device described above can be realized by processing executed by a computer program. This program can be executed by being recorded on a computer-readable recording medium 8 and loaded into the control unit 7 (computer) and started as necessary.

The following summarizes the gist of the embodiment described above.

(1) As one aspect of the embodiment, there is provided a smoking detection device that detects a smoking state of a person based on image information from an imaging unit that photographs the person. A smoking detection device detects a cigarette from an image in the smoking area, and a smoking area setting unit that sets a smoking area where a smoking cigarette is expected to exist based on a face image obtained by photographing the person's face And a smoking estimation unit that estimates that the cigarette is in the smoking state when the cigarette is present in the smoking area.

In this smoking detection device, a smoking area where smoking cigarettes are expected to exist is set based on a face image obtained by photographing a human face, cigarettes are detected from the image in the smoking area, and the smoking area is detected. If tobacco is present, it is assumed that it is a smoking state.

That is, in this smoking detection device, a human face image can be taken using a device such as a camera used in a normal driver monitor, and cigarettes can be detected from the face image. Therefore, a special device such as an infrared camera is not necessary, and the device can be simplified and the cost can be reduced.

In addition, if a smoking area is set in the vicinity of the mouth or nose, for example, where it is expected that there is a cigarette currently being smoked, and if tobacco can be detected in that area, it can be estimated as a smoking state, so the determination accuracy is high There is an advantage.

In addition, here, when tobacco exists in the smoking area, it is estimated that the tobacco is in a smoking state because it is highly likely that the tobacco is on fire. As will be described later, a more accurate determination can be made by taking into account the determination of whether or not the fire is actually on.

(2) In another aspect of the embodiment, the smoking area includes at least one of a mouth area and a nose area in an area smaller than the entire area of the face image.

This embodiment illustrates a preferred smoking area. Usually, in the case of smoking, it can be estimated that tobacco exists around the mouth and nose. Therefore, by setting a smoking area around the mouth and nose where there is a high possibility that there is a cigarette during smoking, a highly accurate smoking determination can be made.

(3) In another aspect of the embodiment, the cigarette detection unit compares the image in the smoking area with a basic image that does not include a preset cigarette, and extracts the cigarette image. , Detecting the tobacco.

When comparing an image of a person smoking with an image of no smoking, the image of smoking should have a cigarette image. Therefore, by obtaining a basic image that has not been smoked in advance and comparing the basic image with an image for which smoking determination is performed, an image of a cigarette can be easily extracted (if smoked). . Thereby, a cigarette can be detected.

In addition, since the color of tobacco is usually white and is different from the face color (luminance), extraction of tobacco is easy.

(4) In another aspect of the embodiment, the smoking estimation unit obtains luminance at the tip of the cigarette based on the extracted image of the tobacco, and based on the luminance of the tip, the smoking state Make a decision.

When the cigarette is lit, the tip of the cigarette shines brightly (for example, other than the tip). Therefore, based on the brightness | luminance of the front-end | tip part of this tobacco, a smoking state can be determined correctly.

(5) In another aspect of the embodiment, the smoking estimation unit determines the smoking state when the luminance at the tip of the cigarette is equal to or higher than a predetermined value.

Here, when the brightness at the tip of the cigarette is larger than the value indicating the state in which the cigarette is lit, the cigarette is determined to be a smoking state, so that a precise smoking determination can be made.

(6) In another aspect of the embodiment, the smoking estimation unit determines the smoking state when the luminance at the tip of the cigarette is greater than a predetermined value by the luminance other than the tip.

Here, since the smoking state is determined when the luminance at the tip of the cigarette is higher than the luminance of the other portions, it is possible to make a highly accurate smoking determination.

(7) In another aspect of the embodiment, the imaging by the imaging unit is performed by irradiating near infrared light.

When an object irradiated with near-infrared light is photographed with a camera (for example, a CCD camera) used in an ordinary driver monitor, a white portion appears brighter (compared to photographing with visible light). Since cigarettes are usually white, it is possible to detect cigarettes with high accuracy by photographing with near-infrared light.

(8) Another aspect of the embodiment is a program for causing a computer to function as the smoking area setting unit, the cigarette detection unit, and the smoking estimation unit.

That is, the smoking area setting unit, the cigarette detection unit, and the smoking estimation unit can be realized by processing executed by a computer program.

Such a program can be used by, for example, recording it on a computer-readable recording medium such as FD, MO, DVD-ROM, CD-ROM, hard disk, etc., and loading and starting the computer as necessary. . In addition, the ROM or backup RAM may be recorded as a computer-readable recording medium, and the ROM or backup RAM may be incorporated into a computer and used.

DESCRIPTION OF SYMBOLS 1 ... Driver monitor system 5 ... Imaging part 7 ... Control part 9, 9a, 9b ... Camera 11 ... Floodlight

Claims

Based on image information from an imaging unit that photographs a person, a smoking detection device that detects the smoking state of the person,
Based on a face image obtained by photographing the person's face, a smoking area setting means for setting a smoking area where a cigarette during smoking is expected to exist,
Tobacco detection means for detecting the cigarette from an image in the smoking area;
When the cigarette is present in the smoking area, smoking estimation means for estimating the smoking state;
A smoking detection device comprising:
The smoking detection apparatus according to claim 1, wherein the smoking area includes at least one of a mouth area and a nose area in an area smaller than the entire area of the face image.
The cigarette detection means detects the cigarette by extracting an image of the cigarette by comparing an image in the smoking area with a basic image not including a preset cigarette. Item 3. The smoking detection device according to Item 1 or 2.
The smoking estimating means obtains the luminance at the tip of the cigarette based on the extracted image of the cigarette and determines the smoking state based on the luminance of the tip. The smoking detection device according to 1.
The smoking detecting device according to claim 4, wherein the smoking estimating means determines that the smoking state is present when the luminance at the tip of the cigarette is equal to or higher than a predetermined value.
The smoking detection device according to claim 4, wherein the smoking estimation means determines that the smoking state is present when the luminance at the tip of the cigarette is greater than a predetermined value by more than a luminance other than the tip.
The smoking detection apparatus according to any one of claims 1 to 6, wherein the imaging by the imaging unit is performed by irradiating near infrared light.
A program for causing a computer to function as the smoking area setting unit, the cigarette detection unit, and the smoking estimation unit according to claim 1.
A smoking detection method for detecting a smoking state of the person based on image information from an imaging unit that photographs the person,
Based on the face image obtained by photographing the person's face, set a smoking area where smoking tobacco is expected to exist,
From the image in the smoking area, detect the cigarette,
A smoking detection method, wherein the smoking state is estimated when the cigarette is present in the smoking area.