JP2019191736A - Determination device, determination method and program - Google Patents

Abstract

To accurately distinguish between a left hand and a right hand.SOLUTION: A determination device includes: person extraction means which extracts a region of a person in a captured image; hand extraction means which extracts regions of his or her hands on the basis of the region of the person; orientation specifying means which specifies his or her orientation in accordance with a physical relation between regions other than the regions of his or her hands and the regions of his or her hands; and left/right hand determination means which distinguishes between the region of his or her left hand and the region of his or her right hand on the basis of his or her orientation.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a determination device, a determination method, and a program.

  Conventionally, in various scenes, a method for extracting a hand portion of a person in an image and determining whether the hand is left or right, and a method for performing various processes based on the result have been proposed. Patent Document 1 discloses a technique for determining which hand is left or right based on the arrangement of fingers of a hand in an image and controlling the operation of the in-vehicle device based on the result. Patent Document 2 discloses a technique for estimating the operator's elbow position from the difference between the background image and the captured image and determining the right and left of the hand based on the position assuming that the person is in the center of the image. Yes. Patent Document 3 discloses a technique for preparing left and right hand template images and determining which hand is left or right by template matching in a captured image.

JP 2008-221960 A JP 2006-11965 A International Publication 2011/158511

  However, in the prior art, there is a problem that the right and left hands cannot be correctly determined in some situations.

  The present invention has been made in view of such problems, and an object thereof is to accurately determine the left and right hands.

  Therefore, the present invention is a determination apparatus, which is a person extraction unit that extracts a person region in a captured image, a hand extraction unit that extracts a hand region based on the person region, and the person other than the hand region. An orientation specifying means for specifying the orientation of a person based on a positional relationship between the area and the hand area; and a left / right determination means for determining left and right of the hand area based on the orientation of the person. To do.

  According to the present invention, it is possible to accurately determine the left and right hands.

1 is an overall view of an imaging system according to a first embodiment. It is a hardware block diagram of a server apparatus. It is a functional block diagram of a server apparatus. It is a flowchart which shows a determination process. It is explanatory drawing of the process of S409-S411. It is a figure which shows the mode of work. It is a figure which shows the graph which shows the relationship between head-to-hand distance and weight. It is explanatory drawing of a division | segmentation process. It is a flowchart which shows a determination process. It is a figure which shows an example of a presence rate. It is a functional block diagram of a server apparatus. It is a flowchart which shows a determination process. It is a figure which shows the mode of work. It is a figure which shows likelihood and weight. It is explanatory drawing of work determination processing.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is an overall view of an imaging system according to the first embodiment. The imaging system includes a server device 100 and an imaging device 110. The server device 100 is an example of a determination device that extracts a worker's hand region from the image (video) acquired by the imaging device 110 and determines the left and right hands. In the present embodiment, it is assumed that the imaging device 110 captures a situation where the worker P is performing an operation such as assembling the object Q as shown in FIG. It is assumed that the imaging device 110 is set above the worker P.

  FIG. 2 is a hardware configuration diagram of the server apparatus 100. The CPU 201 executes processing using computer programs and data stored in the RAM 206 and ROM 207. As a result, the CPU 201 controls the operation of the entire server apparatus 100 and executes or controls each process described later as what the server apparatus 100 performs. The input device 202 is a user interface such as a button group, and various inputs can be made to the server device 100 by a user operation. The output device 203 is configured by a liquid crystal screen, a speaker, and the like, and can display a processing result by the CPU 201 and output sound. The touch panel screen may be configured by combining the user interface function by the input device 202 and the display function by the output device 203.

  The communication device 204 includes an interface such as a USB port and a LAN port for receiving input / output of data from the outside of the server device 100. An imaging device 110 for acquiring video data, a medium for recording video data, a medium for recording work time measured by this apparatus, and the like are connected to the communication device 204, and input / output of processing target data is performed. Is called.

  The storage device 205 is configured by a hard disk drive device, a flash memory, or the like. The storage device 205 stores an OS (operating system) and computer programs and data for causing the CPU 201 to execute or control each process described below as performed by the server device 100. In the following description, the registration destination of the video data used for processing and the measured work time data is the storage device 205. Computer programs and data stored in the storage device 205 are appropriately loaded into the RAM 206 under the control of the CPU 201 and are processed by the CPU 201. Note that the functions and processes of the server apparatus 100 described later are realized by the CPU 201 reading a program stored in the ROM 207 or the storage device 205 and executing the program.

  The RAM 206 has an area for storing computer programs and data loaded from the storage device 205. Further, the RAM 206 has a work area used when the CPU 201 executes various processes. As described above, the RAM 206 can provide various areas as appropriate. The ROM 207 stores computer programs and data that do not need to be rewritten, such as setting data and startup programs for the server device 100. The CPU 201, the input device 202, the output device 203, the communication device 204, the storage device 205, the RAM 206, and the ROM 207 are all connected to a bus (BUS) 208.

  FIG. 3 is a functional configuration diagram of the server apparatus 100. The person extraction unit 301 extracts a person existing in the video. The hand extraction unit 302 extracts one or two hand regions from the person region extracted by the person extraction unit 301. For the extraction of the hand area, color information is used together with distance information. The left / right determination unit 303 determines whether the hand region extracted by the hand extraction unit 302 is the left or right hand region.

  FIG. 4 is a flowchart illustrating the determination process performed by the server apparatus 100. First, in step S <b> 401, the CPU 201 acquires a frame image of a moving image acquired by the imaging device 110 through the communication device 204. Here, the CPU 201 acquires a color image and a distance image. Note that shooting by the imaging device 110 is performed according to shooting parameters stored in the storage device 205. This moving image is sequentially processed in a streaming format. As another example, the CPU 201 may record the acquired moving image in the storage device 205 and perform processing at regular intervals.

  Next, in S402, the CPU 201 determines whether or not the frame image acquired in S401 is the last frame. In the case of the last frame (YES in S402), the CPU 201 ends the determination process because there is no further processing target. If it is not the last frame (NO in S402), the CPU 201 advances the process to S403.

  In step S403, the person extraction unit 301 extracts a person area from the frame image. The person extraction unit 301 performs processing using both the color image and the distance image acquired in S401. Specifically, the person extraction unit 301 first takes a difference between a background image without a person prepared in advance and a frame image, and roughly takes a person area. Next, the person extraction unit 301 refers to the color information and the size information of the extracted area, and extracts an area excluding an area that is clearly not a person, such as an extremely small size, as a person.

  Next, in S <b> 404, the hand extraction unit 302 extracts a hand region for the person region extracted in S <b> 403. Specifically, the hand extraction unit 302 assumes that there is no image in the imaging range that is higher than the person, and the highest pixel in the person region extracted in S403, that is, the pixel having the smallest distance value and the surrounding area. Estimate the region as the head. Next, the hand extraction unit 302 deletes the vicinity of the region estimated as the head, and calculates the planar distance between the remaining region and the head region. In the shooting environment shown in FIG. 1, it is considered that the hand area is the farthest plane distance from the head during work, so the hand extraction unit 302 has the first plane distance among the remaining areas. The second most distant region is estimated as a hand region and extracted.

  Next, in S405, the left / right determination unit 303 determines whether two hand regions have been extracted in S404. During work, one hand or both hands may not appear in the image behind the surroundings or the worker itself. In such a case, since it is difficult to determine the left and right of the hand, the processing is divided at this step. If two hand areas are extracted (YES in S405), the left / right determination unit 303 advances the process to S406. If only one hand region is extracted (NO in S405, YES in S413), the left / right determination unit 303 advances the process to S414. If the hand region is not extracted (NO in S405, NO in S413), the left / right determination unit 303 ends the determination process.

  In step S <b> 406, the left / right determination unit 303 determines whether left / right of the hand has already been determined in the previous frame. If processing is performed in units of frames, it is unlikely that the position of the hand will change greatly between frames. Therefore, it is considered that the determination accuracy is higher and the calculation cost is lower when the previous frame information is reflected as it is than when the left and right determinations are newly made. If the left / right determination unit 303 has determined (YES in S406), the process proceeds to S407. If the left / right determination unit 303 cannot determine (NO in S406), the process proceeds to S409. In S407, the left / right determination unit 303 reflects the left / right determination information of the previous frame in the hand area of the current frame that appears in the vicinity of the positions of the left and right hands of the previous frame. The left / right determination unit 303 then advances the process to S408. In step S408, the CPU 201 changes the processing target to the next frame, and then advances the processing to step S401. The process of S407 is an example of an estimation process for estimating the left and right hands of the second captured image based on the determination result of the left and right hands in the first captured image included in the series of captured images.

  On the other hand, in S409, the left / right determination unit 303 first extracts a head region. This process is an example of a head extraction process. Then, the left / right determination unit 303 obtains a head-to-hand distance between the head and the hand. Next, in S410, the left / right determination unit 303 compares the inter-hand distance with a threshold value. Here, the head-to-hand distance is set in advance according to the user operation. If the head-to-hand distance is greater than the threshold (YES in S410), the left / right determination unit 303 advances the process to S411. If the head-to-hand distance is equal to or smaller than the threshold (NO in S410), the left / right determination unit 303 advances the process to S408. In step S <b> 411, the left / right determination unit 303 identifies the direction of the person with the hand direction as the forward direction. Next, in S412, the left / right determination unit 303 determines that the left hand region side is the left hand and the right hand region side is the right hand with reference to the front direction. The left / right determination unit 303 then advances the process to S408.

  FIG. 5 is an explanatory diagram of the processing of S409 to S411. The frame image 500 is a photographed image showing a worker performing a work reaching out forward. On the other hand, the frame image 510 is a photographed image of the worker stretching out. The left / right determination unit 303 obtains the bounding box 504 of the worker's head 501, the worker's right hand 502 and the left hand 503 in the frame image 500, and specifies the position of the bounding box 504 as the hand position. Then, a head-to-hand distance 505 between the hand position and the head position is calculated. The left / right determination unit 303 calculates the inter-hand distance 513 between the operator's head 511 and the bounding box 512 by the same process for the frame image 510.

  When humans are doing normal work, the front of the body has a wider range of movement of the arm than the back. By using this, in the server device 100 of the present embodiment, the threshold value of the head-to-hand distance is set as described above, and when the head-to-hand distance is longer than the threshold value, the hand direction is specified as the forward direction. It was. That is, the left / right determination unit 303 specifies the direction of the hand as the front direction when the head-to-hand distance is greater than the threshold value. As another example, the left / right determination unit 303 may estimate the direction opposite to the hand direction as the forward direction when the head-to-hand distance is equal to or less than the threshold. Note that the processes of S409 to S411 are an example of a direction specifying process for specifying the direction of a person based on the positional relationship between the head area as a person area other than the hand area and the hand area.

  Returning to FIG. 4, in S414, the left / right determination unit 303 determines whether the left / right of the hand has already been determined in the previous frame. This process is the same as the process of S406. If the left / right determination unit 303 has determined (YES in S414), the process proceeds to S415. If the left / right determination unit 303 cannot determine (NO in S414), the process proceeds to S408. In step S415, the left / right determination unit 303 reflects the left / right determination information of the previous frame in the hand area of the current frame that appears in the vicinity of the positions of the left and right hands of the previous frame. This process is the same as the process of S407.

  As described above, in the determination process, the direction in which the worker is facing can be estimated using the positional relationship between the worker's head and the hand region. Thereby, it is possible to determine the left and right of the hand area of the worker. As shown in FIG. 6, the worker P may not only work on the object 601 on the work table 600 but also move to other places 610 and 620 in the video. By moving the work place in this way, the server device 100 according to the present embodiment can accurately determine the left and right hands even when the position and orientation of the worker P change. Further, even when the worker P is facing the back while being positioned in front of the work table 600, the left and right hands are determined after determining the orientation of the worker P, so the left and right hands are accurately determined. be able to.

  As a first modification of the first embodiment, the server device 100 obtains not only the direction determination result but also the likelihood, and based on this likelihood, obtains the likelihood also for the left and right determination results. Also good. This process is an example of the likelihood specifying process.

  As a second modification, the server device 100 may calculate a weight according to the head-to-hand distance, and obtain the likelihood of the left and right determination results according to the weight. The graph shown in FIG. 7 shows the relationship between the head-to-hand distance and the weight. Note that the graph shown in FIG. 9 is a model of a quadratic function, but it is sufficient that the weight increases as the head-to-hand distance increases, and is limited to the relationship in the graph of FIG. is not. As another example, a relationship represented by, for example, a linear function or a step function may be used.

  As a third modified example, the threshold value to be compared with the head-to-hand distance in S410 may be determined by the server device 100 based on the size of the person area. For example, the length of the operator's arm and the size of the head are measured in advance and registered in the server apparatus 100. Then, the server apparatus 100 calculates a range where the arm reaches on the image from the size of the head shown in the image, and determines a threshold based on the calculated range. Furthermore, the server apparatus 100 may use an accumulated value of estimation of a plurality of frames. Thereby, it is possible to prevent a mistake caused by estimation using only one frame. This process is an example of a threshold determination process.

  As a fourth modified example, the server apparatus 100 may detect the left and right hands only at a specific timing, such as a timing at which a specific operation that is likely to cause a false detection is performed.

(Second Embodiment)
Next, the imaging system according to the second embodiment will be described mainly with respect to differences from the imaging system according to the first embodiment. The server device 100 according to the second embodiment performs left / right hand determination based on temporal statistics of the position where the hand region exists instead of the head-to-hand distance. As shown in FIG. 8, the left / right determination unit 303 of the server device 100 has four areas A, B, C, and D in the area around the worker 801 centered on the center position of the worker 801 in the frame image 800. Divide into Note that the left / right determination unit 303 may divide the area with the worker 801 as a reference, and the number is not limited to the embodiment.

  FIG. 9 is a flowchart illustrating the determination process performed by the server apparatus 100 according to the second embodiment. The processing of S901 to S904 is the same as the processing of S401 to S404 described with reference to FIG. After the process of S904, in S905, the left / right determination unit 303 identifies which of the four areas 811 to 814 the hand area belongs to by division. In step S <b> 906, the left / right determination unit 303 determines whether a process for specifying an area to which the frame belongs is performed for a certain number of frames. The left / right determination unit 303 advances the process to S907 when the process for a certain number of frames is completed (YES in S906). The left / right determination unit 303 advances the process to S910 when the process for a certain number of frames has not been completed (NO in S906). In step S910, the CPU 201 changes the processing target to the next frame, and then advances the processing to step S901.

  In step S907, the left / right determination unit 303 performs totalization of regions to which a certain number of frames belong, and calculates the existence ratio in each region. Next, in S908, the left / right determination unit 303 identifies two regions having a high presence ratio as directions in front of the person. Here, the process of S907 is an example of a measurement process for measuring the time during which the hand is present in the region, and the process of S908 is an example of a direction specifying process for specifying the orientation of a person based on the measurement result. Next, in S909, the left / right determination unit 303 determines that the left hand region is the left hand and the right hand region is the right hand with reference to the front direction. This process is the same as the process of S412. CPU201 advances post-processing of processing of S909 to S910. FIG. 10 is a diagram illustrating an example of the existence ratio. In the counting result shown in FIG. 10, the directions of regions C and D are specified as the previous region.

  When humans are doing normal work, it is considered that it takes a long time to do some work in front of the body. The server apparatus 100 according to the present embodiment uses this fact to estimate the orientation of the subject person from the total information. In addition, since the left / right determination is not performed until a certain time has elapsed, there is a high possibility that the estimation will fail when it is considered that there is no difference in the existence time of each region, such as at the beginning of processing. The other configurations and processes of the imaging system according to the second embodiment are the same as the configurations and processes of the imaging system according to the first embodiment.

  As described above, in the imaging system according to the second embodiment, it is possible to add up temporal ratios of regions where hand regions exist from a moving image, and to determine left and right hands from the result of counting.

  As a first modification of the second embodiment, the server device 100 may further determine the likelihood based on the existence ratio.

  As a second modification, when the server apparatus 100 does not move due to some reason, the operation is interrupted or stopped, or the movement is not related to the operation, the information is manually stored. Alternatively, it may be input from an external sensor or the like, and aggregation may be interrupted during that time.

(Third embodiment)
Next, the imaging system according to the third embodiment will be described mainly with respect to differences from the imaging system according to the first embodiment. The server device 100 according to the third embodiment performs left / right hand determination using the relationship between an object and a hand used in the work.

  FIG. 11 is a functional configuration diagram of the server apparatus 100 according to the third embodiment. The server device 100 includes an object specifying unit 1101 and a contact determination unit 1102 in addition to the functional configuration of the server device 100 according to the first embodiment. The object specifying unit 1101 specifies the position of an object used for work with a predetermined object in the frame image. Hereinafter, this object is referred to as a specific object. An example of the specific object is a driver. An example of a method for specifying the position of a specific object is template matching. Other examples include techniques using local feature quantities and machine learning. As another example, information that can identify a marker or the like may be attached to a specific object, and the information may be used. The contact determination unit 1102 determines contact between the specific object and the hand region based on the frame image.

  FIG. 12 is a flowchart illustrating the determination process performed by the server apparatus 100 according to the third embodiment. The processing of S1201 to S1204 is the same as the processing of S401 to S404 described with reference to FIG. After the process of S1204, in S1205, the object specifying unit 1101 specifies the position of the specific object in the frame image. This process is an example of an object extraction process for extracting an object region. In step S1206, the contact determination unit 1102 determines whether the hand is in contact with the specific object. As illustrated in FIG. 13, when the operator P picks up the driver 1300 as the specific object, it is determined that the person P is in contact. Specifically, the contact determination unit 1102 determines that both are in contact when the hand region and the specific object region overlap or are adjacent to each other.

  If the contact determination unit 1102 is in contact (YES in S1206), the process proceeds to S1207. If the contact determination unit 1102 is not in contact (NO in S1206), the process proceeds to S1209. In step S1209, the CPU 201 changes the processing target to the next frame, and then advances the processing to step S901.

  In step S1207, the left / right determination unit 303 specifies the direction in which the contact is specified as the front direction of the person. This process is an example of a direction specifying process that specifies the object region side as the forward direction of the person. Next, in S1208, the left / right determination unit 303 determines that the left hand region is the left hand and the right hand region is the right hand with reference to the front direction. This process is the same as the process of S412. The CPU 201 advances the post-processing of S1208 to S1209.

  When a human is doing normal work, he or she faces the tool when acquiring the tool to use for work. At least, it's very unlikely to reach out without looking back at what's behind it. The server apparatus 100 according to the present embodiment uses this fact to estimate the direction in which the contact of the specific object is specified as the direction of the subject person. The other configurations and processes of the imaging system according to the third embodiment are the same as the configurations and processes of the imaging system according to the other embodiments.

  As described above, in the imaging system according to the third embodiment, a specific object can be extracted from a moving image, and left and right hands can be determined based on a region in which contact between the specific object and the hand is specified.

  As a first modification of the third embodiment, the server device 100 may prepare a plurality of specific objects and perform left and right hand determinations based on determination results for the plurality of specific objects. A correct estimation result may not be obtained with only one estimation. For example, if an object placed almost beside the worker is targeted, there is a high possibility that an erroneous determination result is obtained. On the other hand, according to this example, the determination can be performed more accurately.

  As a second modified example, the server device 100 may count how many times each contact opportunity has been determined, and obtain the likelihood of the determination result of the left and right hands from the determination success rate.

(Fourth embodiment)
Next, an imaging system according to the fourth embodiment will be described mainly with respect to differences from the imaging system according to other embodiments. The server apparatus 100 of this embodiment calculates | requires a final determination result from the determination result of the right and left hand demonstrated in the 1st-3rd embodiment. The server apparatus 100 further weights the estimation results of each method to obtain a final estimation result. For example, as shown in a table 1400 of FIG. 14, the server apparatus 100 multiplies the estimation result of whether or not the right hand is output for each area by a weight set in advance and obtains a final result. be able to.

  In addition, the server apparatus 100 shall set the weight of each method according to user operation. As another example, the server apparatus 100 may automatically calculate the weight based on the estimated success rate so far, for example. In addition, the estimation result of each method may be a likelihood as shown in Table 1400, or may be a binary output. Furthermore, the likelihood and the binary value may be mixed for each method. The configuration and processing of the imaging system according to the fourth embodiment are the same as the configuration and processing of the imaging system according to other embodiments.

  As described above, in the imaging system according to the fourth embodiment, the left and right hands can be more accurately determined by integrating the determination results obtained by a plurality of methods.

(Fifth embodiment)
Next, an imaging system according to the fifth embodiment will be described mainly regarding differences from the imaging system according to other embodiments. The server apparatus 100 according to the present embodiment determines the left and right hands by the method described in the other embodiments, and determines whether the worker is performing a predetermined work (specific work) using the determination result. To do.

  FIG. 15 is an explanatory diagram of the work determination process. The server device 100 detects that the worker P has performed work determined by whether or not the worker P has touched the area 1501. For example, the operator's hand touches the area 1501 when a part determined by the operator is attached to a part 1502 near the area 1501. Using this, it is detected whether or not work has been performed. However, there is a possibility of touching the area 1501 when performing another work. For example, as shown in FIG. 15, the area 1501 may be touched by an operation different from the assumed specific operation of taking a driver 1503.

  Therefore, the server apparatus 100 sets in advance whether the hand to touch the area 1501 is left or right. Then, the server device 100 uses the left / right determination result at the time of the contact determination, and determines which of the left and right hands is in contact with the specific area 1501 to perform the work determination more accurately. Can do. This process is an example of a contact determination process.

  For example, it is assumed that the server apparatus 100 is determined to have worked in the area 1501, for example, when the right hand touches it. In this case, as shown in FIG. 15, when the left hand touches the area 1501, it is not determined that the work has been performed. In this way, accurate determination can be made.

  Thus, the server apparatus 100 can also set the granularity of the work sequence to be correct by using the result of the left / right determination. For example, consider a case where a plurality of areas are set in order to detect a plurality of operations using whether or not the hand region has touched. In this case, if the left and right of the area cannot be distinguished, the area setting position and the correct answer sequence may be restricted so that the left hand touches the area that should be touched with the right hand as described above, and is not erroneously detected. However, if the left and right of the hand region can be distinguished, it is possible to set whether to detect when the right sequence is touched with the right or left hand, so that a finer sequence can be created. As a result, more work can be detected more accurately.

  The server apparatus 100 can also register the dominant hand of the worker in advance and change the correct answer sequence according to the information. For example, it is assumed that a dominant hand handles a tool such as a driver. Therefore, the server apparatus 100 determines one hand based on the dominant hand information, and sets the determined hand as a detection target. This process is an example of a left / right determination process. The server device 100 detects when the right hand is used to handle the tool in the right sequence created for right-handed use, and detects when the left hand is used to handle the tool in the correct answer sequence created for left-handed use. To do.

  In addition, for a while after starting the process, there is a possibility that sufficient information cannot be obtained to perform the left / right determination, and the determination cannot be performed. In that case, the server apparatus 100 temporarily assumes work detection by assuming both a right hand region and a left hand region. When the left and right estimation result likelihood exceeds the threshold, the server device 100 cannot determine the left and right of the hand by propagating the information in the reverse direction of the time series and leaving only the correct work detection result. Processing can also be performed for timing. Similarly, when the estimated likelihood of the left and right hands is less than the threshold value, the server apparatus 100 can re-evaluate using the result at the time when the hand likelihood becomes equal to or greater than the threshold value.

  As described above, according to the present embodiment, the detection accuracy of the work is improved by using the extraction of the worker's hand region and the left and right estimation results of the region from the moving image of the worker. be able to. Thereby, a finer work and various work can be detected.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the present invention described in the claims.・ Change is possible.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 100 Server apparatus 110 Imaging device 301 Person extraction part 302 Hand extraction part 303 Left-right determination part

Claims (14)

A person extracting means for extracting a person area in the photographed image;
Hand extraction means for extracting a hand region based on the person region;
From the positional relationship between the person area other than the hand area and the hand area, direction specifying means for specifying the direction of the person,
A determination apparatus comprising: a left / right determination unit configured to determine the right / left of the hand region based on the orientation of the person. Further comprising a head extraction means for extracting a head region based on the person region;
The determination apparatus according to claim 1, wherein the direction specifying unit specifies a direction of a person based on a positional relationship between the head region and the hand region.   The determination apparatus according to claim 2, wherein the direction specifying unit specifies a direction of a person based on a distance between the head region and the hand region.   The determination apparatus according to claim 3, wherein the orientation specifying unit specifies the hand region side as a forward direction of a person when a distance between the head region and the hand region is equal to or greater than a threshold value.   The determination apparatus according to claim 4, further comprising a threshold value determination unit that determines the threshold value based on a size of the person area. It further has a measuring means for measuring the time in which the hand region exists in each region divided with reference to the center of the person region,
The determination apparatus according to claim 1, wherein the direction specifying unit specifies a direction of a person based on a measurement result by the measuring unit. Further comprising object extracting means for extracting an object region based on the photographed image,
The determination apparatus according to claim 1, wherein the orientation specifying unit specifies the object region side as the front direction of the person when the hand region and the object region overlap each other.   The determination apparatus according to claim 1, further comprising likelihood specifying means for specifying a likelihood of a determination result obtained by the left / right determination means.   In the first captured image included in the series of captured images, when the determination result by the left / right determination unit is obtained, the hand in the second captured image included in the sequence of captured images is based on the determination result. The determination apparatus according to claim 1, further comprising an estimation unit that estimates the left and right of the region.   The determination apparatus according to claim 1, further comprising a work determination unit that determines whether or not a specific work is being performed based on a determination result by the left / right determination unit.   The determination apparatus according to claim 10, wherein the work determination unit determines that a specific work is being performed when one of the left and right predetermined hand regions contacts a specific position. .   The determination apparatus according to claim 11, further comprising a left / right determination unit that determines the one hand region based on information on a dominant hand of a subject person. A person extraction step of extracting a person area in the photographed image;
A hand extraction step of extracting a hand region based on the person region;
A direction specifying step for specifying the direction of the person from the positional relationship between the person area other than the hand area and the hand area;
And a left / right determination step of determining left / right of the hand area based on the orientation of the person.   The program for functioning a computer as each means of the determination apparatus of any one of Claims 1 thru | or 12.

Images