JP7470069B2 - Pointing object detection device, pointing object detection method, and pointing object detection system - Google Patents

Description

This disclosure relates to a pointing object detection device, a pointing object detection method, and a pointing object detection system.

In recent years, research has been conducted into recognizing various human movements, with the aim of putting man-machine interfaces into practical use. By recognizing human movements, it becomes possible to convey the user's intentions to the computer in a natural way.

Techniques for detecting pointing gestures have been attracting attention as an effective means of communication for conveying a user's intentions to a computer. Pointing gestures are natural and intuitive for humans, and do not require the wearing of any special equipment, so they do not burden the human user and are low-cost.
Techniques for detecting pointing gestures can be used to realize man-machine interfaces in a variety of situations, such as housework, welfare, and industry, enabling services to support people.

One method for detecting a pointing gesture is described in, for example, H. Asano, T. Nagayasu, T. Orimo, K. Terabayashi, M. Ohta and K. Umeda, "Recognition of finger-pointing direction using color clustering and image segmentation," The SICE Annual Conference 2013, Nagoya, Japan, 2013, pp. 2029-2034. (Non-Patent Document 1).
Non-Patent Document 1 proposes an accurate method for recognizing pointing directions using multiple cameras. In this method, the input image is segmented by color clustering before shape analysis. Then, some limitations of previous methods, such as narrow operation areas and predefined skin colors, are lifted. As a result, it is described that eight-direction pointing recognition is achieved with an average recall rate of 90% and accuracy of 93%.

H. Asano, T. Nagayasu, T. Orimo, K. Terabayashi, M. Ohta and K. Umeda, "Recognition of finger-pointing direction using color clustering and image segmentation," The SICE Annual Conference 2013, Nagoya, Japan, 2013, pp. 2029-2034.

In Non-Patent Document 1, narrow regions are extracted as fingers from an image showing a human hand, and then the pointing direction is determined by applying the least squares method to pixels belonging to the fingers.
However, in the method described in Non-Patent Document 1, fingers in an image are extracted based on the relative width of each part of the hand. This limits the accuracy of determining the pointing direction when the width of each part of the hand cannot be determined due to the angle of view, noise, object detection errors, etc., or when the hand is partially hidden by a sleeve or the like and the apparent width of the hand part is inaccurate.

The present disclosure aims to provide a highly robust pointing object detection method that can detect a pointing object with high accuracy even in an image in which the pixels of the hand are not clearly visible due to the angle of view, noise, etc., by determining the pointing direction based on the curvature of the pixels in the hand image and the distance from the center of the hand, and detecting the pointing object that is the target of the pointing action based on the determined pointing direction.

In order to solve the above problem, one representative pointed object detection device of the present disclosure includes an image input unit that acquires an input video including a user's hand and at least one pointed object candidate, an object detection unit that analyzes the input video and detects the hand and the pointed object candidate, a gesture determination unit that determines a gesture made by the hand, a pointing direction determination unit that, if the gesture made by the hand is determined to be a pointing motion, identifies a centroid of the hand, and identifies an area that satisfies a first distance criterion from the centroid and a predetermined curvature criterion as the fingertip of the index finger used in the pointing motion, and determines the pointing direction of the pointing motion based on the distribution of pixels belonging to the identified fingertip, and a pointed object identification unit that identifies an object pointed to by the pointing motion in the input video from among the pointed object candidates based on the pointing direction.

According to the present disclosure, by determining the pointing direction based on the curvature of the pixels in the hand image and the distance from the center of the hand, and detecting the pointing object that is the target of the pointing action based on the determined pointing direction, it is possible to provide a highly robust pointing object detection means that enables highly accurate pointing object detection even in cases where the pixels of the hand are not clearly visible due to the angle of view, noise, etc.
Other objects, configurations and effects will become apparent from the following description of the preferred embodiment of the invention.

FIG. 1 is a diagram illustrating an example of a hardware configuration of a computer system for implementing an embodiment of the present disclosure. FIG. 2 is a diagram illustrating a configuration of a pointing object detection system according to the first embodiment of the present disclosure. FIG. 3 is a flowchart showing a flow of the pointing object detection method according to the first embodiment of the present disclosure. FIG. 4 is a flowchart illustrating a flow of a pointing direction determination process according to the first embodiment of the present disclosure. FIG. 5 is a diagram illustrating an example of a first pointing direction determination unit according to the first embodiment of the present disclosure. FIG. 6 is a diagram illustrating an example of a second pointing direction determination unit according to the first embodiment of the present disclosure. FIG. 8 is a diagram illustrating an example of a third pointing direction determination unit according to the first embodiment of the present disclosure. FIG. 8 is a diagram illustrating an example of a fourth pointing direction determination unit according to the first embodiment of the present disclosure. FIG. 9 is a diagram illustrating an example of a fifth pointing direction determination unit according to the first embodiment of the present disclosure. FIG. 10 is a diagram illustrating a configuration of a pointing object detection system according to the second embodiment of the present disclosure. FIG. 11 is a diagram illustrating an example of the configuration of an object management unit according to the second embodiment of the present disclosure. FIG. 12 is a flowchart showing the flow of a user object sequence information acquisition process according to the second embodiment of the present disclosure. FIG. 13 is a flowchart showing the flow of an object handling sequence verification process according to the second embodiment of the present disclosure. FIG. 14 is a diagram illustrating an example of a configuration of an environment recognition unit according to the second embodiment of the present disclosure. FIG. 15 is a diagram illustrating an example of the configuration of a user information acquisition unit according to the second embodiment of the present disclosure. FIG. 16 is a diagram illustrating an example of a configuration of an update unit according to the second embodiment of the present disclosure. FIG. 17 is a diagram illustrating a configuration of a pointing object detection system according to a third embodiment of the present disclosure. FIG. 18 is a diagram illustrating a flow of a task distribution process performed by a task management unit according to the third embodiment of the present disclosure.

The pointed object detection means according to the embodiments of the present disclosure may be used to realize a man-machine interface and directly assist people in various situations, such as housework, welfare, and industry, but for the sake of convenience, the following description focuses on examples in which the pointed object detection means according to the embodiments of the present disclosure is applied to industry.

In factories, etc., workers may disassemble or assemble machines or equipment while inspecting or repairing them. In such cases, the worker may forget to inspect a particular part or may disassemble or assemble the machine in the wrong order. Therefore, by using the object detection means according to the embodiment of the present disclosure to detect the part the worker is pointing at, it becomes possible to record the worker's actions and detect actions that are out of order, thereby improving the accuracy and efficiency of work.

As an example, if a worker points at a specific part (e.g., one of the parts that make up a machine or device) before handling it, an input image showing this pointing action is captured by an imaging device such as a camera. The captured input image is then processed by each functional unit of the pointing object detection device described below, and the pointing direction indicated by the fingertip is determined based on the curvature of the pixels in the input image and the distance from the center of the hand, and the pointing object that is the target of the pointing action can be detected based on the determined pointing direction. User object sequence information indicating the order of the worker's actions is then recorded, and this user object sequence information can be compared with specified object sequence information that is prepared in advance and indicates the correct order of actions, to detect abnormalities (such as errors) in the machine handling procedures.

In this way, according to the present disclosure, by determining the pointing direction based on the curvature of the pixels in the hand image and the distance from the center of the hand, and detecting the pointed object that is the target of the pointing action based on the determined pointing direction, it is possible to provide a highly robust pointed object detection means that can detect the pointed object with high accuracy even in cases where the hand pixels are not clearly visible in the image due to the angle of view, noise, etc.

Below, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are denoted by the same reference numerals.

First, referring to FIG. 1, a computer system 300 for implementing an embodiment of the present disclosure will be described. The mechanisms and devices of the various embodiments disclosed herein may be applied to any suitable computing system. The main components of the computer system 300 include one or more processors 302, memory 304, a terminal interface 312, a storage interface 314, an I/O (input/output) device interface 316, and a network interface 318. These components may be interconnected via a memory bus 306, an I/O bus 308, a bus interface unit 309, and an I/O bus interface unit 310.

Computer system 300 may include one or more general purpose programmable central processing units (CPUs) 302A and 302B, collectively referred to as processors 302. In some embodiments, computer system 300 may include multiple processors, and in other embodiments, computer system 300 may be a single CPU system. Each processor 302 executes instructions stored in memory 304 and may include an on-board cache.

In some embodiments, memory 304 may include random access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing data and programs. Memory 304 may store all or a portion of the programs, modules, and data structures that implement the functions described herein. For example, memory 304 may store a pointing object detection application 350. In some embodiments, pointing object detection application 350 may include instructions or descriptions that execute on processor 302 the functions described below.

In some embodiments, the pointing object detection application 350 may be implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and/or other physical hardware devices instead of or in addition to a processor-based system. In some embodiments, the pointing object detection application 350 may include data other than instructions or descriptions. In some embodiments, cameras, sensors, or other data input devices (not shown) may be provided to communicate directly with the bus interface unit 309, the processor 302, or other hardware of the computer system 300.

Computer system 300 may include a bus interface unit 309 that provides communication between processor 302, memory 304, display system 324, and I/O bus interface unit 310. I/O bus interface unit 310 may be coupled to an I/O bus 308 for transferring data to and from various I/O units. I/O bus interface unit 310 may communicate via I/O bus 308 with multiple I/O interface units 312, 314, 316, and 318, also known as I/O processors (IOPs) or I/O adapters (IOAs).

The display system 324 may include a display controller, a display memory, or both. The display controller may provide video, audio, or both data to the display device 326. The computer system 300 may also include one or more sensors or other devices configured to collect data and provide the data to the processor 302.

For example, computer system 300 may include biometric sensors to collect data such as heart rate data or stress level data, environmental sensors to collect data such as humidity data, temperature data, or pressure data, and motion sensors to collect acceleration data, movement data, etc. Other types of sensors may also be used. Display system 324 may be connected to a display device 326, such as a standalone display screen, a television, a tablet, or a handheld device.

The I/O interface unit provides the ability to communicate with various storage or I/O devices. For example, the terminal interface unit 312 can be fitted with user I/O devices 320, such as user output devices such as a video display, a speaker television, and user input devices such as a keyboard, a mouse, a keypad, a touchpad, a trackball, buttons, a light pen, or other pointing devices. A user may use a user interface to input input data or instructions to the user I/O devices 320 and the computer system 300 and receive output data from the computer system 300 by operating the user input devices. The user interface may be displayed on a display device, played through a speaker, or printed via a printer via the user I/O devices 320, for example.

The storage interface 314 may be attached to one or more disk drives or direct access storage devices 322 (usually magnetic disk drive storage devices, but may also be an array of disk drives or other storage devices configured to appear as a single disk drive). In some embodiments, the storage device 322 may be implemented as any secondary storage device. The contents of the memory 304 may be stored in the storage device 322 and retrieved from the storage device 322 as needed. The I/O device interface 316 may provide an interface to other I/O devices, such as printers, fax machines, etc. The network interface 318 may provide a communications path to allow the computer system 300 and other devices to communicate with each other. This communications path may be, for example, a network 330.

In some embodiments, computer system 300 may be a device that receives requests from other computer systems (clients) without a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer. In other embodiments, computer system 300 may be a desktop computer, a portable computer, a laptop, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable electronic device.

Next, the configuration of the pointing object detection system according to the first embodiment of the present disclosure will be described with reference to FIG.

FIG. 2 is a diagram showing a configuration of a pointed object detection system 200 according to a first embodiment of the present disclosure. As shown in FIG. 2, the pointed object detection system 200 according to a first embodiment of the present disclosure includes a pointed object detection device 210 and a user terminal 250. The pointed object detection device 210 and the user terminal 250 may be connected via a communication network 234 such as a LAN or the Internet.

Pointing object detection device 210 is a device for analyzing an input video showing a pointing action performed by user 235, determining the pointing direction of the pointing action, and detecting pointing object 240 that is the target of the pointing action based on the determined pointing direction. Pointing object detection device 210 may be implemented by any computing device, such as a desktop computer, a laptop computer, a tablet, or a smartphone.
As shown in FIG. 2 , the pointed object detection device 210 includes an object detection unit 212 , an image processing unit 214 , a gesture determination unit 218 , a pointed direction determination unit 220 , a pointed object identification unit 222 , an image input unit 224 , a processor 226 , and a memory unit 228 .

The image input unit 224 is a functional unit for acquiring an input video showing the hand of the user 235 and at least one pointing object candidate. The image input unit 224 may be, for example, a photographing unit such as an RGB camera, or may be a functional unit for receiving an image signal.
From the standpoint of improving the accuracy of pointing object detection, when the image input unit 224 is an imaging unit, it is desirable that this imaging unit be positioned in a position where it can easily capture an image of the hand of the user 235 or an object that is the subject of the user's work.
In the following description, the image input unit 224 is assumed to mean an image capturing unit, but the image input unit 224 may be a functional unit that receives an image signal.
2 shows an example of a configuration in which image input unit 224 is included in pointed object detection device 210, but the present disclosure is not limited to this, and image input unit 224 may be a separate device independent of pointed object detection device 210. In this case, image input unit 224 may transmit the acquired input video to pointed object detection device 210 via communication network 234.

The object detection unit 212 is a functional unit for analyzing the input image acquired by the image input unit 224 and detecting the hand of the user 235 and candidate pointing objects. The object detection unit 212 may detect all objects appearing in the input image and generate a label indicating the class of the detected object. The means of the object detection unit 212 according to the embodiment of the present disclosure are not particularly limited, and may include any object detection method, such as a machine learning approach based on Haar-Like features, a scale-invariant feature transform (SIFT), Hog features, or a deep learning approach such as region proposal (R-CNN, Fast R-CNN, Faster R-CNN, cascade R-CNN), Single Shot MultiBox Detector (SSD), You Only Look Once (YOLO), Single-Shot Refinement Neural Network for Object Detection (RefineDet), Retina-Net, and Deformable convolutional networks.

The image processing unit 214 analyzes the input video in which each object is detected by the object detection unit 212, and extracts from the input video a hand image showing only the hand of the user 235. For example, the image processing unit 214 may cut out (crop) an image showing the hand of the user 235 from the input video, may use skin-color based image segmentation of the hand of the user 235, or may use a deep learning approach such as segnet, ICNET, or FPN.
As a result of processing by the image processing unit 214, a binary image is obtained in which the brightness values of pixels belonging to the hand of the user 235 are 1 or greater, and the brightness values of pixels not belonging to the hand of the user 235 are 0 (see Figures 5 to 9).

The gesture determination unit 218 is a functional unit that determines a gesture shown by the hand of the user 235 in the hand image extracted by the image processing unit 214. For example, the gesture determination unit 218 may identify key points (fingers, joints, wrist, back of hand, etc.) of the user 235's hand and determine whether the gesture shown by the hand is a pointing motion or not based on the distribution of the identified key points of the hand. The pointing motion here means pointing a finger at a target object. This pointing motion may also be performed with any finger, such as the index finger or middle finger. Here, the finger of the user 235 used in the pointing motion is called the "pointing finger".
The gesture determination unit 218 here may use any existing means, such as a 3D model-based algorithm, a skeletal-based algorithm, an electromyograph-based model, or the like, to determine the gesture indicated by the hand of the user 235 .

The pointing direction determination unit 220 is a functional unit for determining the pointing direction of the pointing motion. The "pointing direction" here refers to the direction pointed by the index finger of the user 235. More specifically, the pointing direction determination unit 220 may identify a centroid of the hand of the user 235, identify an area that satisfies a first distance criterion from the centroid and also satisfies a predetermined curvature criterion as the tip of the index finger, and determine the pointing direction of the pointing motion based on the distribution of pixels belonging to the identified fingertip.
In this disclosure, "centroid" generally refers to the center of mass, but may be the geometric center on an image, or a physical center of gravity may be used instead of the geometric center.
The process of determining the pointing direction using the pointing direction determining unit 220 will be described in detail later.

The pointing object identification unit 222 identifies the pointing object 240 pointed to by the pointing action in the input video from among the pointing object candidates, based on the pointing direction determined by the pointing direction determination unit 220. Here, the pointing object identification unit 222 may use the label of the object detected by the object detection unit 212 in addition to the determined pointing direction.

The processor 226 is a computing device for executing instructions to realize the functions of each functional unit of the pointed-object detection device 210, and is substantially similar to the processor 302 shown in FIG. 1, for example, and therefore will not be described here.

The memory unit 228 is a storage unit for storing various data used by the functional units of the pointed-object detection device 210. The memory unit 228 here may be, for example, a local storage such as a hard disk drive or a solid-state drive, or may be a distributed storage service such as the cloud.

The user 235 is a user who performs a pointing action that is the subject of analysis by the pointing object detection device 210. The user 235 may be, for example, a worker working in a factory, or a user performing any task, such as a caregiver who cares for a patient. In addition, for the sake of convenience in this disclosure, a case in which the user 235 is a human will be described as an example, but the present disclosure is not limited to this, and the user 235 may be, for example, a monkey such as a chimpanzee or an orangutan, or any living organism or artificial object, etc., as long as it has a hand that can perform a pointing action.

The pointing object 240 is an object that is the target of a pointing action by the user 235. The pointing object 240 may be any object depending on the task of the user 235, such as a part that constitutes a machine or device to be inspected, a tool such as a hammer or a screwdriver, or a fork or knife placed on a table.

The user terminal 250 is, for example, a terminal device used by the user 235. The user terminal 250 may accept input of any information that assists in pointing object detection, such as user information about the user 235 and environmental information about the environment of the user 235. In addition, in one embodiment, if it is determined that an abnormality has occurred in the handling procedure of the pointing object, an abnormality notification indicating the abnormality may be transmitted to the user terminal 250. In this way, the user 235 of the user terminal 250 can take action to resolve the detected abnormality.

With the pointed object detection system 200 configured as described above, the pointing direction is determined based on the curvature of the pixels in the hand image and the distance from the center of the hand, and the pointed object that is the target of the pointing action is detected based on the determined pointing direction. This makes it possible to provide highly robust pointed object detection that can detect the pointed object with high accuracy even in cases where the pixels of the hand are not clearly visible due to the angle of view, noise, etc.

Next, the pointing object detection method according to the first embodiment of the present disclosure will be described with reference to FIG.

Fig. 3 is a flowchart showing the flow of a pointing object detection method 360 according to the first embodiment of the present disclosure. The pointing object detection method 360 shown in Fig. 3 is a method for determining the pointing direction of a pointing motion by analyzing an input video showing the pointing motion, and detecting a pointing object 240 that is the target of the pointing motion based on the determined pointing direction, and is implemented by a pointing object detection device (e.g., the pointing object detection device 210 shown in Fig. 2).

First, in step S362, the image capture unit (e.g., image input unit 224 shown in FIG. 2) acquires an input video 363 showing the user's hand and at least one pointing object candidate. This input video 363 may be, for example, a still image or a video composed of a number of image frames. In one aspect of the present disclosure, the user may input an instruction to start work via the above-mentioned user terminal (e.g., user terminal 250 shown in FIG. 2), and then the image capture unit may start capturing the user's work and acquire the input video 363.
The input video 363 acquired by the imaging unit may be stored in the above-mentioned storage unit (for example, the storage unit 228 shown in FIG. 2).

Next, in step S364, an object detection unit (for example, object detection unit 212 shown in FIG. 2) analyzes input image 363 acquired by the image capture unit, and detects the user's hand and a pointing object candidate. The pointing object candidate here is an object that is included in input image 363 and may be a target object for the pointing action.
As described above, the object detection unit 212 may detect all objects included in the input video 363 and generate labels 365 indicating the classes of the detected objects. These labels 365 may be stored in, for example, the storage unit described above.

If the result of the processing by the object detection unit in step S364 is that it is determined that the object recognized as a hand is present in the input video 363 (i.e., an object corresponding to the label "hand" among the labels 365 is detected), the process proceeds to step S366. On the other hand, if the result of the processing by the object detection unit in step S364 is that it is determined that the object recognized as a hand is not present in the input video 363, the process returns to step S362, and the image capture unit continues acquiring the input video 363.

Next, in step S366, the image processing unit (for example, the image processing unit 214 shown in FIG. 2) extracts a hand image showing only the user's hand from the input video 363. For example, the image processing unit may cut out (crop) an image showing the user's hand from the input video, may use skin-color based image segmentation of the user's hand, or may use a deep learning approach such as segnet, ICNET, or FPN. As a result of processing by the image processing unit, a hand image is obtained in which the luminance values of pixels belonging to the user's hand are 1 or more, and the luminance values of pixels not belonging to the user's hand are 0.

Next, in step S368, a gesture determination unit (e.g., gesture determination unit 218 shown in FIG. 2) determines the gesture indicated by the hand by analyzing the hand image extracted in step S366. The gesture determination unit may, for example, identify key points of the user's hand (fingers, joints, wrist, back of the hand, etc.) and determine whether the gesture indicated by the hand is a pointing motion based on the distribution of the identified key points of the hand.

If the result of the processing by the gesture determination unit in step S368 indicates that the gesture indicated by the hand is a pointing motion, the process proceeds to step S370. On the other hand, if the result of the processing by the gesture determination unit in step S368 indicates that the gesture indicated by the hand is not a pointing motion, the process returns to step S362, and the image capture unit continues acquiring the input video 363.

Next, in step S370, a pointing direction determination unit (for example, the pointing direction determination unit 220 shown in FIG. 2) determines the pointing direction of the pointing motion. More specifically, the pointing direction determination unit identifies a centroid of the user's hand, identifies an area that satisfies a first distance criterion from the centroid and also satisfies a predetermined curvature criterion as the tip of the index finger, and determines the pointing direction of the pointing motion based on the distribution of pixels belonging to the identified tip.
The process of determining the pointing direction using the pointing direction determination unit will be described later in detail.

Next, in step S372, a pointing object identification unit (e.g., pointing object identification unit 222 shown in FIG. 2) identifies, from among the pointing object candidates, the pointing object 240 that is pointed to by the pointing action in the input video 363, based on the pointing direction determined by the pointing direction determination unit in step S372 and the label 365 determined in step S364. Information on the pointing object identified here may be stored in the storage unit described above (e.g., storage unit 228 shown in FIG. 2).

According to the pointing object detection method 360 described above, the pointing direction is determined based on the curvature of the pixels in the hand image and the distance from the center of the hand, and the pointing object that is the target of the pointing action is detected based on the determined pointing direction. This makes it possible to provide highly robust pointing object detection that enables highly accurate pointing object detection even in cases where the pixels of the hand are not clearly visible due to the angle of view, noise, etc.

Next, referring to FIG. 4, the pointing direction determination process according to the first embodiment of the present disclosure will be described.

4 is a diagram showing a flow of a pointing direction determination process 400 according to the first embodiment of the present disclosure. The pointing direction determination process 400 shown in Fig. 4 is a process for determining the pointing direction of a pointing motion, and is executed by a pointing direction determination unit (e.g., the pointing direction determination unit 220 shown in Fig. 2).
The pointing direction determination process 400 shown in Fig. 4 substantially corresponds to step S370 in the pointing object detection method 360 shown in Fig. 3. The pointing direction determination process 400 shown in Fig. 4 is a rough process for determining the pointing direction of a pointing motion, and the details of the process for determining the pointing direction of a pointing motion will be described with reference to specific examples of the pointing direction determination means shown in Figs.

First, in step S410, the pointing direction determination unit identifies the tip of the index finger used in the pointing motion in the hand image extracted from the input video. More specifically, the pointing direction determination unit first identifies the centroid of the hand in the hand image. The centroid in the first embodiment may be obtained based on an existing image processing method or statistical method. After identifying the centroid of the hand, the pointing direction determination unit identifies an area that satisfies a first distance criterion from the centroid and also satisfies a predetermined curvature criterion as the tip of the index finger.
This first distance criterion may specify a lower limit on the distance a region must be from the centroid for it to qualify as a fingertip.
Similarly, the curvature criterion may specify a lower limit for the curvature required for a region to qualify as a fingertip.
In this way, by using the distance and curvature criteria described above, an area that is a predetermined distance or more away from the centroid of the hand (i.e., the center of the hand) and has a predetermined curvature or more can be identified as the tip of the index finger.
It should be noted that the first distance and curvature criteria here may be specified by a user or may be specified by a machine learning means trained to set appropriate values based on previously collected test images.

Next, in step S420, the pointing direction determination unit determines the distribution of pixels belonging to the tip of the index finger identified in step S410. Here, "pixel distribution" means the spatial relationship between the pixels belonging to the fingertip and other pixels. As an example, the pointing direction determination unit may identify the relative positional relationship between the pixels belonging to the tip of the index finger identified in step S410 and the pixels belonging to the centroid.

Next, in step S430, the pointing direction determination unit determines the pointing direction of the pointing motion based on the distribution of pixels identified in step S420. For example, in one embodiment, the pointing direction determination unit may determine the direction of an imaginary line passing through pixels belonging to the fingertip of the index finger and pixels belonging to the centroid as the pointing direction. The pointing direction determined in step S430 may be stored in the storage unit described above (for example, the storage unit 228 shown in FIG. 2).
As described above, the details of the process of determining the direction of the pointing action based on the distribution of pixels will be described with reference to specific examples of the pointing direction determining means shown in FIGS.

According to the pointing direction determination process 400 described above, the pointing direction is determined based on the curvature of the pixels in the hand image and the distance from the center of the hand, and the pointing object that is the target of the pointing action is detected based on the determined pointing direction, thereby providing a highly robust pointing object detection means that is capable of detecting the pointing object with high accuracy even in the case of an image in which the pixels of the hand are not clearly visible due to the angle of view, noise, etc.

Next, a specific example of the process for determining the direction of a pointing gesture will be described with reference to Figures 5 to 9.

Fig. 5 is a diagram showing an example of the first pointing direction determination means 500 according to the first embodiment of the present disclosure. As shown in Fig. 5, a user's hand 502 is shown in a hand image 505, and as a result of processing by the above-mentioned gesture determination unit (e.g., the gesture determination unit 218 shown in Fig. 2), it is determined that the hand 502 is making a pointing gesture.

First, as described above, the pointing direction determination unit (for example, the pointing direction determination unit 220 shown in FIG. 2) identifies the centroid 510 of the hand 502. This centroid 510 is the center of mass of the hand 502, and may be determined based on existing image processing or statistical methods. After identifying the centroid 510 of the hand 502, the pointing direction determination unit identifies an area that satisfies a first distance criterion from the centroid 510 and also satisfies a predetermined curvature criterion as the fingertip 512 of the index finger 504.

Next, the pointing direction determination unit determines a target region 511 centered on the fingertip 512 based on a distance d between the identified fingertip 512 and the centroid 510. As an example, the target region 511 may be a circular region with a radius of the distance d/3.
It should be noted that the radius of the target region 511 is a hyperparameter, and may be set by the user or may be optimized based on an existing hyperparameter optimization method.

Next, the pointing direction determination unit specifies pixels in the target region 511 that satisfy a predetermined luminance criterion as finger pixels that belong to the index finger 504 of the hand 502 used for the pointing motion. For example, if the hand image 505 is a binary image in which the luminance value of pixels that belong to the hand 502 is 1 or more and the luminance value of pixels that do not belong to the hand 502 is 0, this luminance criterion may be "1".
In this way, by setting the brightness standard as "1", it is possible to identify bright pixels in the target region 511 (that is, pixels in the target region 511 that belong to the index finger 504).
Note that, although the above describes an example in which finger pixels are identified based on the target region and pixel luminance values, the present disclosure is not limited to this, and any method such as clustering or recursion may be used.

Next, the pointing direction determination unit calculates the mean pixel coordinates of the identified finger pixels 514. The mean pixel coordinates 514 may be calculated, for example, by any existing image processing means.
After calculating the average pixel coordinates 514, the pointing direction determination unit determines the direction of an imaginary line 516 that passes through the calculated average pixel coordinates 514 and the fingertip 512 as the pointing direction.

Fig. 6 is a diagram showing an example of the second pointing direction determination means 600 according to the first embodiment of the present disclosure. As shown in Fig. 6, a user's hand 602 is shown in a hand image 605, and as a result of processing by the gesture determination unit described above (e.g., the gesture determination unit 218 shown in Fig. 2), it is determined that the hand 602 is making a pointing gesture.

First, as described above, the pointing direction determination unit (for example, the pointing direction determination unit 220 shown in FIG. 2) identifies the centroid 610 of the hand 602. This centroid 610 is the center of mass of the hand 602, and may be determined based on existing image processing or statistical methods. After identifying the centroid 610 of the hand 602, the pointing direction determination unit identifies an area that satisfies a first distance criterion from the centroid 610 and also satisfies a predetermined curvature criterion as the fingertip 612 of the index finger 604.

Next, the pointing direction determination unit determines a first target region 611 centered on the fingertip 612 based on the distance d between the identified fingertip 612 and the centroid 610. As an example, the first target region 611 may be a circular region with a radius of the distance d/6.

Next, the pointing direction determination unit specifies pixels in the first target region 611 that satisfy a predetermined luminance criterion as finger pixels belonging to the index finger 604 of the hand 602 used for the pointing motion. For example, if the hand image 605 is a binary image in which the luminance value of pixels belonging to the hand 602 is 1 or more and the luminance value of pixels not belonging to the hand 602 is 0, this luminance criterion may be "1".
In this way, by setting the brightness standard as "1", it is possible to identify bright pixels in the first object region 611 (i.e., pixels in the first object region 611 that belong to the index finger 604).

Next, the pointing direction determination unit calculates the mean pixel coordinate 614 of the identified finger pixels.

Next, the pointing direction determination unit determines a second target region 613 centered on the calculated average pixel coordinate 614. As an example, the second target region 613 may be a circular region with a radius of the distance d/6, similar to the first target region 611.

Next, the pointing direction determination unit determines the direction of an imaginary line 616 that passes through the average pixel coordinate 614, which is the center of the second target area, and the fingertip 612, which is the center of the first target area, as the pointing direction.

The second pointing direction determination means 600 shown in FIG. 6 described above can improve the accuracy of determining the pointing direction by using multiple target regions with smaller radii than the first pointing direction determination means 500 shown in FIG. 5 .
In the above, the case where two target regions are used has been described as an example, but the present disclosure is not limited to this, and three or more target regions may be used. In addition, in the above, the case where the target region is a circle has been described as an example, but the present disclosure is not limited to this, and the target region may be any shape, such as a rectangle.

Fig. 7 is a diagram showing an example of a third pointing direction determination means 700 according to the first embodiment of the present disclosure. As shown in Fig. 7, a user's hand 702 is shown in a hand image 705, and as a result of processing by the gesture determination unit described above (e.g., the gesture determination unit 218 shown in Fig. 2), it is determined that the hand 702 is making a pointing gesture.

First, as described above, the pointing direction determination unit (for example, the pointing direction determination unit 220 shown in FIG. 2) identifies the centroid 710 of the hand 702. This centroid 710 is the center of mass of the hand 702, and may be determined based on existing image processing or statistical methods. After identifying the centroid 710 of the hand 702, the pointing direction determination unit identifies an area that satisfies a first distance criterion from the centroid 710 and also satisfies a predetermined curvature criterion as the fingertip 714 of the index finger 704.

Next, the pointing direction determination unit sets a first imaginary line 706 connecting the identified fingertip 714 and the centroid 710. Thereafter, the pointing direction determination unit sets a second imaginary line 708 that is perpendicular to the first imaginary line 706 and satisfies a second distance criterion with respect to the fingertip 714, and sets a third imaginary line 712 that is perpendicular to the first imaginary line 706, parallel to the second imaginary line 708, and satisfies a third distance criterion with respect to the fingertip 714. In addition, the second imaginary line 708 and the third imaginary line 712 may be determined only in an area where the luminance value is 1 or more (i.e., they do not extend beyond the boundary of the index finger 704 to the background).
The second distance criterion and the third distance criterion here may be different from each other and may be values set by a user or a machine learning means. By being different from each other, the second distance criterion and the third distance criterion can space the second and third imaginary lines 708 and 712 apart.

Next, the indication direction determination unit may determine the direction of a fourth imaginary line 718 that passes through the center point of the second imaginary line 708 and the center point of the third imaginary line 712 as the indication direction.

The third pointing direction determination means 700 shown in FIG. 7 described above can reduce the required computing resources and improve the processing speed, compared to the first and second pointing direction determination means 500, 600 shown in FIGS. 5 and 6.
Note that, although the above describes an example in which two overhead lines are determined to be perpendicular to the first overhead line 706, the present disclosure is not limited to this, and the number of overhead lines perpendicular to the first overhead line 706 may be, for example, three or more.

Fig. 8 is a diagram showing an example of a fourth pointing direction determination means 800 according to the first embodiment of the present disclosure. As shown in Fig. 8, a user's hand 802 is shown in a hand image 805, and as a result of processing by the gesture determination unit described above (e.g., the gesture determination unit 218 shown in Fig. 2), it is determined that the hand 802 is making a pointing gesture.

First, as described above, the pointing direction determination unit (for example, the pointing direction determination unit 220 shown in FIG. 2) identifies the centroid 810 of the hand 802. This centroid 810 is the center of mass of the hand 802, and may be determined based on existing image processing or statistical methods. After identifying the centroid 810 of the hand 802, the pointing direction determination unit identifies an area that satisfies a first distance criterion from the centroid 810 and also satisfies a predetermined curvature criterion as the fingertip 814 of the index finger 804.

Next, the pointing direction determination unit identifies pixels belonging to the index finger 804 as finger pixels based on the identified fingertip 814 and the centroid 810. Here, the pointing direction determination unit may identify pixels that exist within the target area and satisfy a predetermined brightness standard as finger pixels, as described above.

Next, the pointing direction determination unit determines the pointing direction 815 based on the finger pixels using least squares fitting. However, the present disclosure is not limited to the least squares fitting, and after identifying the finger pixels, the pointing direction 815 may be identified using principal component analysis.

Fig. 9 is a diagram illustrating an example of a fifth pointing direction determination means 900 according to the first embodiment of the present disclosure. As shown in Fig. 9, a user's hand 902 is shown in a hand image 905, and as a result of processing by the gesture determination unit described above (e.g., the gesture determination unit 218 shown in Fig. 2), it is determined that the hand 902 is making a pointing gesture.

First, the pointing direction determination unit fits a bounding box 915 to the hand 902. The bounding box here may be, for example, a rectangle, an ellipse, or any other shape. Then, the pointing direction determination unit fits a best fit line 920 to the hand 902 based on the bounding box 915. The pointing direction determination unit may determine the direction of the best fit line 920 as the pointing direction of the pointing motion.

Above, with reference to Figures 5 to 9, a specific example of a process for determining the direction of a pointing gesture according to an embodiment of the present disclosure has been described. However, the present disclosure is not limited to this, and any means may be used as long as it can determine the direction of a pointing gesture with high accuracy.
For example, as another example, the pointing direction determination unit may determine the two-dimensional coordinates (horizontal and vertical coordinates) of the pixels of the user's hand, and analyze the distribution, mean, and variance of these pixel coordinates to determine the orientation and pointing direction of the hand.

Next, the configuration of a pointing object detection system according to Example 2 of the present disclosure will be described with reference to FIG. 10.

FIG. 10 is a diagram showing the configuration of a pointed object detection system 1000 according to a second embodiment of the present disclosure. The pointed object detection device 1010 of the pointed object detection system 1000 according to the second embodiment of the present disclosure is different from the pointed object detection device 210 of the pointed object detection system 200 according to the first embodiment of the present disclosure shown in FIG. 2 in that the pointed object detection device 1010 includes an object management unit 1030, an environment recognition unit 1032, a user information acquisition unit 1034, and an update unit 1036. Except for the inclusion of the object management unit 1030, the environment recognition unit 1032, the user information acquisition unit 1034, and the update unit 1036, the configuration of the pointed object detection system 1000 according to the second embodiment is substantially similar to that of the pointed object detection system 200 according to the first embodiment, and therefore the same parts are denoted by the same reference numerals. For convenience of explanation, the explanation of the above-mentioned parts will be omitted below, and the differences between the pointed object detection system 1000 and the pointed object detection system 200 will be mainly explained.

The object management unit 1030 is a functional unit for managing the handling procedure of the pointed object. In the present disclosure, the handling procedure may include individual tasks or steps such as processing, assembly, treatment, and inspection of the object, their order, and time-related management items.
The object management unit 1030 can create and store user object sequence information indicating the order in which the multiple pointing objects 240 are pointed to by the pointing action of the user 235. Furthermore, this user object sequence information may record the type of the pointing object 240 (hammer, nail, fork, etc.), the time when the pointing object 240 is pointed to, etc., in addition to the order in which the pointing objects 240 are pointed to by the pointing action of the user 235.
Furthermore, the object management unit 1030 may store designated object sequence information that specifies the order in which to handle the multiple pointing objects 240. This designated object sequence information is, for example, information that is created in advance and specifies the correct order in which to handle the pointing objects 240.
The object management unit 1030 can determine whether the pointed object is being handled correctly (i.e., whether the order in which the pointed object is handled is incorrect, etc.) by comparing the above-mentioned user object sequence information with the specified object sequence information.

The environment recognition unit 1032 is a functional unit for acquiring environmental information related to the surrounding environment of the user 235 and the pointing object 240. This environmental information may include any information related to the surrounding environment, such as lighting conditions, the position of the image input unit 224, and the distance between the user 235 and the pointing object 240. The environment recognition unit 1032 may input the environmental information via a specific user interface (such as a user interface of the user terminal 250), and may include a machine learning means for determining the environmental information from the video acquired by the image input unit 224.
As described below, the environmental recognition unit 1032 uses environmental information regarding the surrounding environment of the user 235 and the pointing object 240 to improve the accuracy of determining the pointing direction and the pointing object 240, and to more efficiently verify the user's 235 handling of the pointing object 240.

The user information acquisition unit 1034 is a functional unit for acquiring user information regarding the user 235. This user information may include any information regarding the user 235, such as the user's name, skin color, height, job title, and the presence or absence of qualifications. The user information acquisition unit 1034 may input the user information via a specific user interface (such as the user interface of the user terminal 250), and may include a machine learning means for determining the user information from the video acquired by the image input unit 224.
As described below, the user information acquisition unit 1034 uses user information about the user 235 to improve the accuracy of determining the pointing direction and the pointing object 240, and to more efficiently verify the user's 235 handling of the pointing object 240.

The update unit 1036 is a functional unit for updating the parameters of each functional unit of the pointed object detection device 1010 based on the environmental information and user information acquired by the above-mentioned environmental recognition unit 1032 and user information acquisition unit 1034.

The pointed object detection system 1000 according to the second embodiment of the present disclosure described above can detect the pointed object 240 that is the target of the user's pointing action while taking into account the user's information and the surrounding environment information, and can also verify the handling procedure of the pointed object 240.

Next, the object management unit according to the second embodiment of the present disclosure will be described with reference to FIG.

FIG. 11 is a diagram showing an example of the configuration of the object management unit 1030 according to the second embodiment of the present disclosure. As described above, the object management unit 1030 is a functional unit for managing the handling procedure of the pointed object.

11, the object management unit 1030 includes an object sequence information database 1120 that stores information for managing the handling of pointing objects. More specifically, the object sequence information database 1120 stores designated object sequence information 1130 that specifies the correct order for handling multiple pointing objects, and user object sequence information 1140 that indicates the order in which multiple pointing objects are pointed to by the user's pointing action.
As described above, the designated object sequence information 1130 is information that is created in advance and that specifies for each task the correct order of handling the designated object 240. The user object sequence information 1140 is information that indicates for each task the order of the designated objects that the user actually designated, and may be recorded in response to the designated objects being identified in real time.
As an example, the specified object sequence information 1130 may specify the order in which "Object1, Object2, Object3" are handled for SubTask-1 included in Task-1. Meanwhile, as a result of recording the order in which the pointing object targeted by the user's pointing action is pointed to, the object management unit 1030 may obtain user object sequence information 1140 indicating the order in which "Object2, Object3, Object1" are handled for SubTask-1 included in Task-1.

Next, the user object sequence information acquisition process according to the second embodiment of the present disclosure will be described with reference to FIG.

FIG. 12 is a flowchart showing the flow of a user object sequence information acquisition process 1200 according to the second embodiment of the present disclosure. The user object sequence information acquisition process 1200 according to the second embodiment of the present disclosure is a process for acquiring user object sequence information 1140 indicating the order in which multiple pointing objects are pointed to by the user's pointing action, and is executed by the object management unit 1030.

First, in step S1210, the object management unit 1030 acquires information about the users in charge of each task currently in progress in the workplace managed by the pointed object detection system. Here, the object management unit 1030 may acquire information about the users in charge of each task currently in progress from a user information database described below. Here, the user information may include, for example, the name of the user in charge of the task, an identifier (Worker ID) that uniquely identifies the user, etc.

Next, in step S1220, the object management unit 1030 acquires information on the pointing object identified in the target task. Here, the object management unit 1030 may acquire information on the pointing object identified by, for example, the pointing object detection method 360 shown in Fig. 3 from the above-mentioned storage unit (for example, the storage unit 228 shown in Fig. 10). When multiple pointing objects are identified for the target task, the object management unit 1030 acquires each of the identified pointing objects.
The information on the pointing object here may include, for example, an identifier (Object ID) that uniquely identifies the pointing object and the time when the pointing object is pointed.

Next, in step S1230, the object management unit 1030 can generate user object sequence information 1140 that indicates information on the pointed objects pointed to by a specific user for each work task in the order in which they were pointed to, by associating the information on the users in charge of each work task obtained in step S1210 with the information on the pointed objects obtained in step S1220. This user object sequence information 1140 may be stored, for example, in the object sequence information database 1120 described above.

According to the user object sequence information acquisition process 1200 described above, each time an object is identified as a pointed object targeted by a user's pointing action, information about the identified pointed object (such as the type of pointed object and the time when the pointed object is pointed to) is recorded and associated with user information, so that information indicating the order in which the pointed objects were actually pointed to by a specific user for each task can be acquired as user object sequence information 1140.

Next, the object handling sequence verification process according to the second embodiment of the present disclosure will be described with reference to FIG. 13.

FIG. 13 is a flowchart showing the flow of the object handling sequence verification process 1300 according to the second embodiment of the present disclosure. The object handling sequence verification process 1300 is a process for determining whether or not an abnormality has occurred in the handling procedure of the pointed object, and is performed by the object management unit 1030 described above.

First, in step S1310, the object management unit 1030 acquires user object sequence information and designated object sequence information.
Here, the object management unit 1030 may obtain the user object sequence information using the user object sequence information obtaining process 1200 described above with reference to FIG.
The object management unit 1030 may obtain the designated object sequence information by, for example, having an administrator (such as a factory manager) input the designated object sequence information. In an aspect of the present disclosure, the object management unit 1030 may use, as the designated object sequence information, user object sequence information created based on the actions of a user who has correctly performed a target task in the past.

Next, in step S1320, the object management unit 1030 compares the user object sequence information with the specified object sequence information to determine whether the handling procedure for the pointed object is correct (i.e., whether the order in which the pointed object is handled is incorrect, etc.).
More specifically, the object management unit 1030 verifies the user object sequence information by comparing the user object sequence information with the designated object sequence information, and determines whether the user object sequence information satisfies a predetermined similarity criterion for the designated object sequence information. If it is determined that the user object sequence information satisfies the predetermined similarity criterion for the designated object sequence information, the process returns to step S1310 and continues to acquire the user object sequence information and the designated object sequence information. If it is determined that the user object sequence information does not satisfy the predetermined similarity criterion for the designated object sequence information, the process proceeds to step S1330.
Note that this similarity standard is information that specifies the minimum similarity required to determine that the user object sequence information and the specified object sequence information match each other, and may be set in advance by an administrator, etc.

Next, in step S1330, the object management unit 1030 determines that an abnormality or error has occurred in the handling procedure of the pointed object, and outputs an abnormality notification indicating the abnormality. For example, the object management unit 1030 may output, to, for example, the user terminal 250, an abnormality notification including information such as the identification information of the task in which the abnormality occurred, the details of the abnormality (e.g., the handling procedure of the object is incorrect), the time the abnormality occurred, and the person in charge of the task.

As an example, the designated object sequence information specifies the order of handling "Object1, Object2, Object3" for SubTask-1 included in Task-1. Meanwhile, as a result of recording the order in which the pointing object targeted by the user's pointing action is pointed to, the object management unit 1030 acquires user object sequence information indicating the order of handling "Object2, Object3, Object1" for SubTask-1 included in Task-1.
In this case, the object management unit 1030 compares the designated object sequence information with the user object sequence information, and determines that the user object sequence information does not match the designated object sequence information, and determines that an abnormality has occurred regarding the handling procedure of the pointed object. The object management unit may then generate an abnormality notification indicating the abnormality and notify the user terminal 250. In this way, the user of the user terminal 250 can take action to resolve the detected abnormality.

In this way, according to the object handling sequence verification process 1300 described above, the handling procedure for the pointed-to object can be verified in real time, and it can be determined whether or not an abnormality has occurred in the handling procedure. This allows for early detection of worker mistakes in workplaces such as factories, thereby improving the accuracy and efficiency of work.

Next, the environment recognition unit according to the second embodiment of the present disclosure will be described with reference to FIG. 14.

14 is a diagram illustrating an example of the configuration of the environment recognition unit 1032 according to the second embodiment of the present disclosure. As described above, the environment recognition unit 1032 is a functional unit for acquiring environment information related to the surrounding environment of the user and the pointing object.
As shown in FIG. 14 , the environment recognition unit 1032 may be connected to a user terminal 250 via a communication network 234 .

As shown in FIG. 14 , the environment recognition unit 1032 includes a reception unit 1412 that receives environment information, a machine learning unit 1414 that determines the environment information from video, and an environment information database 1450 .
The environment information database 1450 is a database for storing environment information 1455 related to the surrounding environment of the user and the pointing object. As shown in Fig. 14, the environment information 1455 may include any information related to the surrounding environment, such as the lighting conditions of the surrounding environment, the position of the image capturing unit, and the distance between the user and the pointing object.

This environmental information 1455 may be input directly by the user, or may be determined by the machine learning unit 1414 analyzing the input video captured by the imaging unit.
For example, when the environmental information 1455 is directly input by a user, the reception unit 1412 may receive the environmental information 1455 input via the user interface 1410 of the user terminal 250 via the communication network 234, and then store the received environmental information 1455 in the environmental information database 1450. The user interface 1410 here may be, for example, a web page or an application available from the user terminal.
On the other hand, when the environmental information 1455 is determined by the machine learning unit 1414, the machine learning unit 1414 analyzes an input video acquired by a shooting unit (not shown in FIG. 14 ) using a machine learning means that has been trained to execute predetermined object detection and image processing, thereby determining the environmental information 1455. Thereafter, the machine learning unit 1414 may store the determined environmental information 1455 in the environmental information database 1450.

After the environmental information 1455 is acquired and stored in the environmental information database 1450, the object detection unit, image processing unit, gesture determination unit, pointing direction determination unit, and/or pointing object identification unit described above may use this environmental information 1455 as appropriate. This makes it possible to improve the accuracy of the various processes executed by the object detection unit, image processing unit, gesture determination unit, pointing direction determination unit, and pointing object identification unit.

Next, the user information acquisition unit according to the second embodiment of the present disclosure will be described with reference to FIG. 15.

15 is a diagram illustrating an example of the configuration of the user information acquisition unit 1034 according to the second embodiment of the present disclosure. The user information acquisition unit 1034 is a functional unit for acquiring user information related to a user.
As shown in FIG. 15 , the user information acquisition unit 1034 may be connected to a user terminal 250 via a communication network 234 .

As shown in FIG. 15 , the user information acquisition unit 1034 includes a reception unit 1512 that receives user information, a machine learning unit 1514 that determines the user information from video, and a user information database 1550.
The user information database 1550 is a database for storing user information 1555 relating to users. As shown in Fig. 15, the user information 1555 may include any information relating to the user, such as an identifier for uniquely identifying the user, the name, height, skin color, assigned tasks, and current task.

This user information 1555 may be input directly by the user, or may be determined by the machine learning unit 1514 analyzing the input video acquired by the imaging unit.
For example, when the user information 1555 is directly input by the user, the reception unit 1512 may receive the user information 1555 input via the user interface 1510 of the user terminal 250 via the communication network 234, and then store the received user information 1555 in the user information database 1550. The user interface 1510 here may be, for example, a web page or an application available from the user terminal.
On the other hand, when the user information 1555 is determined by the machine learning unit 1514, the machine learning unit 1514 analyzes an input video acquired by a shooting unit (not shown in FIG. 14 ) using a machine learning means that has been trained to perform predetermined object detection and image processing, thereby determining the user information 1555. Thereafter, the machine learning unit 1514 may store the determined user information 1555 in the user information database 1550.

After the user information 1555 is acquired and stored in the user information database 1550, the object detection unit, image processing unit, gesture determination unit, pointing direction determination unit, and pointing object identification unit described above may use this user information 1555 as appropriate. This makes it possible to improve the accuracy of the various processes executed by the object detection unit, image processing unit, gesture determination unit, pointing direction determination unit, and pointing object identification unit.

Next, the update unit according to the second embodiment of the present disclosure will be described with reference to FIG. 16.

FIG. 16 is a diagram illustrating an example of the configuration of the update unit 1036 according to the second embodiment of the present disclosure. As described above, the update unit 1036 is a functional unit for updating the parameters of each functional unit of the pointed object detection device based on the environmental information and user information acquired by the environmental recognition unit and the user information acquisition unit.

As shown in FIG. 16, the update unit 1036 is connected to the above-mentioned environmental information database 1450 and user information database 1550, and is configured to be able to access environmental information 1455 and user information 1555.

The update unit 1036 acquires environmental information 1455 from the environmental information database 1450, acquires user information 1555 from the user information database 1550, and then updates the parameters of each functional unit of the pointed object detection device based on the acquired environmental information 1455 and user information 1555. The update unit 1036 may update each functional unit based on the environmental information 1455 and user information 1555 periodically (every minute, every five minutes, or every hour), or may update each time new information is added to the environmental information database 1450 or the user information database 1550.

In addition, the update based on the environmental information 1455 and the user information 1555 is a process for improving the accuracy and efficiency of the processing of each functional unit, and the parameters adjusted in this update may be determined appropriately according to the function and performance of each functional unit.
For example, when the environmental information 1455 indicates that the surrounding environment of the user and the pointing object is dark, the update unit 1036 may change the parameters of the image capturing unit to parameters suitable for a dark environment. This makes it possible to obtain a clearer input image, thereby improving the accuracy of object detection by the object detection unit.
As another example, the update unit 1036 may change the parameters of the image processing unit to parameters suitable for the skin color of the user, based on the skin color of the user recorded in the user information 1555. This can improve the accuracy of extraction of the hand image by the image processing unit.

The update unit 1036 configured as described above can update the parameters of each functional unit of the pointed object detection device using user information and environmental information, thereby improving the accuracy and efficiency of processing by each functional unit.

Next, the configuration of a pointing object detection system according to Example 3 of the present disclosure will be described with reference to FIG. 17.

FIG. 17 is a diagram showing the configuration of a pointed object detection system 1700 according to a third embodiment of the present disclosure. A pointed object detection device 1710 of the pointed object detection system 1700 according to the third embodiment of the present disclosure differs from the pointed object detection device 1010 of the pointed object detection system 1000 according to the second embodiment of the present disclosure shown in FIG. 10 in that the pointed object detection device 1710 includes an operation management unit 1740. Except for the inclusion of the operation management unit 1740, the configuration of the pointed object detection system 1700 according to the third embodiment is substantially similar to that of the pointed object detection system 1000 according to the second embodiment, and therefore the same parts are denoted by the same reference numerals. For convenience of explanation, the following description will omit the above-mentioned parts and focus on the differences between the pointed object detection system 1700 and the pointed object detection system 1000.

The work management unit 1740 is a functional unit for managing work performed by workers such as users. For example, the work management unit 1740 may manage the allocation of work to workers and the distribution of work based on, for example, user object sequence information acquired by the object management unit 1030. As one example, the work management unit 1740 may determine the progress of work of each worker based on the user object sequence information, and allocate work according to this progress. As another example, the work management unit 1740 may determine the efficiency of each work of each worker based on the user object sequence information, and allocate work according to this efficiency of each work.
The details of the processing performed by the operation management unit 1740 will be described later, and therefore will not be described here.

The pointed object detection system 1700 according to the third embodiment of the present disclosure described above can detect the pointed object 240 that is the target of the user's pointing action, and can manage the distribution of work according to the work status of each worker determined by the user object sequence information.

Next, the work distribution process performed by the work management unit according to the third embodiment of the present disclosure will be described with reference to FIG.

Figure 18 is a diagram showing the flow of work distribution processing 1800 by the work management unit according to the third embodiment of the present disclosure. The work distribution processing 1800 shown in Figure 18 is a process for distributing work to users, and is performed by the work management unit 1740.

First, in step S1810, the work management unit 1740 acquires user object sequence information from the above-mentioned object sequence information database (for example, the object sequence information database 1120 shown in FIG. 11 ), and determines the progress of the target work based on the acquired user object sequence information. As described above, the user object sequence information indicates information on the pointing objects designated by a specific user for each task of the target work in the order in which they were designated. Therefore, it is possible to determine the progress of the work currently in progress by comparing the user object sequence information with the designated object sequence information indicating the correct order of actions for the work.
For example, the work management unit 1740 may compare the user object sequence information obtained from the object sequence information database with the specified object sequence information to determine the completion rate of the work (30%, 54%, 70%, etc.) as information indicating the progress of the work currently in progress.
In step S1810, the work management unit 1740 acquires at least user object sequence information for determining the progress of the work, but the present disclosure is not limited to this, and the work management unit 1740 may acquire other information such as the above-mentioned user information and environmental information in addition to the user object sequence information. As described later, in addition to the user object sequence information, the acquired user information and environmental information may be used when distributing the work.

Next, in step S1820, the work management unit 1740 may distribute the work based on the progress of the work determined in step S1810 and the work distribution rules 1825 stored in the work management unit 1740. More specifically, here, the work management unit 1740 may assign a newly generated new work to a specific user (first user) based on the progress of the work and the work distribution rules 1825, and may redistribute a work that has already been assigned to a specific user (first user) to another user (second user).
The work distribution rules 1825 are rules that stipulate a method of distributing work. In other words, the work distribution rules 1825 are information that specifies what work should be assigned, at what timing, and to what user. For example, the work distribution rules 1825 may include a rule that "if the completion rate of the user's current work is 70% or more, assign a new work" or a rule that "if the completion rate of the user's current work is 20% or less, do not assign a new work".
Furthermore, the work distribution rules 1825 may include rules that consider the above-mentioned user information and environmental information in addition to the current progress of the work. For example, the work distribution rules 1825 may determine the distribution of work based on the presence or absence of a user's qualifications determined from the user information, or may determine the distribution of work based on the certification conditions of the environment, etc.

Next, in step S1830, the work management unit 1740 may update the user information database (e.g., the user information database 1550 shown in FIG. 15) and the object sequence information database (e.g., the object sequence information database shown in FIG. 11 and FIG. 12) after distributing the work. For example, the work management unit 1740 may update the content of the work for which each user is responsible in the user information database and the object sequence information database in accordance with the distribution of work performed in step S1820.

According to the task distribution process 1800 described above, tasks can be efficiently distributed to users using information (user information, environmental information, user object sequence information) acquired by the pointed object detection method according to the embodiment of the present disclosure.

Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the gist of the present invention.
The configurations of the object management unit and the environment recognition unit, and the items to be managed are not limited to those described above, and various modifications are possible.

200, 1000, 1700 Pointing object detection system 210, 1010, 1710 Pointing object detection device 212 Object detection unit 214 Image processing unit 218 Gesture determination unit 220 Pointing direction determination unit 222 Pointing object identification unit 224 Image input unit 226 Processor 228 Storage unit 234 Communication network 235 User 240 Pointing object 250 User terminal 1030 Object management unit 1032 Environment recognition unit 1036 Update unit

Claims (10)

A pointing object detection device that detects a pointing object that is a target of a pointing action,
an image input unit that acquires an input image including a user's hand and at least one pointing object candidate;
an object detection unit that analyzes the input video and detects the hand and the pointing object candidate;
a gesture determination unit for determining a gesture made by the hand;
If the gesture by the hand is determined to be a pointing gesture,
Identifying a centroid of the hand;
Identifying a region that satisfies a first distance criterion from the centroid and also satisfies a predetermined curvature criterion as the fingertip of the index finger used in the pointing motion;
a pointing direction determination unit that determines a pointing direction of the pointing motion based on the identified distribution of pixels belonging to the fingertip;
a pointing object identification unit that identifies an object pointed to by the pointing motion in the input video from among the pointing object candidates based on the pointing direction;
Including,
The instruction direction determination unit
In a hand image extracted from the input video and showing only the hand,
determining a region of interest centered on the fingertip based on a distance between the fingertip and the centroid;
Identifying pixels in the target region that satisfy a predetermined luminance standard as finger pixels belonging to the index finger;
Calculating the average pixel coordinates of the finger pixels;
determining a direction of an imaginary line passing through the average pixel coordinates and the fingertip as the pointing direction;
A pointing object detection device comprising: The instruction direction determination unit
In a hand image extracted from the input video and showing only the hand,
setting a first imaginary line passing through the fingertip and the centroid;
setting a second imaginary line that is perpendicular to the first imaginary line and satisfies a second distance criterion with respect to the fingertip;
setting a third imaginary line that is perpendicular to the first imaginary line, parallel to the second imaginary line, and satisfies a third distance criterion with respect to the fingertip;
determining a direction of a fourth imaginary line passing through a center point of the second imaginary line and a center point of the third imaginary line as the indicated direction;
2. The pointing object detection device according to claim 1, wherein the pointing object detection device is a pointing object detection device. The instruction direction determination unit
In a hand image extracted from the input video and showing only the hand,
determining a region of interest centered on the fingertip based on a distance between the fingertip and the centroid;
Identifying pixels in the target region that satisfy a predetermined luminance standard as finger pixels belonging to the index finger;
determining the pointing direction based on the finger pixels using a least squares method;
2. The pointing object detection device according to claim 1, wherein the pointing object detection device is a pointing object detection device. The instruction direction determination unit
In a hand image extracted from the input video and showing only the hand,
determining a region of interest centered on the fingertip based on a distance between the fingertip and the centroid;
Identifying pixels in the target region that satisfy a predetermined luminance standard as finger pixels belonging to the index finger;
determining the pointing direction based on the finger pixels using principal component analysis;
2. The pointing object detection device according to claim 1, wherein the pointing object detection device is a pointing object detection device. The pointing object detection device is
Further comprising an object management unit for managing a handling procedure of the pointing object,
The object management unit
creating and storing user object sequence information indicating an order in which the plurality of pointing objects are pointed to by the user's pointing action;
2. The pointing object detection device according to claim 1, wherein the pointing object detection device is a pointing object detection device. The object management unit
storing designated object sequence information that designates an order in which the plurality of designated objects are to be handled;
verifying the user object sequence information by comparing the user object sequence information with the designated object sequence information;
If the user object sequence information does not satisfy a predetermined similarity criterion with respect to the designated object sequence information,
determining that an abnormality has occurred in the handling procedure of the pointing object, and outputting an abnormality notification indicating the abnormality;
6. The pointing object detection device according to claim 5 , The pointing object detection device is
A user information acquisition unit that acquires user information related to the user;
an environment recognition unit that acquires environment information related to a surrounding environment of the user;
an update unit that updates at least one parameter of the image input unit, the object detection unit, the gesture determination unit, the pointing direction determination unit, the pointing object identification unit, and the object management unit based on the user information or the environmental information;
The pointing object detection device according to claim 5 , further comprising: The pointing object detection device is
a task management unit for managing tasks performed by the user;
The work management unit includes:
determining a progress status of a first task assigned to a first user based on the user object sequence information;
determining a second user who will be in charge of a second task based on the determined progress status of the first task and a task distribution rule that specifies a task distribution method, and distributing the second task to the second user;
7. The pointing object detection device according to claim 6 , A pointing object detection method for detecting a pointing object that is a target of a pointing action, comprising:
acquiring an input video including a user's hand and at least one potential pointing object;
analyzing the input video to detect the hand and the pointing object candidates;
extracting a hand image showing only the hand from the input video;
analyzing the hand image to determine a gesture made by the hand;
determining a centroid of the hand when the gesture by the hand is determined to be a pointing motion, and determining, as the fingertip of the index finger, an area that satisfies a first distance criterion from the centroid and also satisfies a predetermined curvature criterion;
determining a region of interest centered on the fingertip based on a distance between the fingertip and the centroid;
identifying pixels in the target region that satisfy a predetermined luminance standard as finger pixels belonging to the finger used in the pointing motion;
calculating average pixel coordinates of the finger pixels;
determining a direction of an imaginary line passing through the average pixel coordinates and the fingertip as a direction of the pointing motion;
identifying an object pointed to by the pointing motion in the input video from among the pointing object candidates based on the pointing direction;
A pointing object detection method comprising: A pointing object detection system for detecting a pointing object that is a target of a pointing action, comprising:
The pointing object detection system includes:
A pointing object detection device;
a user terminal;
the pointing object detection device and the user terminal are connected via a communication network;
The pointing object detection device is
an image input unit that acquires an input video showing a user's hand and at least one pointing object candidate;
an object detection unit that analyzes the input video and detects the hand and the pointing object candidate;
a gesture determination unit for determining a gesture made by the hand;
If the gesture by the hand is determined to be a pointing gesture,
identifying a centroid of the hand, and identifying a region that satisfies a first distance criterion from the centroid and also satisfies a predetermined curvature criterion as the fingertip of the index finger used in the pointing motion;
a pointing direction determination unit that determines a pointing direction of the pointing motion based on the identified distribution of pixels belonging to the fingertip;
a pointing object identification unit that identifies an object pointed to by the pointing motion in the input video from among the pointing object candidates based on the pointing direction;
an object management unit that creates user object sequence information indicating the order in which the multiple pointing objects are pointed to by the user's pointing action based on the identified pointing object, verifies the user object sequence information by comparing the user object sequence information with designated object sequence information that specifies the order in which the multiple pointing objects are handled, and outputs an abnormality notification to the user terminal when the user object sequence information does not satisfy a predetermined similarity standard with respect to the designated object sequence information,
The instruction direction determination unit
In a hand image extracted from the input video and showing only the hand,
determining a region of interest centered on the fingertip based on a distance between the fingertip and the centroid;
Identifying pixels in the target region that satisfy a predetermined luminance standard as finger pixels belonging to the index finger;
Calculating the average pixel coordinates of the finger pixels;
determining a direction of an imaginary line passing through the average pixel coordinates and the fingertip as the pointing direction;
A pointing object detection system comprising:

Images