JP5459385B2 - Image display apparatus and indicator image display method - Google Patents

Description

  The present invention relates to an image display device and a method for displaying a pointer image.

JP 2000-335330 A Japanese Unexamined Patent Publication No. 2000-6687 Japanese Patent Laid-Open No. 2004-26046

  Conventionally, as an operation device for an in-vehicle device such as a car navigation device, an operator's hand is photographed with a camera, a finger image is extracted from the photographed data, and the extracted finger image area is displayed on a GUI input screen of the in-vehicle device (for example, A type of display that is superimposed on a navigation screen) has been proposed.

  For example, in Patent Document 1, a camera attached to the ceiling of a vehicle body is used to photograph the state of a hand operating a switch panel at the seat, and the photographed hand and the switch panel are placed opposite to the front of the operator. A configuration for displaying on a liquid crystal panel is disclosed.

  Patent Document 2 discloses a configuration in which a driver's hand is photographed using a video camera arranged on the roof of a vehicle to extract a contour, and the contour image is superimposed on a soft button on a display. Yes.

  Furthermore, Patent Document 3 discloses a configuration in which an operation button and a hand on a switch matrix are imaged, a touch sensor detects that the hand has approached the switch matrix, and a hand image is superimposed and displayed. Has been.

  In the above-described conventional operation system, the size of the operation surface has been reduced for the purpose of saving the mounting space on the vehicle and reducing the amount of finger movement during operation. In this case, the following problem occurs. That is, the operation surface and the input screen (screen region) have a unique coordinate relationship such that the input position on the operation surface by the finger and the indicated position on the screen correspond geometrically. As a result, when the operator's hand (finger) image captured on the operation surface is displayed superimposed on the input screen, if the operation surface size is reduced relative to the screen size, the hand display size is input. There is a problem that the image is enlarged on the screen, and the finger is displayed excessively thick, causing a sense of incongruity.

  FIG. 14 shows an example in which a virtual finger operation input to a soft button on the input screen 15 by a touch panel arranged on the operation surface is supported by a hand image SF superimposed on the screen 15. Yes. The operation surface 102a has a width of about 120 mm, and has a size that can accommodate the entire four fingers excluding the thumb among the palm of an ordinary adult. The corresponding screen 15 displays a Japanese syllable keyboard for keyword input, and the number of soft buttons arranged at most with respect to the image width of one finger, assuming the above-mentioned relationship between the operation surface and the hand. The display dimensions and arrangement intervals of the individual buttons are adjusted so that there are about two. The denseness of the buttons with respect to the fingertip portion of the hand image SF does not occur so much, and input without a sense of incongruity is possible.

  On the other hand, FIG. 15 shows a display example when the width of the operation surface 102a is reduced to 70 mm without changing the dimensions of the screen 15. Unlike the case of FIG. 14, the number of fingers FI that are accommodated on the operation surface 102 a is about two, and these are relatively enlarged and displayed on the screen 15. As a result, it can be seen that the number of soft buttons corresponding to the image width of one finger exceeds three, and the denseness of the buttons with respect to the hand image SF is remarkably increased. This makes it very difficult to understand which of the multiple buttons on the screen is being operated, and it is inevitable that the operability will be deteriorated. Become. Of course, even if the input screen is not a keyboard, for example, a map screen or the like on which a position designation input with a pointer can be performed, the operability is exactly the same.

  In this case, it is conceivable that the display content of the input screen (for example, the above-described soft button) is enlarged in accordance with the finger image. However, the enlargement of the monitor size may increase the size of the display device or change the mounting layout (particularly, In the case of mounting, it is necessary to change the design of the interior resin molded product that is the mounting destination, such as an instrument panel, and it is difficult to avoid an increase in cost. On the other hand, if only the display contents are expanded without changing the monitor dimensions, it will be necessary to divide and scroll the input screen contents that were previously contained in one screen, which will lead to unnecessary development man-hours such as deterioration in operability and changes in display application contents. Is required.

  Therefore, a method of processing a photographed finger image so as to be thinner than actual and superimposing it on an input screen is conceivable. As a specific method of such a fingering process, for example, a method based on image shrinkage processing can be exemplified. For example, in the case of a binary image, when all the pixels in the closed processing target area on the screen are set to the same value (for example, “1”), the pixel having a certain adjacent relationship with the target pixel (For example, 2 pixels on the left and right or top and bottom, 4 pixels on the top, bottom, left, and right, and 8 pixels including 4 pixels diagonally adjacent to this) includes even one pixel whose value is in an inverted relationship with the pixel of interest. If so, this is a process of inverting the value of the pixel of interest. Since the pixels that satisfy these conditions are usually pixels that form the contour line of the region, the pixels along the contour line are uniformly inverted so that the region to be processed shrinks uniformly by one pixel inside. Can do.

  In the contraction process as described above, when finger images separated from each other are obtained as shown in FIG. 15, individual fingers can be finely divided independently. However, as shown in FIG. 16, in the state where the fingers are in close contact with each other, the pixels in the background area that form the contrast of the finger outline are absent in the close contact section, and in this section, two fingers are recognized as an integrated area. Therefore, there is a problem that individual fine fingering cannot be performed. In addition, in a loose contact state in which a contour section in which fingers and fingers can be separately recognized is intermittently formed, image shrinkage proceeds non-uniformly only in the separated and recognized contour sections. However, there is a problem that the non-inverted pixel region remains in a bridging manner, and only an unnatural fingering result in which the sticking portion and the distant portion are mixed is obtained.

  An object of the present invention is to reliably display an image showing an individual finger on a screen in a form narrower than the original finger image regardless of the state of the original finger image on image data acquired by photographing or the like. A method and apparatus and an operation apparatus using the method and apparatus are provided.

In order to solve the above-described problems, an image display device according to the present invention includes image data acquisition means for acquiring image data including an image of a human finger as an original finger image, and a fingertip portion of the original finger image on the image data. An image fingertip point specifying means for specifying the position of the image fingertip point, a display device having a screen area having a unique coordinate correspondence with the image data, and a position corresponding to the image fingertip point of the screen area An indicator image display means for displaying an indicator image prepared separately from the image data,
The indicator image is a simulated finger image imitating the contour shape of a finger or a real finger image of a person captured and obtained by separating individual fingers, and the screen area is a finger image including the finger tip of the indicator image. The indicator image display means displays only a part in the longitudinal direction, and the wrist specified point setting means for setting a wrist specified point on a display coordinate plane that defines the display position of the indicator image on the screen area; The indicator arrangement direction determining means for determining the arrangement direction of the indicator image based on the wrist prescribed point and the image fingertip point,
When the image data includes a plurality of original finger images, the image fingertip point specifying means specifies the image fingertip points for each of the plurality of original finger images, and the wrist prescribed point setting means A common wrist prescribed point is set for a plurality of image fingertip points identified by the fingertip point identifying means, and the wrist prescribed point setting means further sets the vertical direction of the screen area in the Y direction on the display coordinate plane. When the direction orthogonal to this is the X direction, the reference wrist prescribed point is fixedly defined outside the screen area in the Y direction, and a plurality of image fingertip points are specified on the screen area, the Y on the screen area When the original finger image in which the image fingertip point is located on the uppermost side in the direction is the representative original finger image, the greater the inclination angle of the representative original finger image with respect to the Y direction, the greater the X of the wrist reference point from the reference wrist prescribed point. So that the displacement in the direction increases For a representative original finger image that is tilted to the right with respect to the Y direction, the X coordinate value of the wrist prescribed point is determined so as to be displaced to the left with respect to the reference wrist prescribed point, and the left upward form with respect to the Y direction is determined. For the tilted original finger image, the X coordinate value of the wrist prescribed point is determined so as to be displaced to the right with respect to the reference wrist prescribed point.
The indicator arrangement direction determining means determines the arrangement direction for each of the plurality of image fingertip points based on the wrist prescribed point, and the plurality of indicator images are arranged so that the longitudinal direction of each indicator image is along the arrangement direction. And displaying on the screen area.

  According to the present invention, the position of the fingertip portion of the original finger image in the acquired image data is specified as the image fingertip point, and the image of the original finger image is located at the position corresponding to the image fingertip point on the screen area of the display device. A pointer image different from the data is displayed. That is, the original finger image is used only for specifying the user's designated position, and a separately prepared indicator image is pasted and displayed on the screen area. As a result, regardless of the state of the original finger image on the image data, the indicator image indicating each finger can be reliably displayed on the screen in a form narrower than the original finger image. It is possible to prevent a sense of incongruity caused by the image being displayed excessively thick.

  In addition, even when a plurality of original finger images appear on the image data, a plurality of indicator images are combined and displayed in an alternative manner so that the plurality of fingers come into close contact with each other on the image data. Even when it is difficult to separate them, it is possible to achieve fine fingering on the display without problems.

  Thereby, the operator can grasp the finger position on the input operation surface from the indicator image on the screen, and can accurately perform the input operation according to the screen display content. In particular, a configuration in which the screen area (screen) of the display device is arranged with respect to the operator so as to be out of the direction in which the operator directly looks at the finger on the input operation surface (for example, an in-vehicle electronic device such as a car navigation system) In this case, the input operation surface is disposed on the left and right sides (or diagonally forward) of the seat on which the operator is seated, and the screen of the display device is above the input operation surface and forward or diagonally to the operator. In the configuration facing the front), both the hand performing the operation and the screen cannot be directly viewed at the same time. Therefore, the indicator image on the screen is the only information source for grasping the hand operation position. Further, by adopting the present invention, even in a configuration in which the area of the input operation surface (shooting range of the hand-held camera) is reduced relative to the area of the screen area, the photographed original finger image is enlarged and displayed as is Thus, it is possible to effectively solve the problems that hinder operability.

  According to the present invention, since the indicator image is pasted on the image fingertip point specified from the original finger image with respect to the screen area, it is necessary to specify the finger direction as information in addition to the fingertip position. On the premise of this, the indicator image display means includes a wrist prescribed point setting means for setting a wrist prescribed point on a display coordinate plane that defines a display position of the indicator image, a set wrist prescribed point and an image fingertip point, The indicator arrangement direction determining means for determining the arrangement direction of the indicator image based on the indicator image, the position of the tip of the indicator image coincides with the image fingertip point, and the indicator image is arranged in the determined arrangement direction. The indicator image is displayed on the screen area so that the longitudinal direction of the indicator is along.

  In addition to the finger tip position, another point in the longitudinal direction of the finger is designated as the wrist reference point, so that a straight line connecting both points (hereinafter also referred to as a finger straight line) is simple and unambiguous as the indicator image placement direction. Can be determined.

  In addition, since the phalanges of the hand are arranged so as to converge toward the wrist joint, by using the position corresponding to the wrist as a wrist reference point, a pointer image (especially a simulated finger image or an actual finger) is used. Image) can be arranged in a realistic form close to the actual hand. The wrist specified point is configured to be set outside the screen area.

  Here, when analyzing the movement of the hand and arm of the operator when changing the operation position in the input operation surface, when moving the fingertip in the X direction, there are some differences depending on the operator's heel, The movement of the elbow joint is relatively small, and in many cases, the movement of moving the fingertip with the sense of rotating the palm around an axis located near the center of the palm is predominant, and the finger direction angle is also the rotation angle of the palm As a result, it changes in accordance with the X coordinate of the image fingertip point.

  Therefore, the wrist specified point setting means sets the vertical direction (vertical direction) of the screen area on the display coordinate plane as the Y direction, and the direction orthogonal to this as the X direction (horizontal direction or horizontal direction) below the screen area. The wrist reference point is fixedly determined, and the wrist reference point setting means is configured so that the displacement in the X direction from the reference wrist reference point increases as the inclination angle of the original finger image with respect to the Y direction increases. When configured to determine the X coordinate value of a point, it is possible to display a pointer image that more realistically replicates the hand movement pattern in the X direction.

  As described above, assuming that the center of rotation exists inside the palm, as the fingertip rotates in the X direction, the wrist position rotates in the opposite direction. Therefore, for the original finger image that is inclined to the right with respect to the direction, the X coordinate value of the wrist prescribed point is determined so as to be displaced to the left with respect to the reference wrist prescribed point in the X direction. The wrist specified point setting means is configured to determine the X coordinate value of the wrist specified point so as to be displaced to the right side with respect to the reference wrist specified point in the X direction for the original finger image that is inclined to the left upward form. It is reasonable to do.

  In addition, the wrist specified point of the present invention can be set at a position spaced apart from the lower edge of the screen area by a predetermined length in the Y direction. This method has an advantage of simplifying the wrist definition point determination algorithm because the Y direction position of the wrist specification point is fixedly determined with reference to the lower edge of the screen area. In addition, when the image fingertip point exists within a predetermined range on the screen that matches the actual hand size of the operator as viewed from the wrist specified point, the reality in the arrangement direction of the indicator image is relatively good. It can be secured.

  However, when an input screen (for example, a keyboard screen or a map screen) in which the input position in the Y direction can be changed greatly is adopted, the wrist position can also change greatly in the Y direction, so the wrist on the display coordinate plane can be changed. It is desirable to set the specified point so as to have a predetermined positional relationship with the specified image fingertip point in the Y direction in order to increase the reality in the arrangement direction of the indicator image. Specifically, on the premise that a palm image including a finger in the screen area is displayed in a positional relationship where the wrist position is deviated below the screen area, the wrist specified point setting means is displayed on the display coordinate plane. When the wrist prescribed point is set at a position spaced a certain length downward from the image fingertip point specified in the screen area in the Y direction, where the vertical direction of the screen area is the Y direction and the direction orthogonal thereto is the X direction. Good.

  In addition, the image display device of the present invention includes a transparent input support plate that receives a touch operation by a human finger while supporting one finger surface side, and a position input device that detects the touch operation and outputs coordinate information thereof. Further, the image data acquisition means may be configured by a camera arranged from the other plate surface side of the transparent input support plate. Thereby, the shooting distance to the hand-held camera can always be kept constant. Further, the transparent support plate may be configured as a position input device in which an input operation surface corresponding to the shooting range of the hand-photographing camera is set to the plate surface.

  In order to solve the above problem, the present invention acquires image data including an image of a human finger as an original finger image, and uses the position of the fingertip portion of the original finger image as an image fingertip point on the image data. The indicator image prepared separately from the image data is provided at a position corresponding to the image fingertip point of the screen area of the display device having a screen area having a unique coordinate correspondence with the specified image data. A display method of a pointer image to be displayed, wherein the pointer image is a simulated finger image imitating a contour shape of a finger or a real finger image of a person photographed and acquired by separating individual fingers, and a screen region Is for displaying only a part of the indicator image in the longitudinal direction of the finger including the tip of the finger. When the indicator image is displayed in the screen area, the display for defining the display position of the indicator image in the screen area is displayed. Set the wrist reference point on the coordinate plane, the wrist reference point and the image fingertip point In the method for determining the arrangement direction of the indicator image based on the image data, when the image data includes a plurality of original finger images, the image fingertip points are specified for each of the plurality of original finger images, and the image fingertip points are specified. A common wrist prescription point is set for a plurality of image fingertip points specified by the means, and the vertical direction of the screen area on the display coordinate plane is defined as the Y direction, and the direction orthogonal thereto is defined as the X direction. When a point is fixedly fixed outside the screen area in the Y direction and a plurality of image fingertip points are specified on the screen area, an original finger image in which the image fingertip point is located on the uppermost side in the Y direction on the screen area is displayed. When the representative original finger image is used, the larger the angle of inclination of the representative original finger image with respect to the Y direction, the greater the amount of displacement of the wrist reference point from the reference wrist prescribed point in the X direction. Representative inclined to the rising form For the finger image, the X coordinate value of the wrist prescription point is determined so as to be displaced to the left with respect to the reference wrist prescription point, and for the original finger image that is tilted to the left in the Y direction, the reference wrist prescription point The X coordinate value of the wrist prescribed point is determined so as to be displaced to the right side, the arrangement direction is determined based on the wrist prescribed point for each of the plurality of image fingertip points, and a plurality of indicator images are assigned to the indicator images. Is displayed on the screen area so that the longitudinal direction of each is along the arrangement direction.

  According to the present invention, the indicator image indicating each finger can be reliably displayed on the screen in a form narrower than the original finger image, and as a result, the finger image is displayed with an excessively thick feeling. Occurrence can be prevented.

The perspective view which shows an example of the vehicle interior installation form of the vehicle-mounted electronic device operating device used as the application object of this invention. The cross-sectional schematic diagram which shows an example of the internal structure of an operation part. The block diagram which shows an example of the electrical constitution of the vehicle-mounted electronic device operating device of FIG. Explanatory drawing which shows the flow of the image processing which determines an image fingertip point. The figure explaining the reference example which determines the Y coordinate of a wrist prescription | regulation point. Explanatory drawing which shows the state which affixed the indicator image along the finger straight line determined in FIG. The figure explaining another reference example which determines the Y coordinate of a wrist prescription | regulation point. The figure explaining the Example of this invention which determines X coordinate of a wrist prescription | regulation point. The figure explaining the reference example which determines the X coordinate of a wrist prescribed point. The figure which shows the 1st example of a pointer image. The figure which shows a 2nd example similarly. The figure which similarly shows a 3rd example. The figure which shows the example of a screen display which superimposed the indicator image. The figure which shows the positional relationship of the finger image and input operation surface when an input operation surface is made into a normal dimension, and the example of the conventional screen display on which this finger image was superimposed. The figure which shows the example of the conventional screen display which superimposed the positional relationship of the finger image and input operation surface at the time of making an input operation surface into a reduced size, and this finger image. The figure explaining the problem of contraction processing.

DESCRIPTION OF SYMBOLS 1 ... Operation apparatus (image display apparatus) for vehicle-mounted electronic devices, 10 ... Operation ECU, 12 ... Input part, 12a ... Touch panel (position input device), 12b ... Hand photographing camera (image) (Data acquisition means), 15 ... monitor (display device), 101 ... CPU (image fingertip point specifying means, wrist prescribed point setting means, pointer arrangement direction determining means), FI ... original finger image, SF ... Indicator image, G ... Image fingertip point.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 shows an example of an operating device for an in-vehicle electronic device to which the image display device of the present invention is applied. The operation device 1 has a monitor (display device) 15 disposed in the center of the instrument panel in the interior of the automobile, and the operation unit 12 is located at a position where the center console C can be operated from either the driver seat 2D or the passenger seat 2P. Is arranged. Although the purpose of use is not particularly limited, for example, it is for performing functional operations of the car navigation device and the car audio device while looking at the screen of the monitor 15 provided in the center console.

  The operation unit 12 is attached so that the input operation surface faces upward. The input operation surface is formed by a known touch panel 12a, and any of a resistive film method, a surface acoustic wave method, and a capacitance method may be adopted. The base material of the touch panel 12a is a transparent input support plate made of a transparent resin plate, a glass plate, or the like, and accepts a touch operation by the operator's fingertip while supporting it on the upper surface side. Then, an input coordinate system corresponding to the screen area of the monitor 15 is set along the plate surface.

  FIG. 2 is a cross-sectional view schematically showing the internal structure of the input unit 12. The above-described touch panel 12a is fitted on the upper surface of the housing 122e so that the input operation surface 102a side is up. In the housing 12d, a hand photographic camera 12b is housed together with an illumination light source 12c and a photographic optical system, and constitutes an image data acquisition unit. The illumination light source 12c is a light emitting diode light source (ie, a monochromatic light source) having a luminance directivity directly above the element by a convex curved mold, and a plurality of illumination light sources 12c are arranged so as to surround the lower surface of the touch panel 12a. Each illumination light source 12c is attached by tilting the mold front end side with high brightness toward the inside of the lower surface of the touch panel 12a, and the primary imaging reflected light RB1 by the palm surface of the operator's hand H on the input operation surface 102a is It is generated in a downward direction so as to pass through the touch panel 12a.

  The photographing optical system has a first reflecting portion 12p and a second reflecting portion 12r. The first reflecting portion 12p is a prism plate (transparent material plate in which triangular prism-shaped microprisms are arranged in parallel in the plane: hereinafter also referred to as a prism plate 12p) and is disposed directly below the touch panel 12a. By reflecting the primary imaging reflected light RB1 from the hand H obliquely upward, it is guided as the secondary imaging reflected light RB2 to the lateral outside of the facing space 12f between the prism plate 12p and the touch panel 12a. The second reflecting portion 12r is a plane reflecting mirror (hereinafter also referred to as a plane reflecting mirror 12r) disposed on the lateral side outside the facing space, and reflects the secondary imaging reflected light RB2 to the side, thereby The light is guided as the tertiary imaging reflected light RB3 to the hand photographing camera 12b located on the opposite side of the second reflecting portion 12r across the space 12f. The hand photographing camera 12b is provided at a position corresponding to the focal point of the tertiary imaging reflected light RB3, and captures and acquires image data of the hand H including the operator's finger.

  As shown in an enlarged view in FIG. 2, the prism plate 12 p is formed of rib-shaped microprisms each having a reflecting surface inclined at an equal angle with respect to the mirror base surface MBP and parallel to each other along the mirror base surface MBP. Even if the mirror base surface MBP is not inclined, the light incident in the normal direction can be reflected to the (oblique) side. Accordingly, the first reflecting portion 12p for side reflection can be disposed in parallel with the lower side of the touch panel 12a, and the height direction dimension of the facing space 12f can be greatly reduced.

  In addition, the second reflecting portion 12r and the hand photographic camera 12b are disposed facing each other with the opposing space 12f sandwiched from the side, so that the tertiary imaging reflected light RB that directly enters the hand photographic camera 12b crosses the opposing space 12f. Can be guided in shape. As a result, the second reflecting portion 12r and the hand photographing camera 12b can be disposed close to the side edge of the touch panel 12a, and the incident path of the imaging reflected light from the hand H to the hand photographing camera 12b is within the facing space 12f. In other words, since it is folded into a three-fold shape, the entire imaging optical system is greatly reduced in size and the casing 12d is made thinner. In particular, by reducing the size of the touch panel 12a (that is, the vertical and horizontal dimensions of the input operation surface 102a), the entire input unit 12 can be dramatically reduced in size and thickness, and the width of the center console unit C in FIG. It can be mounted on a relatively small vehicle or a vehicle that can utilize only a limited installation space in front of the shift lever.

  In the optical system described above, the imaging light is repeatedly reflected an odd number of times (hand H, first reflecting portion 12p, and second reflecting portion 12r) before entering the hand photographing camera 12b. Since the hand H is photographed from the palm surface side, a hand image appears in the image data acquired by the hand photographing camera 12b in the form of mirror reversal of the hand H viewed from the palm surface side. Since the contour line of the hand image coincides with the contour line when the hand H is viewed from the back side, the image data acquired by the hand-capturing camera 12b is eventually sent to the monitor 15 without performing mirror reversal processing. Can be used for display.

  The input operation surface 102a of the touch panel 12 corresponds to the photographing range 102b of the hand photographing camera 12b, and when assuming an average hand of an adult, only a part of the longitudinal direction including the tip of the middle finger is included. The vertical direction (Y direction) dimension is determined so as to enter the input operation surface 102a (for example, 60 to 90 mm (as a specific example, 75 mm)). As a result, only the portion ahead of the base of the finger is displayed in the screen area of the monitor 15, so that the shape of the palm portion other than the finger is not involved in the display, and will be described later using the indicator image. The display process can be greatly simplified. In addition, the horizontal dimension (X direction) of the input operation surface 102a is 110 to 130 mm (for example, 120 mm), and the index finger, middle finger, ring finger, and little finger are within the imaging range when a hand is put on and each finger is wide open. The thumb is out of the shooting range. However, it is also possible to keep all fingers within the imaging range if each finger is properly closed.

  FIG. 3 is a block diagram showing an electrical configuration of the controller device 1. It is the operation ECU 10 that controls the operation device 1. The operation ECU 10 is configured as a computer hardware board mainly composed of the CPU 101. Specifically, the CPU 101, the RAM 102, the ROM 103, the graphic controller 110, the video interface 112, the touch panel interface 114, the general-purpose input / output unit 104, and the serial communication interface. 116 has a structure interconnected by an internal bus 105. The graphic controller 110 includes a display video RAM 111 and a monitor 15, the video interface 112 includes a shooting video RAM 113 and a hand shooting camera 12 b, the touch panel interface 114 includes a touch panel 12 a, and the general-purpose input / output unit 104 includes a driver ( The illumination light source 12c is connected via a drive circuit 115. The serial communication interface 116 is connected to an in-vehicle serial communication bus 30 such as a CAN communication bus, and communicates with other ECUs connected to the network, specifically, a navigation ECU 200 that controls the car navigation device. It is possible.

  The video interface 112 continuously receives an analog or digital video signal captured and acquired by the hand photographic camera 12b, and is captured as image frame data in the photographic video RAM 113 at predetermined time intervals. The content stored in the video RAM 113 for shooting is updated whenever new image frame data is fetched.

  The graphic controller 110 acquires the input screen image frame data from the navigation ECU 200 via the communication interface 116, and also acquires the indicator image frame data in which the indicator image is pasted at a predetermined position from the CPU 101 side. Frame synthesis is performed on the video RAM 111 by a known alpha blending process or the like, and the result is output to the monitor 15.

  The touch panel interface 114 has a unique drive circuit corresponding to the method of the touch panel 12a, detects an input position by a touch operation on the input operation surface 102a based on a signal input state from the touch panel 12a, and displays the detection result. Output as position input coordinate information.

  It is to be noted that the photographing range of the hand photographic camera 12b (photographing field of view; as a result, image data photographed and acquired by the hand photographic camera 12b), the operation input surface of the touch panel 12b, and the screen area of the monitor 15 (and its display contents are determined) In the input screen image frame data and the indicator image frame data), a two-dimensional coordinate correspondence is uniquely defined.

  The ROM 103 stores the following software executed by the CPU 101. Thus, the CPU 101 functions as an image fingertip point specifying unit, a wrist specified point setting unit, and a pointer arrangement direction determining unit.

  Touch panel control software 103a: The input position coordinates are acquired from the touch panel interface 114, and the operation reference content determination reference information (for example, soft button region specifying information or the related information sent from the navigation ECU 200 together with the input screen image frame data) Information including operation command contents to be output when the soft button is touched. Then, the current operation input content is specified based on the input position coordinates and the acquired determination reference information, and a corresponding operation command execution command is output to the navigation ECU 200.

  Display control software 103b: Instructs the graphic controller 110 to input input screen image frame data, and transfers indicator image frame data created by a method described later to the graphic controller 110.

  Image fingertip point calculation software 103c: binarizes the image data of the operator's hand photographed by the hand photographic camera 12b, and performs calculation to specify the fingertip position of the original finger image as the image fingertip point (note that the image of the hand Data binarization processing may be performed in advance by incorporating a pixel binarization circuit in the output stage of the video interface).

  Indicator image composition software 103d: A process of pasting the indicator image created based on the indicator image data 103e on the indicator image frame so that the fingertip position coincides with the image fingertip point.

  Hereinafter, the operation of the controller device 1 will be described. The screen (area) of the monitor 15 is already similar to FIG. 14 or 15 (except for the hand image SF) by a preceding command input (for example, based on a touch input operation on another screen) (except for the hand image SF). Other input screens such as screens may be displayed). In this state, as shown in FIG. 2, when the hand H is brought close to the input operation surface 102a of the touch panel 12a, the hand photographing camera 12b captures a hand image based on the reflected light of the illumination light from the illumination light source 12c. To do. The pixel of the hand image appears brighter than the background area by receiving the reflected light. Therefore, as shown in FIG. 4A, if the luminance of a pixel is binarized with an appropriate threshold, an area showing a high luminance pixel value (here, “1”) is changed to a hand image area (black in the figure). The region indicating the low luminance pixel value (here, “0”) can be separated from each other as a background region (represented by white in the figure).

  Next, the binarized original image data is defined as the first image data (A), and the first image data is transferred from the first finger joint in a predetermined distance in the palm length direction (Y direction) of the hand image (for example, from the first joint of the middle finger Image data obtained by translating 20% to 80% (actual distance of about 5 to 20 mm) of the tip of the fingertip portion is defined as second image data shown in FIG. 4B. Then, as shown in FIG. 4C, the hand image non-overlapping region that appears on the fingertip side when the two image data are superimposed is specified as the fingertip region. By parallelly moving and superimposing the captured and acquired original image data in the palm length direction of the hand image, the fingertip region can be easily specified as the hand image non-overlapping region. In addition, even when some fingers are closed and in close contact, a rounded fingertip region can be reliably separated and specified.

  In C of FIG. 4, the second image data B is created by moving the first image data A backward in the palm length direction (Y direction) toward the wrist, and the hand image fingertip of the first image data A A non-overlapping area appearing on the side is specified as the fingertip area. The fingertip region can be specified on the first image data A in which the coordinate correspondence relationship with the imaging range (and thus the screen region of the monitor 15) is stored, and the image fingertip point described later, and the corresponding coordinate point on the screen region can be specified. Processing can be performed easily.

  The first and second image data are both binarized, and the hand image non-overlapping area is specified by calculating the image difference between the first image data and the second image data. Thereby, the process of specifying the pixels in the non-overlapping area can be converted into a logical operation between corresponding pixels of the first and second image data (specifically, the exclusive OR between the corresponding pixels can be zero). It can be identified as a pixel in a non-overlapping area). Note that the non-overlapping region of the first and second image data may be narrowly formed on the side surface portion of the finger. However, such a side surface portion can be easily removed by performing a process of inverting those pixel columns by “0” when the number of consecutive “1” pixels in the X direction is less than a predetermined value. it can.

  Each fingertip region extracted in this way is subjected to the contraction process already described, as shown in FIG. Specifically, for all pixels having a value of “1”, pixels that have a certain adjacent relationship with this (specifically, four pixels on the top, bottom, left, and right, and further, four pixels that are diagonally adjacent to each other) If one pixel whose value is inverted with respect to the target pixel is included in the 8 pixels added to the above, a process of inverting the value of the pixel by “0” is performed. The process. It may be repeated several times.

  Then, as shown in E of FIG. 4, the barycentric coordinates of each fingertip region after the contraction process are calculated. Specifically, by separating and specifying individual fingertip areas by, for example, labeling processing of adjacent pixels, calculating the X coordinate sum and Y coordinate sum of all pixels for each fingertip area, and dividing each by the total number of pixels. The X and Y coordinates of the center of gravity of each fingertip region can be specified.

  Note that the touch panel input operation surface 102a actually touches the area near the finger pad slightly lowered in the X direction from the tip of the finger. Therefore, in FIG. 4F, the center of gravity calculated by E is the Y direction. A predetermined distance is offset, and this is determined as the image fingertip point G. However, the barycentric position of E may be used as it is as the image fingertip point G, and in this case, the processing of F is unnecessary.

FIG. 5 shows a state in which the image fingertip points (G _{1 to} G ₅ ) of each finger are coordinate-specified on the input operation surface 102a, and thus on the image frame. Then, an indicator image frame that is unrelated to the original finger image FI is sequentially pasted on each of the image fingertip points G _{1 to} G ₅ on the image frame to create an indicator image frame. This indicator image is created so as to be relatively narrower than the original finger image FI. For example, in the finger width distribution of all Japanese who are 18 years of age or older, the finger width corresponding to the lower limit of the range in which 90% of subjects are included centering on the average value is, for example, 50% to 80%. If the width dimension of the indicator image is set in this way (for example, in the case of the index finger, the actual finger size conversion in the first joint position is 7 mm or more and 14 mm or less), it is possible for almost all operators except children. The indicator image can be made relatively narrower than the original finger image.

  As the indicator image, a simulated finger image simulating the contour shape of a finger can be used. As a simple example, as shown in FIG. 10, a simulated finger image in which an arc shape 201 indicating the contour of the fingertip portion and a rectangular shape 202 indicating the contour of the remaining portion can be exemplified. By making the fingertip portion an arc, there is an advantage that the center point 201g of the arc can be easily used as the fingertip position to be aligned with the image fingertip point. It is also possible to display a simpler figure than the simulated finger image, for example, an arrow-like figure similar to a pointer.

On the other hand, as shown in FIG. 11, even if the finger contour image data SF1 to SF5 in which a finger contour that more closely simulates an actual finger is expressed by a broken line or a curve (for example, a B-spline curve or a Pezier curve) are used. Good. The finger contour image data SF _{1 to} SF ₅ can be configured as vector outline data defined by a series of handling points HP arranged corresponding to the finger contour.

In addition, as the indicator image, an actual finger image of a person captured and acquired in advance in the form of individual fingers (for example, a finger image of the operator himself or a model finger image acquired from a hand specialized part model) Can be used). In this case, finger contour image data SF _{1 to} SF ₅ similar to those in FIG. 11 can be obtained by extracting a contour line from the photographed finger image by a known edge detection process and creating vector outline data that approximates the contour line. it can. Also, as shown in FIG. 12, bitmap graphic data obtained by binarizing a finger photographed image may be used as the indicator image SF (in this case, extraction of a finger outline is not necessary).

As shown on the right of FIG. 6, the indicator fingertip point G ′ is set at a predetermined position at the tip of the indicator image SF. As shown on the right of FIG. The indicator image SF is pasted on the image frame in such a manner that G ′ is matched with each image fingertip point (G _{1 to} G ₅ ). However, in order to perform this pasting, it is necessary to specify the direction of the finger as information in addition to the fingertip position G ′. Therefore, as shown in FIG. 5, a wrist prescribed point W is set on the image frame (display coordinate plane) separately from the image fingertip points (G _{1 to} G ₅ ), and the wrist prescribed point W and the image fingertip point G are set. An operation is performed to determine the connecting straight line as a finger straight line (L _{1 to} L ₅ ). As shown in FIG. 6, each indicator image SF _{1 to} SF ₅ has the fingertip position G ′ coincident with the image fingertip point (G _{1 to} G ₅ ) and the determined finger straight line (L _{1 to} L _5). ) So that the longitudinal reference line of the indicator image SF (predetermined for each of the indicator images SF _{1 to} SF ₅ ) is matched, and an indicator image frame is created.

Since the phalanges of the hand are arranged so as to converge toward the wrist joint, the wrist prescribed point W in FIG. 5 is defined as a wrist prescribed point corresponding to this wrist (hereinafter also referred to as the wrist prescribed point W). ing. As described above, the input operation surface 102a (shooting range) is dimensioned so that only a part of the palm fingertip side can be accommodated, and the operation can be performed by extending the hand from the near side to the shooting area. Therefore, the wrist specified point W is set so as to be off the lower side of the screen area. In Figure 5, the wrist specified point W is set in the Y direction from the lower edge of the screen area at a position spaced just below the predetermined length Y _0. Regardless of the Y coordinate of the image fingertip points (G _{1 to} G ₅ ) on the screen, the Y-direction position of the wrist prescribed point W is fixedly determined based on the lower edge of the screen area. The algorithm can be simplified. In the embodiment of FIG. 5, the X coordinate of the wrist prescribed point W is fixed at the center of the X direction dimension of the imaging range (the input operation surface 102a and the screen of the monitor 15). When the height in the Y direction of the screen area is L, the value of Y ₀ + L / 2 may be adjusted to, for example, 100 mm or more and 200 mm or less.

  The indicator image frame thus created as shown in FIG. 6 is transferred to the graphic controller 110, combined with the separately acquired input screen image frame data, and displayed on the monitor 15. As a method of combining the input screen image frame data and the indicator image frame data, the following method can be adopted depending on the data form of the indicator image SF.

  (1) When the indicator image data is described as bitmap data from the beginning, the indicator image can be displayed in a watermark form on the input screen by alpha blending processing between corresponding pixels.

  (2) When the indicator image data is described by vector outline data, a contour line of the indicator image is generated using the data on the indicator image frame, and the inside thereof is rasterized to generate a bitmap. Thereafter, the same alpha blending process as in (1) is performed.

  (3) A vector outline data constituting the indicator image data is used to draw a contour line on the input screen image frame, and the pixels of the input screen image located inside thereof are extracted, and the set values of the extracted pixels Shift uniformly.

  In any of (1) to (3), for the pixels forming the contour line of the indicator image data, the indicator image with the contour line emphasized is superimposed by increasing the blend ratio on the indicator image data side. Can be displayed. It is also possible to display the indicator image data as image data of only the contour line described by the bitmap data or the vector outline data, and display only the contour line in an overlapping manner.

  The effects that can be realized by the operating device 1 are as follows.

  That is, as shown in FIG. 1, the screen of the monitor 15 is arranged away from the direction of looking directly at the finger on the touch panel 12a when viewed from the operator seated in the driver's seat 2D or the passenger seat 2P. It is impossible to look directly at both the hand performing and the monitor 15 at the same time. Therefore, the operator uses the indicator image on the screen as the only information source for grasping the hand operation position. In the present invention, irrespective of the state of the original finger image FI on the image data, the indicator image SF indicating each finger is reliably displayed on the screen in a form narrower than the original finger image FI. As a result, the photographed original finger image is enlarged and displayed as it is, and the problem that hinders operability can be effectively solved.

  This effect becomes even more prominent when the shooting range (and hence the operation input surface 102b of the touch panel) is reduced, as indicated by the alternate long and short dash line in FIG. Of the index finger, the middle finger, and the ring finger, the entire two can be photographed at least, and the entire four are set to a size that cannot be photographed. That is, as shown in FIG. 14, the index finger, the middle finger, the ring finger, This may be the case where the size of the index finger, middle finger, ring finger, or two of the index finger and middle finger (or middle finger and ring finger) is set as shown in FIG. It corresponds to.

  In particular, in the case of FIG. 15, when the dimensions of the input operation surface 102a (imaging range) are expressed by specific numerical values, the X direction dimension is 60 mm or more and 80 mm or less (for example, 70 mm), and the Y direction dimension is 30 mm or more and 55 mm or less (for example, 43 mm). The number of fingers that fit in the shooting range is two. However, if the captured original finger image FI is binarized and displayed on the screen area of the monitor 15 as it is, as shown by the broken line in FIG. The two original finger images FI are greatly enlarged and displayed as much as the shooting range is reduced. The monitor 15 displays a soft keyboard KB for inputting a Japanese syllabary, but the number of soft buttons SB exceeds 50, and more than 3 soft buttons SB in the width direction of the enlarged original finger image FI. (I.e., when the original finger image FI on the image data is virtually projected onto the display screen of the soft keyboard KB while saving the coordinate dimensions at the corresponding position on the screen area, (The soft button SB image is displayed and formed in a dimension and arrangement interval in which a plurality of soft button SB images (specifically, more than two) are included in the finger width direction with respect to the virtual projection area of the original finger image FI) . In this case, it is very difficult to determine whether or not the finger is correctly directed to the intended soft button, and naturally, it is easy to cause a problem that the next to the aimed button is pushed wrong.

  However, as shown by the solid line in FIG. 13, if this is replaced and displayed by the indicator image SF narrower than the original finger image FI, the number of soft buttons SB overlapping in the width direction of the indicator image SF is 2. The number of buttons is reduced to one or one, the denseness of the buttons near the indicator image SF is reduced, and it is possible to easily grasp which one of the plurality of soft buttons SB arranged on the screen is being operated. As a result, it is clear that problems such as pushing the next to the target button are unlikely to occur, and the operability is remarkably improved.

Hereinafter, various modifications according to the wrist setting point setting method will be described. First, when the input position in the Y direction can change greatly, it is necessary to consider the change in the Y direction of the wrist position associated therewith. In this case, as shown in FIG. 7, when the wrist specified point W is set to have a predetermined positional relationship with the specified image fingertip point G on the display coordinate plane, the indicator image SF is set. The reality of the arrangement direction can be further increased. Specifically, the wrist predetermined point W is located at a position spaced apart from the image fingertip point G specified in the Y direction by a certain length Y ₂ (for example, Y ₂ is 100 mm or more and 200 mm or less). Should be set.

  Next, consider a case where the operation unit 12 having the arrangement form as shown in FIG. 1 is to be operated from the seat 2D or the seat 2P. When moving the hand placed on the operation unit 12 on the left side (in the case of the driver's seat 2D) or right side (in the case of the passenger seat 2P) in the Y direction while sitting, move the shoulder joint and the elbow joint. The main operation is to move the lower collar extending forwards back and forth. As a result, the movement of the palm photographed on the input operation surface 102a is almost parallel to the Y direction, and the direction (angle) of the finger involved in the input does not change so much. However, when the hand is moved in the X direction, the wrist rotational movement is dominant, and the movement of the palm photographed on the input operation surface 102a is close to the form of rotating the hand around the axis near the center of the palm. As a result, the direction (angle) of the finger involved in finger input changes according to the rotation angle of the hand.

Therefore, as shown in FIG. 8, if the method of changing the X coordinate of the wrist prescribed point W that determines the finger direction according to the X coordinate of the image fingertip point G is adopted, the above-described movement of the hand is reflected. Convenient. In Figure 8, the lower screen area (imaging range), defines the reference wrist specified point W ₀ indicating a reference wrist position fixedly, the inclination angle θ becomes larger with respect to the Y direction of the original finger image FI, reference wrist provisions can displacement amount in the X direction from the point W ₀ determines the X coordinate value of the wrist prescribed point W to be larger.

8, defines an X-coordinate of the reference wrist prescribed point W ₀ in the center of the X-direction dimension of the shooting range (screen input operation surface 102a and the monitor 15), also, among the plurality of image fingertip points that have already been identified The reference wrist prescribed point W ₀ is defined as a position where the fingertip position is located at the uppermost position (in FIG. 8, the image fingertip point G ₃ corresponding to the middle finger) is moved downward by a predetermined value Y ₂ from the reference. Coordinates are defined.

As described above, assuming that the center of rotation exists inside the palm, as the fingertip rotates in the X direction, the wrist position rotates in the opposite direction. Therefore, for the original finger image FI that is inclined to the right with respect to the direction, the X coordinate value of the wrist prescribed point W is determined so as to be displaced to the left with respect to the reference wrist prescribed point W0 in the X direction. With respect to the original finger image FI that is inclined leftward with respect to the direction, the X coordinate value of the wrist prescribed point W is determined so as to be displaced to the right with respect to the reference wrist prescribed point W0 in the X direction. Specifically, using the original finger image FI of fingertip position is located uppermost (corresponding to image fingertip point G ₃₎ as the representative original finger image, the inclination angle (clockwise with respect to the Y direction of the raw finger image FI positive Determine θ) (defined as direction). The inclination angle θ can be calculated from, for example, the gradient of a straight line obtained by applying the least square method to the pixel points constituting the original finger image FI. The X-direction displacement (or X coordinate value itself) of the wrist specified point W to be obtained from the reference wrist specified point W ₀ is determined in advance for each value of θ and stored in the ROM 103 of FIG. Then, the displacement in the X direction corresponding to the calculated θ can be easily determined by reading from the table. In Figure 8, the Y coordinate of the wrist prescribed point W, are set to always equal to the Y coordinate of the reference wrist prescribed point W _0. That is, the wrist specified point W is set so as to move on a straight line parallel to the X axis through the reference wrist specified point W ₀ according to the angle θ. However, the wrist specified point W may be determined along an arcuate path.

As the representative original finger image, an image having the X-coordinate (or Y-coordinate) closest to the center in the X direction (or Y direction) of the shooting range as the image fingertip point may be adopted. Further, the wrist specified point W may be determined in such a manner that a point obtained by averaging the X and Y coordinates of a plurality of image fingertip points (G _{1 to} G ₅ ) is used as the representative image fingertip point. Further, when the number of fingertip positions is odd, the image fingertip point of the original finger image at the center is adopted, and when the number of fingertip positions is even, two original finger images close to the center are adopted, and the X coordinate and Y of those image fingertip points are adopted. A method of determining points obtained by averaging coordinates as representative image fingertip points is also possible.

(Reference Example) Next, in an actual human hand, each phalange naturally has an individual width even at the wrist position, and is connected to different positions in the X direction with respect to the wrist joint. Therefore, as in the reference example shown in FIG. 9, independent wrist prescription points W _{1 to} W ₅ are determined for the plurality of image fingertip points G _{1 to} G ₅ , and the arrangement direction of the indicator image is set to each image fingertip point G. it can be determined by using a wrist specified point _W 1 to _W-5 corresponding to the ₁ ~G _5. Specifically, the image fingertip point G ₁ than the reference wrist prescribed point W ₀ is positioned in the X direction _right, for G ₂ is, corresponding so as to be positioned on the right side of the X-coordinate reference wrist specified point W ₀ Wrist specified points W ₁ and W ₂ to be set are set, and for image fingertip points G ₃ , G ₄ and G ₅ which are located on the left side in the X direction from the reference wrist specified point W ₀ , the X coordinate is the reference wrist specified point. Corresponding wrist prescribed points W ₃ , W ₄ , and W ₅ are set so as to be located on the left side of W ₀ .

Incidentally, with respect to the reference wrist specified point W ₀ larger image fingertip points in the X direction displacement (h1 to h5), X-coordinate of the corresponding wrist defining points such that displacement in the X direction from the reference wrist specified point W ₀ is increased It is stipulated in. Specifically, a value obtained by multiplying the X-direction displacement of the image fingertip point with respect to the reference wrist specified point W _{0 by} a constant magnification factor k (for example, 0.1 ≦ k ≦ 0.3) is set to X of the wrist specified point W. It is calculated as a displacement in the X direction from the reference wrist prescribed point W ₀ concerning the coordinates.

  The example in which the image display device of the present invention is applied to an operation device for an in-vehicle electronic device has been described above. However, the present invention is not limited to this, and is also applied to, for example, a GUI input device for a personal computer. Is possible.

Claims (5)

Image data acquisition means for acquiring image data including an image of a human finger as an original finger image;
Image fingertip point specifying means for specifying the position of the fingertip portion of the original finger image as an image fingertip point on the image data;
A display device including a screen area having a unique coordinate correspondence with the image data;
A pointer image display means for displaying a pointer image prepared separately from the image data at a position corresponding to the image fingertip point in the screen area;
With
The indicator image is a simulated finger image imitating the contour shape of a finger or a real finger image of a person captured and acquired by separating individual fingers,
The screen area displays only a part of the indicator image in the finger longitudinal direction including the finger tip,
The indicator image display means includes a wrist prescribed point setting means for setting a wrist prescribed point on a display coordinate plane that defines a display position of the indicator image on the screen area, the wrist prescribed point, and the image fingertip point. And an indicator arrangement direction determining means for determining the arrangement direction of the indicator image based on
When a plurality of the original finger images are included in the image data,
The image fingertip point specifying means is for specifying the image fingertip point for each of the plurality of original finger images,
The wrist prescribed point setting means sets the common wrist prescribed point for the plurality of image fingertip points specified by the image fingertip point specifying means,
Further, the wrist prescribed point setting means includes:
On the display coordinate plane, the vertical direction of the screen area is defined as the Y direction, the direction orthogonal thereto is defined as the X direction, and a reference wrist prescribed point is fixedly defined outside the screen area in the Y direction,
When a plurality of the image fingertip points are specified on the screen area, the original finger image in which the image fingertip point is located on the uppermost side in the Y direction on the screen area is used as a representative original finger image. As the inclination angle of the representative original finger image with respect to the Y direction increases, the amount of displacement of the wrist reference point in the X direction from the reference wrist prescribed point increases.
For the representative original finger image that is inclined to the right upward with respect to the Y direction, the X coordinate value of the wrist prescribed point is determined so as to be displaced to the left with respect to the reference wrist prescribed point,
For the original finger image that is tilted leftward with respect to the Y direction, the X coordinate value of the wrist prescribed point is determined so as to be displaced to the right with respect to the reference wrist prescribed point.
The indicator arrangement direction determining means determines the arrangement direction based on the wrist prescribed point for each of the plurality of image fingertip points, and the longitudinal direction of each indicator image is a plurality of the indicator images. An image display device that displays on the screen area along the arrangement direction.   2. The image display device according to claim 1, wherein the wrist specified point setting unit sets the wrist specified point at a position spaced apart from the lower edge of the screen area by a predetermined length in the Y direction. .   The finger direction specified point setting means sets the wrist reference point so as to have a predetermined positional relationship with the image fingertip point of the representative original finger image in the Y direction. The image display device according to claim 1. A transparent input support plate for receiving a touch operation by the person's finger while supporting it on one plate surface side;
A position input device that detects the touch operation and outputs the coordinate information;
4. The image display device according to claim 1, wherein the image data acquisition unit is a camera disposed from the other plate surface side of the transparent input support plate. 5. Obtain image data that contains the image of a human finger as the original finger image,
Specifying the position of the fingertip portion of the original finger image as the image fingertip point on the image data;
A pointer image prepared separately from the image data at a position corresponding to the image fingertip point in the screen area of a display device having a screen area having a unique coordinate correspondence with the image data A display method of the indicator image for displaying
The indicator image is a simulated finger image imitating the contour shape of a finger or a real finger image of a person captured and acquired by separating individual fingers,
The screen area displays only a part of the indicator image in the finger longitudinal direction including the finger tip,
When displaying the indicator image in the screen area,
Setting a wrist prescribed point on a display coordinate plane defining the display position of the indicator image on the screen area;
In the method for determining the arrangement direction of the indicator image based on the wrist prescribed point and the image fingertip point,
When a plurality of the original finger images are included in the image data,
Specifying the image fingertip point for each of the plurality of original finger images;
For the plurality of image fingertip points specified by the image fingertip point specifying means, the common wrist prescribed point is set,
On the display coordinate plane, the vertical direction of the screen area is defined as the Y direction, the direction orthogonal thereto is defined as the X direction, and a reference wrist prescribed point is fixedly defined outside the screen area in the Y direction,
When a plurality of the image fingertip points are specified on the screen area, the original finger image in which the image fingertip point is located on the uppermost side in the Y direction on the screen area is used as a representative original finger image. As the inclination angle of the representative original finger image with respect to the Y direction increases, the amount of displacement of the wrist reference point in the X direction from the reference wrist prescribed point increases.
For the representative original finger image that is inclined to the right upward with respect to the Y direction, the X coordinate value of the wrist prescribed point is determined so as to be displaced to the left with respect to the reference wrist prescribed point,
For the original finger image that is tilted to the left in the Y direction, the X coordinate value of the wrist prescribed point is determined so as to be displaced to the right with respect to the reference wrist prescribed point,
For each of the plurality of image fingertip points, determine the arrangement direction based on the wrist prescribed point,
A display method of a pointer image, wherein a plurality of the pointer images are displayed on the screen area so that a longitudinal direction of each pointer image is along the arrangement direction.

Images