JP2016055954A - Portable information terminal, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a portable information terminal which can adequately indicate a position of an article which is an acquisition object for a user.SOLUTION: A portable information terminal is equipped with: a reception part 101 which receives one or more article identifiers as an identifier for an article of an acquisition object; a photographing part 102 for acquiring a photographed image in which an area including one or more codes which are located at a position corresponding to the article and are linked with article identifiers of the article; a determination part 104 which acquires an article identifier of the article linked with one or more codes in the photographed image, and determines whether or not the acquired article identifier is coincident with any one of the article identifiers received by the reception part 101; and a display part 105 which, when the determination part 104 determines that the acquired article identifier is coincident with any one of the article identifiers received by the reception part 101, displays acquired notice information, which indicates the article corresponding to a code is an article of an acquisition object, at a position corresponding to the code which corresponds to the coincident article identifier.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a portable information terminal or the like that supports work on an article.

  As a conventional technique, an imaging device capable of capturing an image using an article processing unit as an imaging target region is provided, and a control unit is processed by the article processing unit based on an image of the imaging target region captured by the imaging device. The type and quantity of the selected article are calculated, and based on the calculated article type and quantity, it is determined whether or not the article processing operation has been performed on the article of the type and quantity specified in the article processing information. (See, for example, Patent Document 1).

JP 2014-91608 A (first page, FIG. 1 etc.)

  However, in the related art, there is a problem in that it is not possible to appropriately indicate to the user a place where an article designated as an acquisition target such as picking work is arranged.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide a portable information terminal that can appropriately indicate the position of an article to be acquired to a user.

  The portable information terminal of the present invention includes a receiving unit that receives one or more article identifiers that are identifiers of articles to be acquired, and a code that is arranged at a position corresponding to the article, and is associated with the article identifier of the article. A photographing unit that captures a photographed image of an area including one or more codes, and an article identifier of an article associated with one or more codes in the photographed image, and the accepting part accepts the obtained article identifier. A determination unit that determines whether or not it matches any one of the received product identifiers, and when the determination unit determines that the received product identifier matches the received product identifier, the code corresponding to the matching product identifier It is a portable information terminal provided with a display unit that displays acquisition notification information, which is information indicating that an article corresponding to a code is an acquisition target article.

  With this configuration, the position of the article can be appropriately shown to the user.

  In the portable information terminal of the present invention, in the portable information terminal, the accepting unit accepts two or more article identifiers, the photographing unit acquires a photographed image obtained by photographing an area including two or more codes, and the judging unit Obtains an article identifier of an article associated with each of the two or more codes in the captured image, and determines whether each of the obtained article identifiers matches any one of the article identifiers received by the reception unit. When the determination unit determines that each of the article identifiers of the articles associated with two or more codes in the photographed image matches any one of the article identifiers received by the reception unit. The portable information terminal displays acquisition notification information at positions corresponding to codes corresponding to two or more matching article identifiers.

  With this configuration, even when a plurality of articles to be acquired are included in the same captured image, the positions of the plurality of articles can be shown to the user, and the positions of the plurality of articles to be acquired can be appropriately displayed by the user. Can be shown.

  In the portable information terminal according to the present invention, in the portable information terminal, the reception unit further receives a quantity associated with the article identifier, and the display unit matches the determination unit among the quantities received by the reception unit. Then, the mobile information terminal displays the quantity associated with the determined article identifier as acquisition notification information.

  With this configuration, it is possible to visually indicate the quantity to be acquired of the articles to be acquired to the user in an easily understandable manner.

  In the portable information terminal of the present invention, in the portable information terminal, the display unit obtains the quantity associated with the article identifier determined to be the same by the determination unit among the quantities received by the reception unit. As a portable information terminal that displays a position on a code corresponding to an article identifier in a region where a photographed image is photographed.

  With such a configuration, it is possible to visually indicate the quantity to be acquired of the articles to be acquired to the user in a more easily understandable manner.

  In the portable information terminal of the present invention, in the portable information terminal, whether or not an article indicated by a code associated with the article identifier determined by the determination unit to be matched with the article identifier received by the reception unit is taken out by the user. This is a portable information terminal including an acquisition determination unit that determines a captured image acquired by the imaging unit, and an output unit that performs output according to the determination result of the acquisition determination unit.

  With such a configuration, it is possible to perform output according to the acquisition status of the article, and to confirm the acquisition operation of the article.

  In the portable information terminal of the present invention, in the portable information terminal, the article indicated by the code associated with the article identifier determined by the determination unit to match the article identifier received by the reception unit is an article received by the reception unit. An acquisition determining unit that determines whether or not only the quantity associated with the identifier has been acquired by the user using the captured image acquired by the imaging unit; and an output unit that performs output according to the determination result of the acquisition determining unit; It is a portable information terminal provided with.

  With this configuration, it can be confirmed whether or not the correct quantity of articles has been acquired.

  In the portable information terminal according to the present invention, in the portable information terminal, the acquisition determination unit detects a hand in a captured image captured by the imaging unit, and whether or not the article is acquired by the user using the detected hand. It is a portable information terminal that judges

  With this configuration, the movement of the article can be detected without depending on the shape or color of the article, and the acquisition of the article can be appropriately determined.

  In the portable information terminal of the present invention, the portable information terminal further includes a head mounted display, the display unit displays the acquisition notification information on the head mounted display, and the photographing unit is in the forward direction of the head mounted display. Is a portable information terminal arranged so as to be an imaging region.

  With this configuration, it is not necessary to hold the display by hand, and workability can be improved. Further, the direction of the captured image can be directed to the direction of the user's visual field without being conscious of the user, and the operability can be improved.

  According to the portable information terminal and the like according to the present invention, it is possible to appropriately indicate the position of the article to be acquired to the user.

Block diagram of portable information terminal in an embodiment of the present invention The figure which shows an example of the head mounted display of the portable information terminal Flow chart for explaining the operation of the portable information terminal The figure which shows an example of the shelf by which the code | cord is arrange | positioned for describing operation | movement of the portable information terminal (FIG. 4 (a)), and the figure which shows the code | cord | chord arrange | positioned on a shelf (FIG.4 (b)) The figure which shows the identifier management table of the portable information terminal The figure which shows the reception management table of the portable information terminal The figure which shows an example of the picked-up image of the portable information terminal The figure which shows the example of a display of the portable information terminal The figure which shows an example of the picked-up image of the portable information terminal The figure which shows an example of the external appearance of the computer system in embodiment of this invention The figure which shows an example of a structure of the computer system

  Hereinafter, embodiments of a portable information terminal and the like will be described with reference to the drawings. In addition, since the component which attached | subjected the same code | symbol in embodiment performs the same operation | movement, description may be abbreviate | omitted again.

(Embodiment)
FIG. 1 is a block diagram of portable information terminal 1 in the present embodiment.

  The portable information terminal 1 includes a reception unit 101, an imaging unit 102, an identifier storage unit 103, a determination unit 104, a display unit 105, an acquisition determination unit 106, an output unit 107, and a head mounted display 108.

  The receiving unit 101 receives one or more article identifiers. The article identifier received by the receiving unit 101 is an identifier of an article to be acquired. The receiving unit 101 may receive two or more article identifiers. The article is, for example, a product. Further, the article may be a loaned item, a free sample, or the like like a library book. Acquisition here is, for example, taking out or collecting articles from a position (for example, a shelf or a conveyor) where the articles are arranged. The acquisition may be, for example, taking out or collecting articles that are to be sold, shipped, or delivered. The acquisition here is a concept including, for example, so-called picking, taking out, selecting, and the like. The item identifier is, for example, an identifier of an item to be picked up or taken out for sales or delivery. For example, the article identifier received by the receiving unit 101 is an article identifier of an article managed in a warehouse, a distribution center, or the like. The article identifier is, for example, an article name, a character string or a numeric string assigned to the article, a manufacturing number, or the like.

  The accepting unit 101 may further accept the quantity associated with the article identifier. For example, the reception unit 101 may receive one or more pieces of information having an article identifier and a quantity associated with each other. Hereinafter, the combination of the article identifier and the quantity may be referred to as article quantity information. The receiving unit 101 further includes an identifier assigned to a group of articles to be delivered in a batch such as an order identifier such as an order number associated with the article identifier, a delivery number associated with the article identifier, Date information or the like may be received.

  The receiving unit 101 normally receives an article identifier transmitted from a server apparatus (not shown), an apparatus (not shown) for instructing picking (taking out) an article to be delivered, or the like. A server device (not shown) is, for example, a server device or the like that receives a delivery or shipping instruction and transmits an identifier of a delivery target or an article to be shipped to the portable information terminal 1 or the like. However, the receiving unit 101 may receive the article identifier by reading a code such as a barcode associated with the article identifier of the article to be picked (removed) printed on the paper or the like with a camera or the like. . In addition, an article identifier manually input by the user via an input device (not shown) or the like may be received.

  Hereinafter, the article identifier received by the receiving unit 101 may be referred to as a received article identifier for convenience of explanation. The received article identifier may be considered as an article identifier of an article to be acquired.

  The reception here is usually reception of information transmitted via a wired or wireless communication line, but reception of information input from an input device such as a keyboard, a mouse, a touch panel, an optical disk or a magnetic disk The concept includes reception of information read from a recording medium such as a semiconductor memory. The receiving unit 101 can be realized by a device driver for input means such as a numeric keypad or a keyboard, control software for a menu screen, and the like.

  The imaging unit 102 acquires a captured image obtained by capturing an area including one or more codes. The one or more codes are codes arranged at positions corresponding to the articles, and are codes associated with the article identifiers. For example, the photographing unit 102 may acquire a photographed image obtained by photographing a region including two or more codes.

  The photographing unit 102 is a camera including an image sensor such as a CCD or a CMOS, for example. The photographing unit 102 may be a video camera or a digital still camera. The captured image is usually a moving image, but may be a plurality of continuous still images captured at predetermined time intervals. The captured image may be considered as one or two or more frame images constituting a moving image. Two or more frame images may or may not be continuous. For example, the two or more frame images may be two or more frame images obtained by separating a predetermined number of frame images. Further, the number of pixels of the captured image captured by the capturing unit 102, the file format of the captured image transmitted by the capturing unit 102, the compression method, and the like are not limited. Moreover, the timing at which the imaging unit 102 captures a captured image is not limited. Note that the photographing unit 102 may be considered as a driver for controlling an external camera or the like.

  The code may be anything that can be identified, such as a barcode, a two-dimensional barcode, a QR code (registered trademark), a chameleon code, a character string, and a numeric string. The code is preferably a two-dimensional code having a color arrangement as identification information, such as a chameleon code. That the code is associated with the article identifier may be, for example, that the information read from the code is associated with the article identifier. For example, information stored in association with information read from a code and an article identifier is stored in advance in a storage unit (not shown), so that the code and the article identifier can be managed in association with each other. The information read from the code may be an article identifier. For example, the code identifier read from the code may be an article identifier.

  Hereinafter, in the present embodiment, the information read from each code is a code identifier that is an identifier of the code, and the code corresponds to the article identifier by managing each code identifier in association with the article identifier. The case where it is attached will be described as an example. The code identifier is, for example, a character string or a numeric string assigned to the code.

  The code arranged at a position corresponding to the article is, for example, a code arranged (for example, attached) to the article. Moreover, the code arrange | positioned in the position corresponding to an article | item may be the code arrange | positioned in the position which can identify an article | item, for example. The position here is a concept including a region. The position where the article can be specified may be, for example, a position (for example, an area) where the article is arranged, or a position (for example, an area) around the position where the article is arranged. For example, the cords arranged around the position where the article is arranged are arranged on a shelf or a pallet on which the article is arranged (for example, attached), on a partition plate which partitions the article ( For example, it may be an attached code.

  The identifier storage unit 103 stores one or more identifier management information. The identifier management information is information having a code identifier and an article identifier. Usually, different identifier management information has different code identifiers. The article identifier stored in the identifier storage unit 103 is an article identifier associated with the code identifier. Hereinafter, an article identifier associated with a code identifier may be referred to as a corresponding article identifier for convenience of explanation. The corresponding article identifier may be considered as an article identifier stored in the identifier storage unit 103.

  The identifier storage unit 103 is preferably a nonvolatile recording medium, but can also be realized by a volatile recording medium. The same applies to other storage units.

  The determination unit 104 uses the captured image acquired by the imaging unit 102 to acquire an article identifier of one or more articles associated with one or more codes in the captured image. Then, the determination unit 104 determines whether or not the acquired one or more article identifiers match any one of the article identifiers received by the reception unit 101.

  For example, the determination unit 104 obtains an article identifier of an article associated with each of two or more codes in the captured image. Then, for example, the determination unit 104 determines whether each of the acquired two or more item identifiers matches any one of the item identifiers received by the reception unit 101.

  For example, the determination unit 104 detects one or more codes in the captured image acquired by the imaging unit 102. The code detection here may be considered to detect a code image or a code area. For example, when the captured image is a moving image, the determination unit 104 binarizes the frame image acquired by the imaging unit 102, and forms a shape such as an outer frame of the code by processing such as pattern matching or feature amount detection. Recognizing and detecting a region where a code in the photographed image is arranged. Then, an image of the detected code is acquired. The same applies when the captured image is a still image. Note that the determination unit 104 may acquire images of a plurality of codes from one captured image. The determination unit 104 may appropriately correct the shape of the acquired code image. In addition, since the process etc. which detect a code | cord | chord and acquire a code identifier from these images are well-known techniques, detailed description is abbreviate | omitted here.

  Next, the determination unit 104 acquires a code identifier from one or more detected codes. Then, an article identifier (that is, a corresponding article identifier) corresponding to each acquired code identifier is acquired using the identifier management information stored in the identifier storage unit 103. If the information read from the code is an article identifier, the information read from the code is acquired as the article identifier.

  Next, the determination unit 104 determines whether or not the acquired one or more article identifiers match any one of the article identifiers received by the receiving unit 101 (that is, the received article identifiers). For example, the determination unit 104 determines whether one of the one or more corresponding article identifiers acquired above matches one of the one or more reception article identifiers received by the reception unit 101. To do. When the corresponding article identifier matches any one of the received article identifiers, for example, a determination result indicating that the corresponding article identifier matches is acquired. The determination result is accumulated in a storage unit (not shown) in association with the corresponding article identifier, for example. Note that acquiring and storing the determination result for the corresponding article identifier may be considered to be substantially the same as acquiring and storing the determination result for the received article identifier whose value matches the corresponding article identifier. . The determination unit 104 performs the same process for the other corresponding article identifiers acquired above.

  For example, in the case of a two-dimensional code having a color array as identification information, such as a chameleon code, even if there are a plurality of codes in one photographed image, a code identifier or the like It is known that there is something that can acquire information corresponding to a code. For this reason, by using such a code, each of two or more corresponding article identifiers acquired for a plurality of codes in one photographed image matches one of the accepted article identifiers received by the receiving unit 101. It is possible to determine whether or not.

  Of the received article identifiers received by the receiving unit 101, a flag indicating that a received article identifier that has been determined to coincide with one corresponding article identifier acquired for the code of the photographed image has already been detected. By giving such information, it may not be used in the above determination processing in the subsequent processing.

  When the determination unit 104 determines that one or more corresponding article identifiers acquired for one or more codes in the captured image match the received article identifier received by the receiving unit 101, the display unit 105 Information indicating that the article corresponding to the code is the article to be acquired is displayed at a position corresponding to the code corresponding to the identifier. The article corresponding to the code is, for example, an article indicated by a corresponding article identifier corresponding to the code. The acquisition target article is, for example, an article indicated by any one of the received article identifiers received by the receiving unit 101. Information indicating that the article corresponding to the code displayed at the position corresponding to the code is the acquisition target article is referred to as acquisition notification information here. The acquisition notification information may be considered as information indicating that the code corresponds to any one of the received article identifiers received by the receiving unit 101.

  The display unit 105 determines that each of the article identifiers of the corresponding articles associated with the two or more codes in the photographed image matches any one of the received article identifiers received by the receiving unit 101. In this case, the acquisition notification information may be displayed at a position corresponding to a code corresponding to each of two or more matching article identifiers.

  The position corresponding to the code here may be an area on the code or an area including at least a part of the area on the code. It may be a position in a positional relationship. Further, it may be a position separated by a predetermined distance in a predetermined direction with respect to the code. Moreover, the position where the distance from a code | cord is in the range designated beforehand may be sufficient. Further, the position here may be a region, or may be coordinates of a position where one point of the acquisition notification information is arranged.

  The acquisition notification information displayed at a position corresponding to one code may be, for example, an image designated in advance, specifically, an image having a shape designated in advance. This image is, for example, a symbol image such as a rectangle of a color designated in advance or an arrow indicating the position of the article corresponding to the code. Further, the acquisition notification information displayed at the position corresponding to one code may be, for example, the quantity associated with the accepted article identifier that matches the corresponding article identifier corresponding to this code. For example, the quantity included in the article quantity information having an article identifier that matches the corresponding article identifier corresponding to this code may be used. Further, the acquisition notification information displayed at a position corresponding to one code may be, for example, a corresponding article identifier corresponding to this code, and a delivery associated with a received article identifier that matches the corresponding article identifier. Information such as an identifier may be used.

  In this way, by displaying the acquisition notification information, for example, a code corresponding to one article is arranged on the one article, or arranged at a position where the article is arranged. Thus, it is possible to indicate to the user that the article at the position where the acquisition notification information is displayed is the article to be acquired.

  For example, the display unit 105 displays, as the acquisition notification information, the quantity associated with the received article identifier determined by the determination unit 104 among the quantities received by the reception unit 101. In addition, the display unit 105 uses the quantity associated with the received article identifier determined by the determination unit 104 out of the quantities received by the reception unit 101 as the acquisition notification information in the area where the captured image is captured. May be displayed at a position on the code corresponding to the article identifier. In this way, by displaying the acquisition notification information, which is the quantity associated with the article identifier corresponding to the code, at the position corresponding to the code, the user can appropriately grasp the number of articles to be acquired. In particular, even if the acquisition notification information is displayed at the same time at a position corresponding to each of a plurality of codes, the quantity corresponding to the article identifier corresponding to each corresponding code is displayed as each acquisition notification information. By doing so, it is possible to immediately and appropriately grasp the quantity to acquire each article, and to reduce errors such as quantity when taking out the article.

  The display unit 105 displays the acquisition notification information, for example, superimposed on the captured image captured by the imaging unit 102. For example, the display unit 105 displays an image obtained by combining the acquisition notification information and the image captured by the imaging unit 102 on a display device (not shown) such as a liquid crystal monitor. There is no limitation on the composition mode for superimposing, the transmittance of the superimposed image, and the like. In this case, for example, the position corresponding to the code in the area where the captured image is captured is the position corresponding to the position where the code is displayed in the captured image.

  The display unit 105 may project and display the acquisition notification information on a transmissive mirror or lens without displaying the captured image captured by the capturing unit 102. In such a case, the position corresponding to the code is, for example, a position corresponding to a position where the code can be seen in the user's visual field. For example, by adjusting the correspondence between the field of view of the photographing unit 102 and the field of view of the display to be projected and displayed (for example, the correspondence of coordinates), the code on the display can be seen at a position where the code can be seen. Acquisition notification information can be displayed.

  The display unit 105 may display the acquisition notification information on the head mounted display 108, for example. The head mounted display 108 is a display device worn on the user's head. The head mounted display 108 may be, for example, a non-transmissive head mounted display provided with a display panel such as a liquid crystal panel or an organic EL panel at a position covering the user's visual field. A transmissive display type head mounted display that projects onto a half mirror or the like may be used. Since the head mounted display 108 is a known technique, a detailed description thereof will be omitted. When the display unit 105 displays acquisition notification information or the like on the head mounted display 108, the photographing unit 102 (for example, a camera) is arranged so that the front direction of the head mounted display 108 is the photographing region. Is preferred. The forward direction of the head mounted display 108 is, for example, the forward direction of the user wearing the head mounted display 108.

  The display unit 105 may or may not include a display device. For example, the display unit 105 may display acquisition notification information or the like on a display device (not shown) provided outside the portable information terminal 1, and a display device (not shown) provided inside. Display may be performed. The display unit 105 may be realized by display device driver software, or display device driver software and a display device.

  In the following, a case where the display unit 105 displays the acquisition notification information on the non-transmissive head mounted display 108 will be described as an example. However, the head mounted display 108 may be a transmissive head mounted display.

  FIG. 2 is a diagram showing a non-transmissive head mounted display 108 that is used by the display unit 105 to display acquisition notification information. Here, as an example, the imaging unit 102 (for example, a camera) is attached to the head mounted display 108 such that the front direction of the head mounted display 108 is the imaging area. For example, the photographing unit 102 is installed so that the field of view of the photographing unit 102 substantially matches the field of view of the user wearing the head mounted display 108. Here, as an example, it is assumed that a case 108 a that accommodates a portion of the portable information terminal 1 excluding the head mounted display 108 and the photographing unit 102 is attached to the head mounted display 108. However, the imaging unit 102 and the part other than the head mounted display 108 and the imaging unit 102 of the portable information terminal 1 may not be attached to the head mounted display 108. For example, a portion other than the head-mounted display 108 of the portable information terminal 1 can perform wired communication and wireless communication with the head-mounted display 108, a mobile phone (not shown), a multi-function mobile phone such as a so-called smartphone, Further, it may be configured by a tablet terminal, a portable computer, or the like. Further, instead of the photographing unit 102 attached to the head mounted display 108, a camera (not shown) included in the mobile phone, the multifunctional mobile phone, the tablet terminal, the portable computer, or the like may be used as the photographing unit 102. Good.

  The head mounted display 108 is a display attached to the user's head so as to cover the user's eyes. The head mounted display 108 normally covers both eyes, but may cover only one eye.

  Since the non-transmissive head-mounted display 108 has a display such as a liquid crystal panel disposed at a position that is a visual field of the user, the user cannot directly see the outside in a mounted state. For this reason, the user can behave safely by displaying a captured image showing a state outside the image captured by the imaging unit 102 on the display inside the head mounted display 108. For this reason, when the non-transmissive head mounted display 108 is used as a display device, the display unit 105 preferably displays the acquisition notification information superimposed on the captured image.

  By using the head mounted display 108 as a display device, there is an advantage that it is not necessary to hold the display device or the like by hand, the hand can be used freely, and workability can be improved.

  The acquisition determining unit 106 determines whether or not the user has acquired the article indicated by the code associated with the corresponding article identifier determined by the determining unit 104 to match the received article identifier received by the receiving unit 101. Judgment is made using the acquired photographed image. This may be considered as determining whether or not an article associated with the accepted article identifier received by the receiving unit 101 has been acquired.

  For example, the acquisition determining unit 106 determines that the received article identifier is determined to match from the change in the image at the position corresponding to the code in the captured image that the determining unit 104 determines to match the received article identifier received by the receiving unit 101. It is determined whether or not an article corresponding to is acquired. The position corresponding to the code here is a concept including an area corresponding to the code. As the position corresponding to the code, for example, a position similar to the position where the acquisition notification information is displayed can be used. The position corresponding to the code is, for example, an area on the code.

  For example, the acquisition determining unit 106 determines that one article indicated by the code has been acquired (for example, picked) when an object that has moved from the position corresponding to the code determined to match to another position is detected. May be. A technique for detecting an object from a captured image by detecting a feature amount, detecting an object moving from a difference between frame images, and detecting a moving direction of the detected object are known techniques. Therefore, detailed description is omitted here.

  In addition, the acquisition determination unit 106 may detect a hand in the captured image captured by the imaging unit 102 and determine whether or not the article has been acquired by the user using the detected hand. For example, the acquisition determining unit 106 detects a hand in the captured image, and the detected hand position is associated with the corresponding article identifier that the determining unit 104 determines to match the received article identifier received by the receiving unit 101. When the position corresponds to the code in the captured image (for example, the position in the area corresponding to the code), it may be determined that the article has been acquired by the user. The position corresponding to the code is, for example, a position on the code, an area including the position on the code, or an area located above the code in the photographed image. Further, it may be determined that the article is acquired by the user when the continuous time detected at the position corresponding to the code in the photographed image of the user's hand is equal to or greater than a predetermined threshold value. For example, when the time when the detected hand is continuously located in the area corresponding to the code is equal to or greater than a predetermined threshold value, it is determined that the article has been acquired. It may be determined that the article has not been acquired. In addition, since the process etc. which detect a hand using the feature amount (for example, color feature amount) etc. of a hand from a picked-up image are well-known techniques, detailed description is abbreviate | omitted here.

  Further, the acquisition determination unit 106 acquires a movement pattern of the hand from the hand detected in the photographed image, and when the movement pattern matches a predesignated pattern for determining acquisition of the article, the acquisition is performed. You may judge having acquired. The movement pattern is information such as a moving direction of the hand, a route indicating the movement of the hand, a vector, and the like. For example, when the hand performs a reciprocating motion with a region designated in advance for one cord as a turning point, it may be determined that an article corresponding to the cord has been acquired.

  In addition, the acquisition determination unit 106 is configured such that the article indicated by the code associated with the corresponding article identifier determined by the determination unit 104 that matches the received article identifier received by the reception unit 101 is associated with the received article identifier. Therefore, whether or not the image has been acquired by the user may be determined using the captured image acquired by the imaging unit 102. The quantity associated with the received article identifier here is specifically the quantity received by the receiving unit 101 in association with the received article identifier.

  For example, as described above, the acquisition determination unit 106 detects an object that has moved from the position corresponding to the code associated with the corresponding article identifier determined to match the received article identifier received by the reception unit 101 to another position. In addition, the number of moved objects is counted, and when the counted number matches the quantity associated with the accepted article identifier accepted by the accepting unit 101, the number is associated with the corresponding article identifier corresponding to the code. It may be determined that the product has been acquired by the user by the quantity associated with the received product identifier corresponding to this code.

  In addition, the acquisition determination unit 106 detects a hand in the photographed image captured by the photographing unit 102, and an article associated with the corresponding article identifier corresponding to the code using the detected hand corresponds to the corresponding article identifier. It may be determined whether or not the user has acquired only the quantity associated with the received received article identifier.

  For example, as described above, the acquisition determining unit 106 detects a hand in the photographed image, and the position of the detected hand is one of the photographed images associated with the corresponding article identifier that matches the one accepted article identifier. When the number of times of overlapping with the position corresponding to the code coincides with the quantity associated with this one accepted article identifier, the article associated with the accepted article identifier corresponding to this one code is It may be determined that only the quantity associated with the identifier has been acquired by the user. That is, whether or not the number of times the hand detected in the photographed image has entered the area corresponding to the code and the number indicated by the item identifier corresponding to the code is acquired by the user. May be determined. In addition, it is preferable to count the number of times of overlapping as one time while the position of the hand continues to overlap the position corresponding to the code. Further, it is preferable to count the number of times that the detected hand has entered the area corresponding to the code and the number of times it has entered the area corresponding to the code as one. Also in such a case, as described above, when the user's hand has a continuous time detected at a position corresponding to the code in the captured image is equal to or greater than a predetermined threshold value, The value may be increased by one.

  In addition, for example, as described above, the acquisition determination unit 106 acquires one of the captured images associated with the corresponding article identifier that matches the one received article identifier acquired from the hand movement pattern detected in the captured image. When the number of times of acquisition of the article that acquired the article at the position corresponding to the code matches the quantity associated with this one accepted article identifier, it is associated with the accepted article identifier corresponding to this one code. It may be determined that the product has been acquired by the user by the quantity associated with the received product identifier.

  The acquisition determining unit 106 determines that one or more articles indicated by the code associated with the corresponding article identifier determined by the determining unit 104 to match the received article identifier received by the receiving unit 101 have been acquired. If the user leaves the vicinity of the position where the code is placed before it is determined that the article indicated by the code is acquired by the user by the quantity associated with the received article identifier, It may be determined that the article corresponding to this code has not been properly acquired. For example, after determining that one or more articles indicated by the above code have been acquired, it is determined that the article indicated by this code has been acquired by the user by the amount associated with the received article identifier. Before, when this code cannot be detected in the photographed image, it may be determined that an article corresponding to this code has not been properly acquired. In this case, in particular, a determination result indicating that the acquired quantity is insufficient may be acquired.

  In addition, the acquisition determination unit 106 is configured such that the article indicated by the code associated with the corresponding article identifier determined by the determination unit 104 that matches the received article identifier received by the reception unit 101 is associated with the received article identifier. When it is determined that the user has acquired the code more, the article corresponding to the code may be determined not to be acquired appropriately. In this case, in particular, a determination result indicating that the number of articles acquired corresponding to the code is excessive may be acquired. Such determination can be made, for example, by comparing the above-described count number with the quantity associated with the received article identifier.

  The output unit 107 performs output according to the determination result of the acquisition determination unit 106. The output according to the determination result may be an output of the determination result itself, an output determined according to the determination result, an output specified in advance according to the determination result, or a template according to the determination result It is also possible to use an output created using the above. Note that outputting according to the determination result may include not outputting depending on the determination result. The output here includes display on a display, projection using a projector, printing on a printer, sound output, transmission to an external device, storage in a recording medium, processing to another processing device or other program, etc. It is a concept that includes delivery of results.

  For example, the output unit 107 is a determination result indicating that the article indicated by the code associated with the accepted article identifier determined by the determination unit 104 that the determination result of the acquisition determination unit 106 matches is acquired by the user. In this case, information indicating that the article has been appropriately acquired is output. For example, the output unit 107 outputs a pre-designated confirmation sound as pre-designated information, or displays a pre-designated character string or image such as “product acquisition complete” or the like as a display device such as the head-mounted display 108. To display. Further, the output unit 107 may transmit information indicating that the article has been appropriately acquired to a server device or the like (not shown).

  Further, the output unit 107 is configured so that an article indicated by a code associated with the received article identifier that the determination result of the acquisition determining unit 106 matches is determined by the quantity corresponding to the received article identifier. In the case of the determination result indicating that it has been acquired by the above, information indicating that an appropriate quantity of an appropriate article has been acquired may be output. For example, the output unit 107 outputs a pre-designated confirmation sound as pre-designated information, or displays a pre-designated character string or image such as “predetermined quantity product acquisition” on the head mounted display 108 or the like. Display on the display device. Further, the output unit 107 may transmit information indicating that the correct quantity of articles has been acquired to a server device or the like (not shown).

  Further, when the determination result of the acquisition determination unit 106 is a determination result indicating that the article has not been acquired properly, the output unit 107 outputs information that warns that the article has not been acquired properly. Also good. For example, a warning sound, a character string indicating a warning, or an image may be output. Further, the output unit 107 may transmit information for warning that an article has not been acquired properly to a server device or the like (not shown).

  In particular, when the determination result of the acquisition determination unit 106 is a determination result indicating that the acquired quantity is insufficient, the output unit 107 provides information that warns that the article has not been acquired appropriately. Information indicating that the quantity is insufficient or information indicating the insufficient quantity may be output.

  In particular, when the determination result of the acquisition determination unit 106 is a determination result indicating that the acquired quantity has exceeded, the output unit 107 provides information that warns that the article has not been acquired appropriately. Information indicating that the quantity has been exceeded or information indicating the quantity that has been exceeded may be output.

  The output unit 107 may or may not include an output device such as a display, a speaker, or a communication unit. The output unit 107 can be realized by output device driver software, or output device driver software and an output device.

  Next, an example of operation | movement of the portable information terminal 1 is demonstrated using the flowchart of FIG. Here, an example will be described in which it is determined whether or not an article has been acquired by determining whether or not the position of the hand detected in the captured image is a position corresponding to the code. ing. However, in the present embodiment, it may be determined whether or not the article has been acquired by a process other than the example described here.

  (Step S101) The receiving unit 101 determines whether or not one or more sets of an article identifier and a quantity have been received. The received article identifier is the received article identifier. If accepted, the process proceeds to step S102. If not accepted, the process returns to step S101.

  (Step S102) The photographing unit 102 acquires a photographed image. For example, the imaging unit 102 acquires a frame image of a captured image that is a moving image.

  (Step S103) The determination unit 104 performs a process of detecting one or more codes in the captured image (for example, a frame image) acquired in Step S102, and determines whether or not the code has been detected. When it can detect, it progresses to step S104, and when it cannot detect, it returns to step S102.

  (Step S104) The determination unit 104 substitutes 1 for the value of the counter m.

  (Step S105) The determination unit 104 determines whether or not the corresponding article identifier corresponding to the m-th code among the codes detected in Step S103 matches any one or more of the received article identifiers received in Step S101. to decide. For example, the determination unit 104 uses the identifier management information stored in the identifier storage unit 103 to obtain a corresponding product identifier corresponding to the code identifier read from the mth code, and the corresponding product identifier is determined in step S101. It is sequentially determined whether or not it matches one of the received one or more received article identifiers. If they match, the process proceeds to step S106, and if they do not match, the process proceeds to step S108. In step S115, which will be described later, the received article identifier associated with the acquired information is not used for determining whether or not it matches the received article identifier corresponding to the code. Also good.

  (Step S106) The display unit 105 acquires, as the acquisition notification information, the quantity associated with the received article identifier determined to match in Step S105 among the quantities received in Step S101.

  (Step S107) The display unit 105 displays the acquisition notification information at a position corresponding to the m-th code. For example, the display unit 105 displays the quantity acquired in step S105 on the mth code. For example, the display unit 105 performs display on the head mounted display 108.

  (Step S108) The determination unit 104 increments the value of the counter m by 1.

  (Step S109) The determination unit 104 determines whether or not there is an m-th code in the code detected from the captured image in step S103. If there is, the process returns to step S104. If not, the process proceeds to step S110.

  (Step S110) The acquisition determination unit 106 moves the hand image to the position corresponding to one of the codes from which the acquisition notification information is acquired in Step S106 of the captured image (for example, a frame image) acquired in Step S102. It is determined whether or not is detected. For example, the acquisition determining unit 106 detects a hand in the captured image acquired in step S102, and the detected hand is an area corresponding to any one of the codes from which the acquisition notification information is acquired in step S106. It is determined whether or not they overlap, and if they overlap, it is determined that a hand has been detected at a position corresponding to one of the codes. When a hand is detected at a position corresponding to one of the codes, information indicating the code, for example, information indicating the code identifier or the code area is acquired, and the process proceeds to step S111. If no hand is detected at the corresponding position, the process returns to step S102. In the following, for convenience of explanation, any one of the codes from which the acquisition notification information has been acquired, and the code in which the hand is detected in the corresponding position in step S110 is referred to as the code in which the hand is detected. Call it.

  (Step S111) The acquisition determining unit 106 determines whether or not the code in which the hand is detected in Step S110 is the same as the code in which the hand has been detected immediately before. For example, in step S110, when a hand is detected at a position corresponding to any one of the codes from which the acquisition notification information has been acquired in step S106, a storage unit (not shown) such as a code identifier read from the code By comparing the code identifier of the code whose hand has been detected in step S110 with the code identifier of the code whose hand has been detected in step S110 immediately before (that is, the immediately preceding step), Are the same. If the codes are the same, the process returns to step S102, and if the codes are different, the process proceeds to step S112.

  In addition, the position where the hand is detected immediately before here may be considered as the position where the hand is detected in the immediately preceding captured image (for example, frame image), for example. You may think that it is the position where the hand was detected in the process of step S110 performed previously. In addition, when a hand is not detected at a position corresponding to any one of the codes in the immediately preceding photographed image or in step S110 immediately before the immediately preceding step S110, the acquisition determining unit 106 performs step S110. It may be determined that the code corresponding to the position where the hand is detected is not the same as the code corresponding to the position where the hand was detected immediately before.

  (Step S112) The acquisition determination unit 106 matches the corresponding product identifier corresponding to the code whose hand is detected in Step S111, out of the counters associated with the received product identifier received by the reception unit 101 in Step S101. The counter value associated with the received article identifier is incremented by one. Note that the value of this counter is set to an initial value such as 0 when the receiving unit 101 receives the received article identifier in step S101, for example.

  (Step S113) The acquisition determination unit 106 determines whether or not the counter value incremented in Step S112 matches the quantity associated with the received article identifier corresponding to the counter. If they match, the process proceeds to step S114, and if they do not match, the process proceeds to step S102.

  (Step S114) The output unit 107 outputs information indicating that the article has been appropriately acquired. For example, the output unit 107 outputs information indicating that an article has been appropriately acquired in association with the corresponding article identifier or the received article identifier corresponding to the code whose hand is detected in step S111.

  (Step S115) The acquisition determination unit 106 sets the received article identifier that matches the corresponding article identifier corresponding to the code whose hand is detected in Step S111 among the received article identifiers received by the receiving unit 101 in Step S101. Information (for example, flag information) indicating that the article corresponding to the received article identifier has been acquired is assigned.

  (Step S116) The acquisition determining unit 106 determines whether or not information indicating that the corresponding article has been acquired is assigned to all the received article identifiers received by the receiving unit 101 in step S101. . If the acquired information is associated with all, the process ends. If not, the process returns to step S102.

  Hereinafter, a specific operation of portable information terminal 1 in the present embodiment will be described with an example. In this specific example, the case where the portable information terminal 1 is an apparatus for supporting the shipment operation of an article will be described as an example. Here, an example will be described in which the articles indicated by the received article identifiers transmitted from a server device (not shown) are acquired and collected for shipment from among the articles (products) arranged on the shelves in the warehouse. To do. Here, a case where a color barcode is used as the code will be described as an example.

  FIG. 4 is a diagram (FIG. 4A) showing an example of a shelf on which a plurality of articles arranged in a warehouse is arranged, and a diagram showing an enlarged view of a cord 20 arranged on the shelf (FIG. 4B). )).

  For example, as shown in FIG. 4A, a plurality of articles are arranged on the shelf 50, and the front of the shelf 50 on which one article is arranged is associated with the article identifier of the article. It is assumed that a code 20 is attached. Here, it is assumed that the code 20 is a color barcode. In this example, the color barcode is, as an example, a rectangular design having a maximum of eight colors of light blue, purple, yellow, green, red, blue, and orange on a black background. A case of using one having a configuration arranged in a dimension will be described.

  FIG. 5 is an identifier management table for managing identifier management information stored in the identifier storage unit 103. The identifier management table has attributes of “code ID” and “corresponding article ID”. “Code ID” is a code identifier. The “corresponding article ID” is a corresponding article identifier associated with the code identifier.

  First, the user wears the non-transmissive head mounted display 108 as shown in FIG. 2 on the head. Here, as described above, as an example, it is assumed that the configuration of the portable information terminal 1 is attached to the head mounted display 108.

  The receiving unit 101 of the portable information terminal 1 has at least one set (that is, one or more sets) of a received article identifier to be acquired transmitted from a server device (not shown) and quantity information for acquiring the article indicated by the received article identifier. Item quantity information). Here, it is assumed that a plurality of sets are received. Here, a request identifier, which is an identifier for requesting acquisition of an article, is received together with a set of these accepted article identifiers and quantities. For each request identifier, the user acquires and collects the articles indicated by the corresponding accepted article identifiers by the quantity indicated by the corresponding quantity. The receiving unit 101 accumulates the received information in a storage unit (not shown) or the like.

  FIG. 6 is a reception management table for managing information received by the reception unit 101. The reception management table has attributes of “request ID”, “article ID”, “quantity”, and “acquisition”. “Request ID” is a request identifier, “Article ID” is an article identifier, and here is a received article identifier. “Quantity” is the quantity associated with the received article identifier. “Acquired” is information indicating whether or not the item indicated by “Item ID” in the same record (row) has been acquired in the quantity indicated by “Quantity”, and the value “0” is indicated by “Item ID”. The article indicates that the quantity indicated by “quantity” has not been acquired, and the value “1” indicates that the article indicated by “article ID” has already been acquired by the quantity indicated by “quantity”. It is assumed that the default value of the “acquired” value is “0”.

  The receiving unit 101 may further receive an identifier of a shelf on which an article indicated by each received article identifier is arranged, information indicating a route to the shelf, and the like. Further, the received information may be appropriately displayed on the head mounted display 108 by the display unit 105.

  The imaging unit 102 sequentially captures areas in front of the user, and sequentially acquires frame images constituting a moving image as captured images.

  The determination unit 104 sequentially performs a process of detecting a code for the captured image acquired by the imaging unit. When the code is not detected, the display unit 105 sequentially displays the captured images acquired by the imaging unit 102 on the head mounted display 108. As a result, the user can view the inside of the warehouse via the head-mounted display 108, and can move within the warehouse or hold items in the warehouse.

  Here, for example, if the user has moved to the front of the shelf 50 shown in FIG. 4A and turned his body to the front of the shelf 50, the imaging unit 102 has the shelf 50 positioned in the imaging area. Shoot a part of the front.

  FIG. 7 is a diagram illustrating a frame image obtained by photographing a part of the shelf 50 photographed by the photographing unit 102. The frame image shown in FIG. 7 is displayed on the head mounted display 108.

  The determination unit 104 detects one or more codes in the captured image shown in FIG. For example, the photographed image is binarized, and an image area surrounded by a black frame is detected as a code area. Then, an image of a code that is an image of the detected area is acquired in a memory or the like.

  Here, for example, it is assumed that codes 20a to 20d are detected in FIG.

  Next, the determination unit 104 reads the image of the code 20a as one of the acquired code images, and acquires the code identifier of the code 20a. It is assumed that the acquired code identifier is “C001”.

  The determination unit 104 acquires a corresponding article identifier corresponding to the acquired code identifier. Here, in the identifier management table shown in FIG. 5, the record (line) whose “code ID” value matches “C001” acquired above is detected, and the “corresponding article ID” value of the detected record is detected. “A1001” is acquired.

  In the reception management table shown in FIG. 6, the determination unit 104 has the value “A1001” of “corresponding article ID” that the value of “acquisition” is “0” and the value of “article ID” is acquired above. Search for records (rows) that match. If there is a matching record, get the quantity of this record. Here, since the record in the first row is a record that satisfies the condition, the display unit 105 acquires the value “2” of “quantity” of the record as acquisition notification information. The acquired “quantity” value and “corresponding article ID” value are associated with the “code ID” value and stored in a storage unit (not shown) or the like. The storage here may be temporary storage. Here, since the article identifier associated with the code identifier as a result only needs to be stored in association with the code identifier, “article ID” having the same value instead of the value of “corresponding article ID”. May be accumulated.

  Further, the determination unit 104 performs the same process for the code 20b detected above. If the code identifier acquired for the code 20b is “C002”, the determination unit 104 acquires “A1002” that is the “corresponding article ID” corresponding to the code identifier from the identifier management table of FIG. . Then, in the reception management table illustrated in FIG. 6, the determination unit 104 has the value “acquisition” of “0” and the value of “article ID” is the value “ A record (row) matching “A1002” is searched. In this case, since there is no matching record, the process for this code 20b ends and the process for the next code 20c is performed.

  If the “code ID” acquired for the code 20c is “C003”, the determination unit 104 performs the same processing as described above, so that the determination unit 104 performs the code ID “C003” and the corresponding article ID “A1003” for the code 20c. Further, the display unit 105 acquires the quantity “1” as the acquisition notification information, and the determination unit 104 and the display unit 105 associate these pieces of information and store them in a storage unit (not shown).

  Further, the same processing as described above is performed for the code 20d, but, similarly to the case of the code 20b, the record having the “article ID” that matches the “corresponding article ID” acquired for the code 20d is an identifier. Since it was not detected in the management table, it is assumed that a value of “quantity” or the like was not acquired for this code 20.

  The display unit 105 displays “quantity” which is acquisition notification information acquired and accumulated by the determination unit 104 for these codes in an area corresponding to the area where the code 20a and the code 20c are detected in the captured image shown in FIG. An image in which numbers indicating the values “2” and “1” are superimposed and displayed is displayed on the head mounted display 108. Here, an image in which the value of each “quantity” is arranged on the area where the corresponding code is detected is displayed.

  FIG. 8 is a diagram illustrating a display example of the acquisition notification information by the display unit 105.

  Here, for example, as a result of the user extending his / her hand in order to acquire the article 1001 whose “article ID” is “A1001” arranged above the code 20a, the photographing unit 102 is shown in FIG. In this way, it is assumed that the user's hand has acquired a captured image that overlaps the area where the code 20a associated with the article whose “article ID” is “A1001” is arranged.

  The acquisition determining unit 106 detects the user's hand in the captured image shown in FIG. Here, information indicating the contour 70 of the hand is acquired. Then, it is determined whether or not the detected hand overlaps any one of the codes for which the acquisition notification information is acquired, in other words, the codes for which the acquisition notification information is displayed. Here, the determination is made using the information of the outline of each code acquired when the code is detected. Specifically, it is determined whether or not at least a part of the contour 70 of the hand overlaps at least a part of the contour of each code, and if it overlaps, the detected hand acquires the acquisition notification information as described above. Judge that it overlaps with the code.

  Here, since it is determined that the code 20a and the hand overlap, the acquisition determining unit 106 uses the counter prepared in correspondence with “A1001” that is the “corresponding article ID” associated with the code 20a. Increment the value by one. As a result, the value of the counter becomes “1”.

  The acquisition determination unit 106 determines whether or not the value of the counter matches the quantity “2” stored in the storage unit (not shown) in association with the code 20a. Here, it is determined that they do not match. For this reason, the output unit 107 does not perform output.

  In addition, when a user's hand is detected in the area on the same code 20a in the captured image acquired thereafter, the acquisition determination unit 106 detects a hand in the area on the same code 20a in the immediately preceding captured image. It is judged whether it was done. Here, since the acquisition judgment unit 106 has already detected the user's hand in the area on the code 20a in the immediately preceding photographed image, the acquisition judgment unit 106 displays the “corresponding article ID associated with the code 20a. The value of the counter prepared corresponding to “A1001” that is “is not changed”. Further, even when the user acquires the article corresponding to the code 20a from the shelf 50 and the user's hand does not overlap the area on the code 20a in the captured image, the counter value is not changed.

  Here, as a result of the user reaching out to the shelf 50 in order to acquire an article corresponding to the same code 20a, the photographing unit 102 is arranged in the same manner as in FIG. 9, and the user's hand is arranged with the code 20a. Suppose that a new captured image that overlaps the area is acquired.

  In this case, as described above, the hand overlaps with the code 20a, and in the previous photographed image, the hand does not overlap with the code 20a. Therefore, the acquisition determination unit 106 acquires a new article. Therefore, it is determined that the hand has been extended, and the value of the counter prepared corresponding to “A1001” which is the “corresponding article ID” associated with the code 20a is incremented by one. As a result, the counter value is “2”.

  The acquisition determination unit 106 determines whether or not the value of the counter matches the quantity “2” stored in the storage unit (not shown) in association with the code 20a. Here, it is determined that they match. Therefore, the output unit 107 outputs a confirmation sound designated in advance indicating that the article has been appropriately acquired as information according to the determination result. Further, a message indicating that an article has been appropriately acquired may be output on a part of the head mounted display 108. Note that the confirmation sound, the message, and the like may be output by the output unit 107 when it is determined that the hand is not overlapped with the code 20a in the subsequent captured image.

  Then, the acquisition determination unit 106 detects a record in which the value of the “article ID” matches the corresponding article ID “A1001” in the reception management table illustrated in FIG. 6, and the “acquired” value of the detected record Is updated to “1”.

  Hereinafter, similarly, when the value of “acquisition” of all the records in the reception management table shown in FIG. 6 becomes “1”, the processing of the portable information terminal 1 ends.

  As described above, according to the present embodiment, when one or more codes detected in the photographed image photographed by the photographing unit 102 matches one of the accepted article identifiers accepted by the accepting unit 101, the one or more codes are included. By displaying the acquisition notification information, which is information indicating that the article corresponding to the code is the acquisition target article, at the corresponding position, the position of the article is associated with the area where the photographed image is captured. It can be shown to the user, and the position of the article to be acquired can be appropriately shown to the user. Thereby, for example, it is possible to appropriately support the shipment operation of the article.

  Further, according to the present embodiment, by displaying the quantity of the article corresponding to the code at the position corresponding to the code, the quantity to be obtained of the article to be obtained can be visually and easily shown to the user. It is possible to improve the work efficiency when acquiring an article.

  Further, according to the present embodiment, when each of the two or more codes detected in the photographed image photographed by the photographing unit 102 matches one of the accepted article identifiers accepted by the accepting unit 101, these two or more codes are used. By displaying the acquisition notification information, which is information indicating that the articles corresponding to the two or more codes are the acquisition target articles, at positions corresponding to the codes, a plurality of acquisition target articles are displayed. Even if they are included in the same captured image, the position of a plurality of articles can be shown to the user instead of only one of them, so that the user can be shown appropriately. Accordingly, it is possible to prevent a situation in which only one item can be shown to the user even though there are a plurality of items in the imaging region, and it is possible to improve the efficiency when acquiring the plurality of items.

  Further, according to the present embodiment, it is possible to determine whether or not an article has been acquired from a captured image, and perform output according to the determination result, thereby performing output according to the acquisition status of the article. For example, by outputting when an article is acquired, a notification for confirming acquisition of the article can be sent to a user, an external device, or the like, and the acquisition operation of the article can be confirmed.

  Further, according to the present embodiment, it is determined from the photographed image whether or not an article has been acquired by the quantity associated with the received article identifier corresponding to the article, and an output corresponding to the determination result is output. By doing so, it can be confirmed whether or not the correct quantity of articles has been acquired.

  Further, according to the present embodiment, the movement of the article does not depend on the shape, color, or the like of the article by determining whether the article has been acquired using the hand detected from the captured image. And the acquisition of the article can be appropriately determined.

  In the above embodiment, the case where the head mounted display 108 is mainly used as the display device has been described. However, in the present invention, any display device may be used. For example, the display device may be a monitor such as a liquid crystal monitor or a projector.

  For example, the portable information terminal 1 of the present embodiment can be realized using a mobile phone, a multifunctional mobile phone such as a so-called smartphone, or a mobile device such as a tablet terminal. A portable device monitor can be used. In addition, a camera (not shown) of these portable devices can be used as the photographing unit 102.

  For example, in the above embodiment, the portable information terminal 1 includes a monitor (not shown) instead of the head mounted display 108, and the display unit 105 captures the captured image captured by the imaging unit 102 with respect to the monitor. And the acquisition notification information acquired similarly to the said embodiment may be displayed. Further, when an operation receiving unit (not shown) of the portable information terminal 1 receives an operation for selecting an area in which a code on the captured image corresponding to the displayed acquisition notification information is displayed once from the user, this code The acquisition determination unit 106 may determine that one product associated with is extracted. The fact that one product has been taken out is a concept that includes the handling of one product. Then, the output unit 107 may perform output according to the determination result of the acquisition determination unit 106 as in the above embodiment.

  In this case, a single operation for selecting an area to be received by the operation receiving unit is performed by single-clicking on the area where the code is displayed using a mouse or a trackpad (not shown) or on the monitor. This is an operation of tapping on an area where a code of a touch panel (not shown) is displayed. In this case, the acquisition determination unit 106 may not perform the process of determining whether or not a product has been acquired from the captured image as described in the above embodiment.

  Further, an operation receiving unit (not shown) of the portable information terminal 1 performs an operation for selecting an area in which a code on the captured image corresponding to the displayed acquisition notification information is displayed n times (n is two). If it is received, the acquisition determining unit 106 determines that all the products associated with this code have been extracted in the quantity received by the receiving unit 101 together with the product identifier indicating the product. You may do it. The fact that all the products have been taken out may be considered, for example, that the products indicated by the item quantity information have been taken out (for example, shipped) for the quantity indicated by the item quantity information. Then, the output unit 107 may perform output according to the determination result of the acquisition determination unit 106. Then, the output unit 107 may perform output according to the determination result of the acquisition determination unit 106 as in the above embodiment.

  In this case, the continuous n times of selecting the area received by the operation receiving unit is an operation of clicking n times continuously on the area where the code is displayed using a mouse, a trackpad, etc. (not shown) This is an operation of continuously tapping n times on a region where a code of a touch panel (not shown) arranged on the monitor is displayed. For example, when n is 2, consecutive n operations are so-called double click and double tap.

  In the present invention, the identifier storage unit 103 may not be a storage unit included in the portable information terminal 1 as long as the portable information terminal 1 can access the portable information terminal 1. For example, portable information other than the portable information terminal 1 The terminal 1 may be a storage unit stored in a server device (not shown) that can transmit and receive information by wireless communication or the like. In this case, the portable information terminal 1 transmits the code identifier acquired from the code of the captured image by the determination unit 104 to the server device, and acquires the corresponding article identifier corresponding to the acquired code identifier from the server device. It may be.

  In each of the above embodiments, each process (each function) may be realized by centralized processing by a single device (system), or by distributed processing by a plurality of devices. May be.

  Further, in each of the above embodiments, it goes without saying that two or more communication means (such as a receiving unit and an output unit) existing in one device may be physically realized by one medium.

  In the above embodiment, information related to processing executed by each component, for example, information received, acquired, selected, generated, transmitted, and received by each component. In addition, information such as threshold values, mathematical formulas, addresses, etc. used by each component in processing is retained temporarily or over a long period of time on a recording medium (not shown) even when not explicitly stated in the above description. It may be. Further, the storage of information in the recording medium (not shown) may be performed by each component or a storage unit (not shown). Further, reading of information from the recording medium (not shown) may be performed by each component or a reading unit (not shown).

  In each of the above embodiments, the case where the portable information terminal is stand-alone has been described. However, the portable information terminal may be a stand-alone device or a server device in a server / client system. In the latter case, the output unit or the reception unit receives an input or outputs a screen via a communication line.

  In each of the above embodiments, each component may be configured by dedicated hardware, or a component that can be realized by software may be realized by executing a program. For example, each component can be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. At the time of execution, the program execution unit may execute the program while accessing a storage unit (for example, a recording medium such as a hard disk or a memory).

  In addition, the software which implement | achieves the portable information terminal in each said embodiment is the following programs. That is, this program associates a computer with a receiving unit that receives one or more article identifiers that are identifiers of articles to be acquired, and a code that is arranged at a position corresponding to the article, and associates the article with the article identifier of the article. An imaging unit that acquires a captured image obtained by capturing an area including the one or more codes, and an article identifier of an article that is associated with the one or more codes in the captured image. Corresponding to the code corresponding to the matching item identifier when the determination unit determines whether or not the received item identifier matches one of the received item identifiers, and the determination unit determines that the received item identifier matches the received item identifier Is a program for causing the position corresponding to the code to function as a display unit that displays acquisition notification information that is information indicating that the article corresponding to the code is an acquisition target article.

  In the program, the functions realized by the program do not include functions that can be realized only by hardware. For example, a function that can be realized only by hardware such as a modem or an interface card in an acquisition unit that acquires information or an output unit that outputs information is not included in the function realized by the program.

  Further, the computer that executes this program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

  FIG. 10 is a schematic diagram showing an example of the external appearance of a computer that executes the program and realizes the portable information terminal according to the embodiment. The above-described embodiment can be realized by computer hardware and a computer program executed on the computer hardware.

  In FIG. 10, a computer system 900 includes a computer 901 including a CD-ROM (Compact Disk Read Only Memory) drive 905, a keyboard 902, a mouse 903, and a monitor 904.

  FIG. 11 is a diagram showing an internal configuration of the computer system 900. In FIG. 11, in addition to the CD-ROM drive 905, a computer 901 is connected to an MPU (Micro Processing Unit) 911, a ROM 912 for storing a program such as a bootup program, and the MPU 911, and receives instructions of an application program. A RAM (Random Access Memory) 913 that temporarily stores and provides a temporary storage space, a hard disk 914 that stores application programs, system programs, and data, and a bus 915 that interconnects the MPU 911, ROM 912, and the like Prepare. The computer 901 may include a network card (not shown) that provides connection to the LAN.

  A program that causes the computer system 900 to execute the functions of the portable information terminal and the like according to the above-described embodiment may be stored in the CD-ROM 921, inserted into the CD-ROM drive 905, and transferred to the hard disk 914. Instead, the program may be transmitted to the computer 901 via a network (not shown) and stored in the hard disk 914. The program is loaded into the RAM 913 when executed. The program may be loaded directly from the CD-ROM 921 or the network.

  The program does not necessarily include an operating system (OS) or a third-party program that causes the computer 901 to execute the functions of the portable information terminal according to the above embodiment. The program may include only a part of an instruction that calls an appropriate function (module) in a controlled manner and obtains a desired result. How the computer system 900 operates is well known and will not be described in detail.

  The present invention is not limited to the above-described embodiments, and various modifications are possible, and it goes without saying that these are also included in the scope of the present invention.

  As described above, the portable information terminal or the like according to the present invention is suitable as a portable information terminal or the like that supports work on an article, and is particularly useful as a portable information terminal or the like that indicates the position of an article to be acquired.

DESCRIPTION OF SYMBOLS 1 Portable information terminal 101 Reception part 102 Image | photographing part 103 Identifier storage part 104 Judgment part 105 Display part 106 Acquisition judgment part 107 Output part 108 Head mounted display

Claims (10)

A receiving unit that receives one or more article identifiers that are identifiers of articles to be acquired;
A photographing unit that obtains a photographed image obtained by photographing an area including one or more codes associated with an article identifier of the article, the code being arranged at a position corresponding to the article;
A determination unit that acquires an article identifier of an article associated with one or more codes in the captured image and determines whether the acquired article identifier matches any one of the article identifiers received by the reception unit When,
When the determination unit determines that it matches the article identifier received by the reception unit, the article corresponding to the code is the acquisition target article at a position corresponding to the code corresponding to the matching article identifier. A portable information terminal comprising: a display unit that displays acquisition notification information that is information indicating the information. The reception unit receives two or more article identifiers,
The photographing unit obtains a photographed image obtained by photographing a region including two or more codes,
The determination unit acquires an article identifier of an article associated with each of two or more codes in the photographed image, and each of the acquired article identifiers matches one of the article identifiers received by the receiving unit. Decide whether or not to
The display unit, when the determination unit determines that each of the article identifiers of the articles associated with two or more codes in the captured image matches any one of the article identifiers received by the reception unit The portable information terminal according to claim 1, wherein the acquisition notification information is displayed at a position corresponding to a code corresponding to each of the two or more matching article identifiers. The reception unit further receives a quantity associated with the article identifier;
The said display part displays the quantity matched with the article | item identifier determined that the said judgment part corresponds among the quantities received by the said reception part as said acquisition notification information. Mobile information terminal. The display unit uses, as the acquisition notification information, a quantity associated with an article identifier determined to be the same among the quantities received by the reception unit in the area where the captured image is captured. The portable information terminal according to claim 3, wherein the portable information terminal is displayed at a position on a code corresponding to the article identifier. It is determined using the captured image acquired by the imaging unit whether or not the user has taken out the article indicated by the code associated with the article identifier determined by the determination unit when it matches the article identifier received by the reception unit. An acquisition determination unit to
The portable information terminal according to claim 1, further comprising: an output unit that performs output according to a determination result of the acquisition determination unit. The article indicated by the code associated with the article identifier determined by the determination unit when it matches the article identifier received by the reception unit is acquired by the user in a quantity corresponding to the article identifier received by the reception unit. An acquisition determination unit that determines whether or not it has been performed using a captured image acquired by the imaging unit;
The portable information terminal according to claim 3, further comprising: an output unit that performs output according to a determination result of the acquisition determination unit. 7. The mobile phone according to claim 5, wherein the acquisition determination unit detects a hand in the captured image captured by the imaging unit, and determines whether or not the article has been acquired by the user using the detected hand. Information terminal. It is further equipped with a head mounted display,
The display unit displays the acquisition notification information on the head mounted display,
The portable information terminal according to claim 1, wherein the photographing unit is arranged such that a front direction of the head mounted display is a photographing region. An information processing method performed using a reception unit, a photographing unit, a determination unit, and a display unit,
A receiving step in which the receiving unit receives one or more article identifiers that are identifiers of articles to be acquired;
A photographing step in which the photographing unit obtains a photographed image obtained by photographing an area including one or more codes associated with an article identifier of the article, which is a code arranged at a position corresponding to the article;
Whether the determination unit acquires an article identifier of an article associated with one or more codes in the photographed image, and whether the acquired article identifier matches any one of the article identifiers received in the receiving step A determination step for determining
In the determination step, when it is determined that the article identifier matches with the article identifier received in the reception step, the display unit displays the article corresponding to the code at the position corresponding to the code corresponding to the matching article identifier. An information processing method comprising: a display step of displaying acquisition notification information that is information indicating that the article is a product. Computer
A receiving unit that receives one or more article identifiers that are identifiers of articles to be acquired;
A photographing unit that obtains a photographed image obtained by photographing an area including one or more codes associated with an article identifier of the article, the code being arranged at a position corresponding to the article;
A determination unit that acquires an article identifier of an article associated with one or more codes in the captured image and determines whether the acquired article identifier matches any one of the article identifiers received by the reception unit When,
When the determination unit determines that it matches the article identifier received by the reception unit, the article corresponding to the code is the acquisition target article at a position corresponding to the code corresponding to the matching article identifier. The program for functioning as a display part which displays the acquisition notification information which is information which shows.

Images