JP2022039553A - Information processing system, information processing apparatus, and information processing method - Google Patents

Abstract

To specify the type of an imaging object without a user manually and sequentially scanning the imaging object.SOLUTION: An information processing system comprises an imaging unit, a gate unit, and a control unit. The imaging unit picks up an image of an imaging object in a multifaceted manner to create a picked-up image. The gate unit includes the imaging unit, and the imaging object passes through the gate unit. The control unit extracts an arbitrary area of the imaging object from the picked-up image and specifies the imaging object based on the arbitrary area.SELECTED DRAWING: Figure 2

Description

The present invention relates to an information processing system, an information processing apparatus, and an information processing method.

In the settlement of purchased products in the retail industry, for example, Patent Document 1 discloses an invention that tracks the period from when a product is taken out of a shopping cart to when it is put in a storage bag and determines whether or not the product is properly scanned. Has been done.

JP 2011-54038

In the invention according to this disclosure, it is necessary for the user (purchaser) to manually scan the barcode or the like of the image pickup target (product) to specify the type of the image pickup target in order to identify the product, which is an improvement in this respect. There is room for.

Therefore, an object of the present disclosure made in view of the above points is to specify the type of the image pickup target without the user manually scanning the image pickup target.

In order to achieve the above-mentioned object, the information processing system, which is one aspect of the present disclosure, includes an image pickup unit that multifacetedly captures an image pickup target to generate an image pickup image, and the image pickup section, and the image pickup target passes through the image pickup unit. The present invention includes a gate unit and a control unit that extracts an arbitrary region of the imaging target from the captured image and identifies the imaging target based on the arbitrary region.

Further, the information processing apparatus which is one aspect of the present disclosure includes an image pickup unit that multifacets an image pickup target, a gate portion through which the image pickup target passes, and an image pickup image of the image pickup section. The present invention includes a control unit that extracts an arbitrary region to be imaged and identifies the imaging target based on the arbitrary region.

Further, in the information processing method which is one aspect of the present disclosure, the image pickup unit provided in the gate portion multi-facetedly captures the image pickup target passing through the gate section, and the image pickup target of the image pickup target is taken from the image pickup image of the image pickup section. This includes extracting an arbitrary region and specifying the imaging target based on the arbitrary region.

According to the present disclosure configured as described above, it is possible for the user to specify the type of the imaging target without manually scanning the imaging target sequentially.

FIG. 1 is a functional block diagram showing a configuration of an example of an information processing system according to the present disclosure. FIG. 2 is an external view of an example of an embodiment of the information processing apparatus according to the present disclosure. FIG. 3 is an image diagram of an image captured by the image pickup unit in an example of the embodiment of the information processing system and the information processing apparatus according to the present disclosure. FIG. 4 is a flowchart of an example of an embodiment of the information processing method according to the present disclosure. FIG. 5 is a flowchart of an information processing method according to an embodiment of the information processing method according to the present disclosure, which is different from the example shown in FIG. FIG. 6 is an external view of the information processing apparatus according to the present disclosure in another example. FIG. 7 is an external view when the information processing apparatus according to the present disclosure is used for discriminating the type or solid of the image pickup target mounted on the transport path.

Hereinafter, an example of an embodiment of the information processing system to which the present disclosure is applied will be described with reference to the drawings.

FIG. 1 is a functional block diagram of an embodiment of the information processing system according to the present disclosure.

FIG. 2 is an external view of an embodiment of the information processing system and the information processing apparatus according to the present disclosure. The external view shows the physical configuration (hardware configuration) of the information processing system and the information processing apparatus according to the present disclosure.

As shown in FIG. 1, the information processing system according to the present disclosure includes a detection unit 1, a control unit 2, a gate unit 3, an image pickup unit 4, an accommodation unit 5, a database 6, and a presentation unit 7. However, the accommodating unit 5 and the presenting unit 7 do not necessarily have to be included in the constituent requirements of the information processing system. Further, as shown in FIG. 2, the information processing system and the information processing apparatus according to the present disclosure include a detection unit 1, a gate unit 3, an image pickup unit 4, an accommodation unit 5, and a presentation unit 7 as physical components. It is composed. However, as will be described later in paragraph [0023], when the accommodating portion 5 is a removable basket or bag, the physical component of the information processing system does not necessarily necessarily include the accommodating portion 5. Further, as will be described later in paragraph [0037], the control unit 2 and the database 6 do not necessarily have to be included in the same device, are installed remotely, and are connected to the information processing device by wire communication or wireless communication. May be good.

In the following figures, the solid arrows connecting the functional blocks indicate the flow of control signals (detection signals, imaging signals, presentation signals, etc.) or information to be communicated. The communication indicated by the arrow may be wired communication, wireless communication, or a combination of both. When communication is performed wirelessly, the information processing system includes a communication unit for performing wireless communication conforming to various communication standards including, for example, Bluetooth (registered trademark) or 802.11. The wireless communication may be optical communication.

The detection unit 1 determines whether or not the image pickup target T is inserted inside the gate unit 3. When it is determined that the image pickup target T is moving toward the image pickup unit 4, the detection unit 1 transmits a detection signal meaning the above determination to the control unit 2.

The detection unit 1 may be, for example, a photoelectric sensor, a fiber sensor, a laser sensor, an ultrasonic sensor, or an image sensor. When a photoelectric sensor, a fiber sensor or a laser sensor is used as the detection unit 1, the detection unit 1 may include a floodlight and a photoreceiver or a light emitter / receiver. When an ultrasonic sensor is used as a detection unit 1, the detection unit 1 may include an ultrasonic element. When the image sensor is used as the detection unit 1, the detection unit 1 may include an image pickup optical system and an image pickup element. The imaging optical system may include, for example, one or more lenses and optical members such as an aperture. The image pickup device may include, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor. Further, the image sensor may continuously capture still images (moving images) at an arbitrary frame rate. Further, the detection unit 1 may be configured by one or more sensors, or may be configured by combining one or more types of sensors. For example, in one example of the embodiment shown in FIG. 2, a light emitting unit and a light receiving unit are provided on the upper surface of the gate unit 3 (the surface on which the user inputs the image pickup target T).

When the control unit 2 receives the detection signal from the detection unit 1, the control unit 2 transmits the image pickup signal to the image pickup unit 4. Further, when the control unit 2 receives the captured image from the image pickup unit 4, the control unit 2 extracts an arbitrary region from the captured image and identifies the image pickup target T by image recognition processing while referring to the database 6. The arbitrary region is, for example, an identifier Tm of the imaging target T or an imaging region on which the imaging target T is projected. The identifier Tm of the image pickup target T is, for example, a barcode or a QR code (registered trademark). The identification means by the image recognition process of the image pickup target T may be, for example, recognition of the identifier Tm of the image pickup target T reflected in the image pickup image, or recognition of a specific object by using pattern matching or machine learning. May be good. When a plurality of captured images are received from the image pickup unit 4, the control unit 2 may select one or more captured images satisfying a predetermined condition and perform image recognition processing on the selected captured images. The predetermined condition may be, for example, whether the identifier Tm is clearly projected (whether the control unit 2 can recognize the identifier Tm), or can be calculated together with the recognition of a specific object. It may be whether the reliability (confidence) of a certain recognition result satisfies a predetermined threshold value. Further, when the identifier Tm is captured over a plurality of captured images, the control unit 2 may extract a region of the identifier Tm partially captured from each of the plurality of captured images. The control unit 2 may be able to specify the image pickup target T by using an image processing technique such as image composition or SfM (Structure from Motion) for a plurality of captured images straddling the identifier Tm.

The control unit 2 may record the specified imaging target T as registration target data.

The control unit 2 may calculate the billing amount based on one or more image pickup targets T specified and recorded, determine whether or not the image pickup target T can be specified, and transmit the presentation signal to the presentation unit 7. Whether or not the image pickup target T can be specified means whether or not the image pickup target T is specified successfully or unsuccessfully. The presentation unit 7 may present the billed amount to the user to request payment, or may present the user by presenting whether or not the image pickup target T can be specified.

The control unit 2 may also include a program that executes the above-mentioned processing and the like, and a recording unit that stores data to be registered.

The control unit 2 includes one or more processors and a memory. The processor may include a general-purpose processor that loads a specific program and performs a specific function, and a dedicated processor specialized for a specific process. The general-purpose processor may include a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The dedicated processor may include an application specific integrated circuit (ASIC). The processor may include a PLD (Programmable Logic Device). The PLD may include an FPGA (Field-Programmable Gate Array). The control unit 2 may be either a System (System-on-a-Chip) or a SiP (System In a Package) in which one or a plurality of processors cooperate. The memory may include any one or more of, for example, a semiconductor memory, a magnetic memory, an optical memory, and the like. The semiconductor memory may include a volatile memory and a non-volatile memory. The magnetic memory may include, for example, a hard disk, a magnetic tape, or the like. The optical memory may include, for example, a CD (Compact Disc), a DVD (Digital Versaille Disc), a BD (Blu-ray (registered trademark) Disc), and the like.

The gate portion 3 is installed to input the image pickup target T. That is, the user puts the product (imaging target T) into the gate portion 3. The gate unit 3 includes an image pickup unit 4.

The gate portion 3 may be, for example, cylindrical and may have a shape having a cavity inside. In this case, the detection unit 1 may be provided at the entrance of the cavity of the gate portion 3 and the image pickup unit 4 may be provided in the cavity. Further, the gate portion 3 may be provided with a support portion 8 for supporting the gate portion 3. The support portion 8 may have, for example, a cantilever shape, a double-sided shape, a suspended shape, or the like, or may have another shape. For example, in an example of the embodiment shown in FIG. 2, the support portion 8 supports the gate portion 3 with a double-sided structure and is installed on the ground plane U.

When the image pickup unit 4 receives the image pickup signal from the control unit 2, the image pickup unit 4 multifacetedly captures the image pickup target T to generate an image pickup image. That is, the image pickup unit 4 multifacetedly captures the image pickup target T input to the gate portion 3. The multifaceted image captured by the image pickup unit 4 is, for example, the entire circumference facing the inner surface of the gate portion 3 of the image pickup target T. It is an image. The image pickup unit 4 takes an image, and the generated captured image is transmitted to the control unit 2. Further, the image pickup unit 4 may be provided with a plurality of images to be imaged T input to the gate unit 3 so as to be capable of multi-faceted imaging. Further, the image pickup unit 4 may include a fisheye camera or an omnidirectional camera. Further, the number of captured images generated by the imaging unit 4 is not limited to one, and may be a plurality of images.

The image pickup unit 4 may include, for example, one or more lenses and an optical member such as an aperture. The image pickup device may include, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor. Further, the image pickup unit 4 may continuously capture still images at an arbitrary frame rate. Continuously capturing still images can be rephrased as capturing moving images. For example, in an example of the embodiment shown in FIG. 2, four image pickup units 4 are provided on the inner surface of the cavity of the gate unit 3 so that the image pickup target T can be imaged from multiple surfaces.

When the detection unit 1 includes an image pickup optical system and an image pickup element, the image pickup unit 4 may be configured integrally with the detection unit 1. That is, the detection unit 1 and the image pickup unit 4 may use the same image pickup optical system and image pickup element.

FIG. 3 is an image diagram of an captured image generated by the imaging unit 4 imaging the image pickup target T.

For example, when the gate unit 3 is provided with four image pickup units 4, four captured images are generated as shown in FIG.

In this case, among the four captured images generated, the image in which the identifier Tm of the imaging target T is clearly projected is Pic. It is 2.

The control unit 2 executes image recognition processing on four captured images, and the identifier Tm is clearly projected in the Pic. Select the captured image of 2. The control unit 2 is the selected Pic. The image capture target T is specified using the image captured in 2.

The accommodating portion 5 accommodates the image pickup target T that has passed through the gate portion 3. That is, the user inputs the image pickup target T into the accommodating unit 5 via the gate unit 3.

The accommodating portion 5 may be a container represented by, for example, a removable basket or bag. The accommodating portion 5 may be provided on the lower surface (ground plane U side) of the gate portion 3. The accommodating portion 5 may be manually fixed to the lower surface of the gate portion 3 by the user, or may be automatically fixed to the lower surface of the gate portion 3 by the information processing system. For example, in an example of the embodiment shown in FIG. 2, a certain space is secured between the gate portion 3 and the ground plane U, and the accommodating portion 5 is placed in this space.

Database 6 records related information about the imaging target T. The related information may be, for example, a product name, a price, a product image, and an identifier Tm.

Database 6 is configured to include one or more memories. The memory may include any one or more of, for example, a semiconductor memory, a magnetic memory, an optical memory, and the like. The semiconductor memory may include a volatile memory and a non-volatile memory. The magnetic memory may include, for example, a hard disk, a magnetic tape, or the like. The optical memory may include, for example, a CD (Compact Disc), a DVD (Digital Versaille Disc), a BD (Blu-ray® Disc), and the like.

The presentation unit 7 presents information related to the identification process of the image pickup target T by the control unit 2. For example, any of the product name, price, or quantity of the image pickup target T specified by the control unit 2 may be presented. Further, any one of a product name list, a total price, and a total number of images of one or more images T specified by the control unit 2 may be presented. Further, if the identification of the image pickup target T fails, an error may be presented to the user to re-input the image pickup target T or to equip the information processing system with a separate code scanner to urge the user to reidentify the image pickup target T by the code scanner. .. The code scanner may be based on a code scanning method including, for example, a CCD method, a laser method, a pen method, or the like.

The presenting unit 7 may be, for example, at least one of a display, a speaker, a motor, and a light emitting device, or may be a combination of at least two or more of these. The light emitting device may be a halogen lamp, an LED (Light Emitting Diode) lamp, an EL (Electroluminescence) lamp, or the like. The presentation unit 7 may display image information via a display. The presentation unit 7 may output sound via a speaker. The presentation unit 7 may vibrate the motor. The presentation unit 7 may make the light emitting device emit light.

Although the control unit 2 and the database 6 are not shown in FIG. 2, they do not necessarily have to be included in the same information processing apparatus. For example, the information processing system may separately include one or more arithmetic units, and the arithmetic unit may include a control unit 2 and a database 6, and the detection unit 1 included in the same information processing apparatus of the information processing system. And the image pickup unit 4, the control unit 2 included in the calculation unit, and the database 6 may communicate with each other. The arithmetic unit may be, for example, a computer, a server, a cloud service, or the like, or may be a combination thereof. Further, the presentation unit 7 is provided separately from the detection unit 1, the gate unit 3 and the image pickup unit 4 integrally configured in FIG. 2, but is embedded in, for example, the outer surface of the gate unit 3 (opposite surface inside the cavity). It may be in shape.

Next, the image pickup and recognition processing flow of the information processing system according to the present disclosure will be described with reference to FIG.

FIG. 4 is a flowchart of the image pickup and recognition process of the information processing system.

<S01> The detection unit 1 detects the image pickup target T. Then, the detection unit 1 transmits a detection signal to the control unit 2. After that, it shifts to <S02>.

<S02> After the control unit 2 receives the detection signal, the control unit 2 transmits the image pickup signal to the image pickup unit 4. After that, the imaging unit 4 that has received the imaging signal captures the detection target and generates an captured image. After that, the image pickup unit 4 transmits the captured image to the control unit 2. After that, it shifts to <S03>.

<S03> After the control unit 2 receives the captured image from the image pickup unit 4, the control unit 2 executes an image recognition process on the captured image and attempts to extract the identifier Tm of the image pickup target T. When the control unit 2 succeeds in extracting the identifier Tm of the image pickup target T, the process proceeds to <S04>. If the control unit 2 fails to extract the identifier Tm of the image pickup target T, the process proceeds to <S06>.

<S04> When the control unit 2 succeeds in extracting the identifier Tm of the image pickup target T, the control unit 2 identifies the image pickup target T based on the extracted identifier Tm. After that, it shifts to <S05>.

<S05> After the control unit 2 identifies the image pickup target T, the control unit 2 records the specified image pickup target T as a registration target. The recording destination is, for example, a recording unit included in the control unit 2. After that, the process ends.

<S06> When the control unit 2 fails to extract the identifier Tm of the image pickup target T, the control unit 2 executes error processing. The error processing means, for example, that the control unit 2 transmits a presentation signal to the presentation unit 7, and the presentation unit 7 presents an error indicating that the identification of the image pickup target T has failed. After that, the process ends.

In the flowchart of the image pickup and recognition process of the information processing system shown in FIG. 4, the image pickup target T is specified based on the identifier Tm, but for example, specific object recognition can also be used. With reference to FIG. 5, the image pickup and recognition processing flow of the information processing system when the specific object recognition is used will be described.

In FIG. 5, since <S01>, <S02>, <S05> and <S06> are the same processes as the identification of the image pickup target T based on the identifier Tm shown in FIG. 4, the description thereof will be omitted.

<S03'> After the control unit 2 receives the captured image from the image pickup unit 4, the control unit 2 executes an image recognition process on the captured image to extract the image pickup target T region on which the image pickup target T is projected. Try. When the control unit 2 succeeds in extracting the image pickup target T region of the image pickup target T, the process proceeds to <S04'>. If the control unit 2 fails to extract the image pickup target T region of the image pickup target T, the process proceeds to <S06>.

<S04'> When the control unit 2 succeeds in extracting the image pickup target T region of the image pickup target T, the control unit 2 performs specific object recognition for the extracted image pickup target T region and specifies the image pickup target T. After that, it shifts to <S05>.

Up to this point, the present disclosure of the image registration system has been described based on various drawings and examples, but it should be noted that those skilled in the art can easily make various modifications and modifications based on the present disclosure.

FIG. 6 is an external view of the information processing apparatus according to the present disclosure in different embodiments.

For example, as shown in FIG. 6, the gate portion 3 may be shorter in the ground plane U direction than the example shown in FIG.

Further, as shown in FIG. 6, the gate portion 3 may include a hook portion 9 for hooking the handle portion of the accommodating portion 5. The hooking portion 9 may be, for example, a hook or a stopper.

Further, the information processing system may install a light projecting unit that emits light at the same timing as the image pickup of the image pickup unit 4 inside the cavity of the gate unit 3. By installing the light projecting unit, the brightness of the captured image may be improved and the image pickup target T may be easily specified.

Further, the control unit 2 may perform correction processing on the image captured by the image pickup unit 4. The correction process may include, for example, correction of saturation, color tone, contrast, distortion, and the like. The correction process may be performed so that the identifier Tm of the imaging target T or the imaging target region can be extracted from the captured image. By performing the correction process, it may be easy to extract the identifier Tm of the imaging target T or the imaging target region from the captured image.

Further, when the user's finger or arm or the like is reflected in the image captured by the image pickup unit 4, the control unit 2 may recognize the finger or arm or the like and specify the image pickup target T in consideration of the existence thereof. .. For example, when the control unit 2 determines that the hand H covers the identifier Tm captured in the captured image, the control unit 2 may specify the image pickup target T by recognizing a specific object. Further, when the control unit 2 determines that the finger H covers the area of the image pickup target T excluding the identifier Tm captured in the captured image, the control unit 2 may specify the image pickup target T by recognizing the identifier Tm. good.

Further, the accommodating unit 5 may be provided with a sensor for determining whether or not the image pickup target T is accommodated in the accommodating unit 5. The sensor may be, for example, a photoelectric sensor, a fiber sensor, a laser sensor, an ultrasonic sensor, an image sensor or a weight sensor. When the sensor installed in the accommodating unit 5 determines that the image pickup target T is accommodated in the accommodating unit 5, the accommodating signal is transmitted to the control unit 2. The control unit 2 may compare the presence / absence of reception of the captured image from the image pickup unit 4 with the accommodation signal, and determine whether the image pickup target T is correctly accommodated. When it is not determined that the image pickup target T is correctly accommodated, in other words, for example, when the control unit 2 receives the captured image but does not receive the accommodation signal, or the captured image is not received. Nevertheless, when the accommodation signal is received, the control unit 2 may execute error processing.

Further, the presentation unit 7 may not be presented to the user but may be presented to, for example, a store employee or a system administrator. The user in this case is a purchaser in the store. By presenting it to the store employee or system administrator, the store employee or system administrator can grasp the imaging and recognition processing status, and the user can quickly follow up as needed. In some cases.

Further, the user according to the present disclosure is not limited to the purchaser. The user may be, for example, a store employee or a system administrator.

Further, the captured image or the imaging target region on which the imaging target T imaged by using the information processing system according to the present disclosure may be recorded in the database 6 in association with the specified imaging target T. By associating the captured image or the imaging target area with the imaging target T in a database, it may be easy to additionally collect image data used in, for example, pattern matching or machine learning.

Further, the information processing system and the information processing apparatus according to the present disclosure can also be used for discriminating the type or solid of the image pickup target T mounted on the transport path C such as a belt conveyor. For example, the information processing device may be installed on the belt conveyor as shown in FIG. FIG. 7 is an external view when the information processing device is installed on the belt conveyor. In this case, the user may be, for example, a field worker or a work supervisor.

Further, as described above, the solution means of the present disclosure have been described as an information processing system, an information processing apparatus, and an information processing method, but the present disclosure can also be realized as an embodiment including these. It is also possible to realize a program and a storage medium on which the program is recorded, which is substantially equivalent to these, and it should be understood that these are also included in the scope of the present disclosure.

1 Detection unit 2 Control unit 3 Gate unit 4 Imaging unit 5 Storage unit 6 Database 7 Presentation unit 8 Support unit 9 Hooking unit H Hand finger T Imaging target Tm identifier

Claims (7)

An imaging unit that captures an image to be imaged from multiple surfaces and generates an image.
A gate portion provided with the image pickup unit and through which the image pickup target passes,
A control unit that extracts an arbitrary region of the imaging target from the captured image and identifies the imaging target based on the arbitrary region is provided.
Information processing system. Further, it is provided with a detection unit for detecting the image pickup target.
The information processing system according to claim 1, wherein the image pickup unit performs imaging when the detection unit detects the image pickup target. The information processing system according to claim 1 or 2, wherein the image pickup target that has passed through the gate portion can be accommodated in the accommodating portion. The information processing system according to any one of claims 1 to 3, wherein the captured image is an omnidirectional image facing the inner surface of the gate portion to be imaged. The imaging unit generates a plurality of captured images and generates a plurality of captured images.
The control unit selects one or more images satisfying a predetermined condition from a plurality of the captured images, extracts an arbitrary region to be imaged from the selected captured images, and captures the image based on the arbitrary region. The information processing system according to any one of claims 1 to 4, which specifies an object. An image pickup unit that captures images to be imaged from multiple sides,
A gate portion provided with the image pickup unit and through which the image pickup target passes,
It includes a control unit that extracts an arbitrary region of the imaging target from the image captured by the imaging unit and identifies the imaging target based on the arbitrary region.
Information processing equipment. The imaging unit provided in the gate unit multifacetedly images the image pickup target passing through the gate unit.
An arbitrary region of the imaging target is extracted from the image captured by the imaging unit, and the imaging target is specified based on the arbitrary region.
Information processing method.

Images