JP2023095103A - Electronic apparatus, method for controlling electronic apparatus, and program - Google Patents

Abstract

To provide an electronic apparatus, a method for controlling an electronic apparatus, and a program for more easily managing objects by using a result of detection of the objects from an image.SOLUTION: In an electronic apparatus 100, a CPU 101 controls to display code information to be registered, controls to perform processing of photographing an object corresponding to the code information, controls to perform learning processing on the object by using an image photographed in the photographing processing, and controls to record a result of learning from the learning processing and the code information to be registered in association with each other.SELECTED DRAWING: Figure 2

Description

The present invention relates to technology for managing objects by detecting objects from images.

A system is known that detects an object (for example, a product) from an image and performs automatic processing.

In Patent Document 1, a purchaser obtains image information of products placed in a product cart and notifies it to a product settlement device, and the product settlement device recognizes the products by image recognition, and recognizes each product stored in the product cart. It is proposed to accept total price liquidation.

Japanese Unexamined Patent Application Publication No. 2003-18733

If a system other than an existing system (for example, a cash register) is introduced as in the commodity checkout apparatus of Patent Literature 1, system construction becomes complicated. For example, in a store, if there are products for which payment is made using a cash register and products for which payment is automatically made using image recognition without using a cash register, it is necessary to introduce a system for automatic payment separately from the cash register. As a result, costs such as system installation costs and installation space increase. In addition, there are multiple payment methods, and product sales management becomes complicated. The same applies not only to merchandise management at stores, but also to parts management at factories and article management in physical distribution.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a mechanism for more easily managing objects by using detection results of objects from images.

In view of the above problems, the electronic device of the present invention is
display control means for controlling display of code information to be registered;
a photographing control means for controlling to perform photographing processing of an object corresponding to the code information;
a control means for controlling to perform a learning process of the object using the image captured in the imaging process;
and recording control means for performing control so as to record the learning result of the learning process and the code information to be registered in association with each other.

ADVANTAGE OF THE INVENTION According to this invention, a target object can be managed more easily using the detection result of the target object from an image.

It is a system block diagram in this embodiment. 2 is a block diagram showing an example of a hardware configuration of electronic device 100. FIG. It is a flow chart of product recognition application processing. It is a flow chart of product registration processing. It is an example of a display of the product recognition result in product recognition application processing. It is a display example in product registration processing. This is a display example in the product registration process. It is an example of a registration information table.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing an example of a system configuration according to an embodiment of the invention.

The electronic device 100 is an electronic device such as a tablet terminal with a camera. A cash register 200 is a cash register system for accounting in a store, and a barcode reader 201 is a barcode reader connected to the cash register 200 . Products 41 to 45 are products to be recognized in this system. Products to be recognized in this embodiment are assumed to be products (for example, side dish bread) that are not directly attached with paper or stickers on which barcodes are written. A tray 30 is a tray on which products are placed. The table 20 is a table (mounting table) on which the tray 30 is placed, such as a counter of a store. A manager 10 is a person (for example, a store clerk) who operates this system. In this system, a customer (not shown) or a worker 10 (for example, a store clerk) who puts products on a tray puts the tray 30 with products 41 to 45 on the table 20 and mounts it on the electronic device 100 . The tray 30 and the products 41 to 45 are photographed by the attached camera. After that, the electronic device 100 automatically recognizes the product type (item) of each of the products 41 to 45 from the photographed image, and displays the barcode corresponding to the recognized product on the display section of the electronic device 100 . The worker 10 reads this bar code with the bar code reader 201 and settles the account at the register 200 . If it is a self-checkout, this work may be done by the customer. In this way, simply by introducing the electronic device 100 into the existing system (cash register 200), payment and product management using the existing system (cash register 200) become possible.

FIG. 2 shows an example configuration of an electronic device 100 as an example of a device to which the present invention can be applied. The electronic device 100 can be configured using a tablet terminal, a smart phone, a personal computer (hereinafter referred to as PC), or the like.

2, CPU 101, memory 102, nonvolatile memory 103, image processing unit 104, display 105, operation unit 106, recording medium I/F 107, external I/F 109, and communication I/F 110 are connected to internal bus 150. ing. Each unit connected to the internal bus 150 is capable of exchanging data with each other via the internal bus 150 .
The memory 102 is, for example, a RAM (a volatile memory using a semiconductor element, etc.). The CPU 101 controls each part of the electronic device 100 using the memory 102 as a work memory, for example, according to a program stored in the nonvolatile memory 103 . The nonvolatile memory 103 stores image data, audio data, other data, various programs for the CPU 101 to operate, and the like. The nonvolatile memory 103 is composed of, for example, a hard disk (HD) or ROM.

Under the control of the CPU 101 , the image processing unit 104 processes images stored in the nonvolatile memory 103 and the recording medium 108 , video signals obtained via the external I/F 109 , and images obtained via the communication I/F 110 . , various image processing is performed on the captured image. The image processing performed by the image processing unit 104 includes A/D conversion processing, D/A conversion processing, image data encoding processing, compression processing, decoding processing, enlargement/reduction processing (resize), noise reduction processing, color conversion, and so on. processing, etc. The image processing unit 104 may be composed of a dedicated circuit block for performing specific image processing. Further, depending on the type of image processing, the CPU 101 can perform image processing according to a program without using the image processing unit 104 .

The display 105 displays an image, a GUI screen that constitutes a GUI (Graphical User Interface), and the like, under the control of the CPU 101 . CPU 101 controls each unit of electronic device 100 to generate a display control signal according to a program, generate a video signal to be displayed on display 105 , and output the video signal to display 105 . A display 105 displays an image based on the output image signal. Electronic device 100 itself includes an interface for outputting a video signal to be displayed on display 105, and display 105 may be configured by an external monitor (such as a television).

The operation unit 106 is an input device (accepting device) for accepting user operations, including a character information input device such as a keyboard, a pointing device such as a mouse and a touch panel, buttons, dials, touch sensors, and touch pads. It should be noted that the touch panel is an input device that is superimposed on the display 105 and configured in a two-dimensional manner so as to output coordinate information according to the touched position.

A recording medium I/F 107 can be loaded with a recording medium 108 such as a memory card, a CD, or a DVD. I do. The external I/F 109 is an interface for connecting to an external device via a wired cable or wirelessly and inputting/outputting a video signal and an audio signal. The communication I/F 110 is an interface for communicating with an external device, the Internet 111, etc., and transmitting/receiving various data such as files and commands.

The camera unit 112 is a camera unit including an image sensor (image sensor) including a CCD or CMOS element for converting an optical image into an electric signal. The camera unit 112 includes a lens group (photographing lens) including a zoom lens and a focus lens, a shutter having an aperture function, an image sensor, an A/D converter that converts an analog signal output from the image sensor into a digital signal, and an image sensor. Contains a barrier that covers the system to prevent contamination and damage. The image processing unit 104 performs resizing processing such as predetermined pixel interpolation and reduction, and color conversion processing on data acquired by imaging with the camera unit 112 . Based on the calculation result obtained by the image processing unit 104, the CPU 101 performs exposure control, distance measurement control, and AWB (auto white balance) processing. Display image data captured by the camera unit 112 and image-processed by the image processing unit 104 is displayed on the display 105 . The digital signal captured by the camera unit 112, A/D converted once by the A/D converter, and stored in the memory 102 is converted to analog by the D/A converter, and sequentially transferred to the display 105 for display. Live view display (LV display) can be performed. The live view can be displayed in a still image shooting standby state, a moving image shooting standby state, or when recording a moving image, and the imaged subject image is displayed almost in real time. The CPU 101 controls the camera unit 112 to start operations such as AF (autofocus) processing, AE (automatic exposure) processing, AWB processing, etc., in response to a shooting preparation instruction based on a user operation performed on the operation unit 106. It controls the image processing unit 104 . In response to a photographing instruction, the CPU 101 performs main exposure, reads out signals from the imaging unit element, processes the captured image in the image processing unit 104 to generate an image file, and records the image in the recording medium 108 . It controls to start the operation of a series of photographing processes (actual photographing). A shooting instruction can be given by a user operation on the operation unit 106 . The camera unit 112 is capable of capturing still images and moving images. Note that the electronic device 100 itself may include an interface for inputting an image captured by a camera, and the camera unit 112 may be configured with an external camera.

FIG. 3 is a flow chart showing an example of processing at checkout in the product recognition application in electronic device 100 . Each process in this flow chart is implemented by the CPU 101 developing a program stored in the nonvolatile memory 103 in the memory 102 and executing the program. When the application software (merchandise recognition application) according to the present embodiment installed in the electronic device 100 is activated, the approval recognition application process of FIG. 3 is started.

In S<b>301 , the CPU 101 starts capturing images from the camera unit 112 , sequentially holds predetermined frames of images for LV display (LV images) in the memory 102 , and performs LV display on the display 105 .

In S302, the CPU 101 determines whether or not the tray 30 has been placed on the stand 20 (whether it is within the angle of view of the camera section 112) based on the image held in the memory 102. FIG. For example, it can be detected that a new tray 30 has been placed when a subject with many linear components, which is assumed to be a tray, enters the angle of view. If it is placed, the process proceeds to S303; otherwise, the process proceeds to S304.

In S<b>303 , the CPU 101 determines whether or not the image held in the memory 102 is stabilized (stopped). For example, it calculates the amount of change in luminance between the images of the previous and subsequent frames, and determines whether the images have stabilized (whether the tray 30 has stopped completely). Further, for example, it may be detected that the image is stabilized (that the amount of motion is equal to or less than a predetermined amount) based on a condition such as motion vectors detected from the preceding and succeeding frames being equal to or less than a predetermined value. If the image is stable, proceed to S304; otherwise, proceed to S308.

In S304, the CPU 101 inputs the LV image (image captured by the camera unit 112) held in the memory 102 to the learned model stored in the nonvolatile memory 103 and held in the memory 102, and puts it on the tray 30. Infer the type of merchandise (item) placed. The inference result is held in memory 102 . This processing is image recognition processing for analyzing an image captured by the camera unit 112 and determining whether or not the captured image includes a registered object (product) registered in the product registration processing described later. be. In other words, it is a process of detecting each of a plurality of registered objects (products) from an image captured by the camera unit 112 . As a result, for example, items such as “roll bread” and “melon bread” can be detected from the image captured by the camera unit 112 . Note that the recognition processing itself in S304 may be performed by an external device (for example, a server device) (not shown) that can communicate with the electronic device 100 . In this case, the CPU 101 controls to transmit the image captured by the camera unit 112 to the external device and perform recognition processing. Then, the recognition result (inference result) is received from the external device and recorded in the memory 102 .

In S305, the CPU 101 refers to the product type inference result held in the memory 102, and determines whether or not any of a plurality of registered products (objects) has been detected. If any of the registered products (objects) is detected, proceed to S306; otherwise, that is, if none of the registered products (objects) are detected, proceed to S307. .

In S<b>308 , the CPU 101 reads from the nonvolatile memory 103 the product identification information (for example, product code, product number) recorded in association with the item of the product detected in S<b>306 . A barcode is generated based on the read identification information, and an image of the barcode is displayed on the display 105 . The identification information is information corresponding to the product item (object type), and is information necessary to generate a barcode. Specifically, from the registration information table shown in FIG. Generate a barcode that represents a numeric string (product code). The check digit may not be read out from the registration information table, but may be calculated based on the previous 12-digit number string. FIG. 5 shows a display example in S306.

FIG. 5 is a display example of the product recognition result on the display 105. As shown in FIG. A photographed image 500 is an image photographed by the camera unit 112, and is a still image to be detected in S304. An LV image captured by the camera unit 112 may be displayed instead of the still image. In the illustrated example, the photographed image 500 includes products 501 to 504, which are detected. A product label indicating the product name and a product detection frame indicating the detected object are displayed in each portion of the image 500 where the product is detected. For example, a product detection frame 508 indicating that the product 501 has been detected is displayed at a position surrounding the product 501 . Also, a product label 507 displaying the product name of the product 501 is displayed at a position corresponding to the position where the product 501 is detected (in this embodiment, above the product detection frame 508). Although the product label 507 in the figure is labeled as "product name 1" for convenience, the name of the product 501 is actually displayed as, for example, "roll bread". Similarly, product labels and product detection frames are displayed for products 502 to 504, respectively.

The barcode display area 510 is an area where the product names and barcodes detected (recognized) from the captured image 500 are collectively displayed (as a list). The barcode display area 510 displays a list of product names and barcodes of all recognized products. Barcodes 511-514 are barcodes corresponding to the detected products 501-504, respectively. By reading the bar codes 511 to 514 with the bar code reading unit 201, information specifying the product is input to the cash register 200, and information necessary for accounting such as the name and price of the product can be obtained. In other words, it is possible to perform accounting processing similar to accounting at a normal cash register.

Product names 515 to 518 are the names of the detected products 501 to 504, respectively. For the sake of convenience, "product name 1" and "product name 2" are shown, but in reality, product names (information for identifying items) such as "roll bread" and "melon bread" are displayed.

By collectively displaying the barcodes of the detected products in the barcode display area 510, the barcode reading unit 201 can easily perform the reading operation. The barcodes 511 to 514 may be displayed in the vicinity of the corresponding product on the photographed image 500 (for example, the position of each product label). It is necessary to read the This not only increases the amount of movement of the barcode reading unit 201, but also requires different movements depending on the arrangement of the pan rather than a uniform movement, making it impossible to perform quick reading. Therefore, there is a possibility that the workload of the worker 10 and the time required for accounting processing will increase. On the other hand, in the present embodiment, a plurality of barcodes respectively corresponding to the plurality of detected products are collectively displayed in the barcode display area 510 instead of near the detected products in the photographed image 500 . By displaying in this way, all barcodes can be read by moving the barcode reading unit 201 within the limited range of the barcode display area 510, so the moving distance of the barcode reading unit 201 is shortened. be able to. Also, a plurality of barcodes are arranged in line (in this embodiment, they are displayed in a vertical line). Therefore, all barcodes can be read by moving the barcode reading unit 201 in a uniform motion (in this embodiment, by moving vertically once). Therefore, the worker 10 can quickly read the bar code with a familiar and simple motion, and the workload and the time required for accounting processing can be reduced.

Furthermore, when a plurality of identical products (products of the same item, products of the same type) are detected from the photographed image 500, a plurality of barcodes corresponding to the detected number are displayed in the barcode display area 510. FIG. In the illustrated example, two products (for example, melon bread) with product name 2 are detected (products 502 and 504 are the same item). Therefore, two barcodes of product name 2 are displayed in the barcode display area 510 (barcodes 512 and 513). Therefore, barcodes 512 and 513 are the same barcode (identical barcode). By displaying in this manner, all the detected products and the correct number of products can be displayed at the cash register 200 by simply reading the bar codes displayed by the bar code reading unit 201 in order without inputting the number of products. can be entered. That is, it is possible to omit the numerical input operation of inputting the number of products at the register 200 . Therefore, the operation can be simplified and work efficiency is improved.

In S307, the CPU 101 determines whether the image is stable, as in S303. If the image is stable, the process of S307 is repeated, and if the image is no longer stable (when a change in the image is detected), the process proceeds to S308. When there is a change in the image, the tray 30 may be removed, an additional product (bread) may be placed on the tray 30, or the product (bread) may be removed from the tray 30. . In such a case, it is necessary to perform the image recognition processing of S304 again. Conversely, until there is a change in the image, the image recognition processing is not performed, and the bar code once displayed in S306 continues to be displayed. The barcode can be read by the barcode reading unit 201 .

In S<b>308 , the CPU 101 determines whether or not the tray 30 has been pulled out (moved) from the table 20 based on the image held in the memory 102 . For example, removal of the tray 30 can be detected when a subject with many linear components assumed to be the tray 30 moves out of the angle of view. If it has been removed, the process proceeds to S309, and if it has not been removed, the process proceeds to S303. If the product is added to or removed from the tray 30, the determination in S308 is No, and after waiting for the image to stabilize in S303, image recognition processing is performed in S304. As a result, it is possible to detect products in a situation where products have been added to or removed from the tray 30, and update the screen described with reference to FIG. Further, in response to detecting that the tray 30 has been pulled out in S308, the screen is changed so that all the barcodes displayed in the barcode display area 510 are hidden, and the product label and product detection frame are hidden. You may update. By doing so, the worker 10 can recognize that normal image recognition processing is being performed. In addition, it is possible to prevent an erroneous operation such as re-accounting a product for which accounting processing has already been completed (duplicate accounting).

In S309, the CPU 101 determines whether or not there is an instruction operation to shift to the product registration mode. For example, when an icon (not shown) for switching to the registration mode displayed on the display 105 is operated, it is determined that an instruction operation to switch to the product registration mode has been performed. If there is an instruction operation to shift to the product registration mode, the process proceeds to S310, and if not, the process proceeds to S311.

In S310, the CPU 101 performs product registration processing. Details of the product registration process will be described later with reference to FIG.

In S<b>311 , the CPU 101 determines whether or not there is an end event such as closing the application software according to the present embodiment or turning off the power of the electronic device 100 . If there is no end event, the process proceeds to S302, and if there is an end event, the process of FIG. 3 ends.

FIG. 8 shows an example of a registration information table (simply referred to as registration information) recorded in the nonvolatile memory 103 and recording the registration status (registration details) of the product. The product is registered in the registration information table by the product registration process of S310. Using the information registered in the registration information table, image recognition processing in S304 and bar code display in S306 are performed. In the registration information table, the first two-digit number string for the in-store code, the item code (10-digit number string), the check digit (1-digit number), whether it has been registered, the product name, and the recognition conditions are related. remembered. Note that "-" in the drawing indicates no data (NULL). Although the registration information is recorded in the non-volatile memory 103 in this embodiment, it may be recorded in a server device capable of communicating with the electronic device 100 or in a removable recording medium 108 . By doing so, one piece of registration information can be shared by a plurality of electronic devices, and can be used by a plurality of electronic devices corresponding to a plurality of cash registers.

An in-store code is a code that can be arbitrarily numbered by at least one of a store, an organization, and a business. In this embodiment, when a product is registered in product registration processing, an in-store code, which is a 13-digit JAN code starting with 20-29, is stored in association with the product to be registered. By using the in-store code as the code to be registered, it is possible to prevent duplication with the JAN code for distribution.

In the registration information table of FIG. 8, the first two-digit number string for the in-store code is a number string of 20 to 29 indicating the in-store code. An item code is a number for identifying a product. The check digit is a check digit added to detect errors in the number string, and is a numerical value determined based on a 12-digit number string in total, consisting of two digits from 20 to 29 and a 10-digit item code. The product code (in-store code) is a 13-digit number string combining the first two-digit number string for the in-store code in each row, the item code (10-digit number string), and the check digit (1-digit number). That is, each row in the registration information table corresponds to one product code (intaccord). In S306 described above, when a registered product is detected, a barcode representing this 13-digit product code associated with the product is generated.

In the registration information table of FIG. 8, the information in the column indicating whether or not the product has been registered (registered information) indicates whether or not the product has been registered in association with the product code of each row.

In the registered information table of FIG. 8, the information in the product name column is the product name of the product registered in association with the product code of each row.

In the registration information table of FIG. 8, the information in the recognition condition column is information necessary for detecting the product registered in association with the product code of each row by image recognition. As this information, information such as the information of the trained model and the feature amount of the product are recorded. Since this part contains a large amount of information, it is not limited to directly recording conditional data in the registration information table. information). In the image recognition processing of S304 in FIG. 3 described above, detection processing based on this recognition condition is performed for a plurality of products registered in the registration information table.

FIG. 4 shows a flowchart of product registration processing. This process is the detailed process of S310 in FIG. 3 described above. Each process in this flow chart is implemented by the CPU 101 developing a program stored in the nonvolatile memory 103 in the memory 102 and executing the program. This process is a process of registering products to be detected (more specifically, recognized in S304) in checkout processing. When the user adds a new product to the management target (detection target), the user uses this process to provide the electronic device 100 with recognition information (recognition conditions) for image recognition of the product, information for displaying the corresponding barcode, It is recorded in association with the product name. At the same time, the user registers information about a new product at the cash register 200 so that the product can be specified at the cash register 200 when the barcode displayed on the electronic device 100 is read.

In S<b>401 , the CPU 101 displays a product name input screen on the display 105 . FIG. 6A shows a display example of the product name input screen on the display 105. As shown in FIG. A text box 601 is an area for accepting input from the user of the name of a product to be newly registered.

In S402, the CPU 101 receives a product name input operation from the user. The user can specify a text box 601 by touching it and enter an arbitrary product name by entering characters from a software keyboard or the like. In the illustrated example, "crunchy curry bread" is input according to the user's operation. The input product name is held in the memory 102 .

In S403, the CPU 101 determines whether the next button 602 has been operated. If the next button 602 has been operated, the process proceeds to S404; otherwise, the process proceeds to S402.

In S404, the CPU 101 compares the input product name held in the memory 102 (the product name input in S402) with the registered product name registered in the registration information table, and determines the registered product name. Determine if it is the same as the first name (duplicate). If they overlap, the process proceeds to S405, and if they do not overlap, the process proceeds to S404.

In S405, the CPU 101 displays an error indicating that the product name is duplicated on the display 105, and displays the product name input screen again in S401.

In S410, the CPU 101 acquires an available in-store code number. Specifically, the CPU 101 refers to the registration information table, searches for a row with an empty registered column (that is, unregistered), and acquires one in-store code of the unregistered row. It is held in memory 102 . It should be noted that the number numbered in S410 is the in-store code number that starts with a specific two-digit number (29 in this embodiment) among the first two digits of 20 to 29 representing the in-store code. By doing so, an in-store code beginning with a specific two-digit number (29) can be used as a number for a product category (bread in this embodiment) registered with this application software. In-store code numbers starting with other two-digit numbers (eg, 28) can be used for classification, such as numbers for other categories (eg, seafood), facilitating product management.

In S411, the CPU 101 generates a barcode from the unregistered in-store code acquired in S410 and displays it on the display 105. FIG. FIG. 6B shows a display example on the display 105 at this time. The registration target number 611 is the in-store code obtained in S410. A barcode 612 is a displayed barcode image generated from the in-store code acquired in S410. The user reads the barcode 612 with the barcode reading unit 201 of the cash register 200 and confirms at the cash register 200 whether another product has already been registered for this in-store code. If it is found that the in-store code represented by the bar code 612 is a vacant number even at the cash register 200, it can be determined that this in-store code can be used as a target for new product registration. For example, if the in-store code represented by the bar code 612 has already been registered as the number of another product at the cash register 200, this in-store code is not used as a product to be newly registered, and another number is acquired. You can decide that you should. In this way, in order to display candidates for registration numbers as barcodes before photographing and learning registration target products, which will be described later, it is first confirmed whether the registration numbers are available in the register 200 as well. Then you can do further processing. Therefore, it is possible to reduce wasteful rework. The subsequent processing referred to here includes the processing of S434 to S438 according to the photographing instruction received in S432, which is one of the registration operations (details will be described later). In this embodiment, the input of the product name is accepted before the bar code is displayed in S411. You can do it.

In S412, the CPU 101 determines whether or not the button 614 has been touched (operated). For example, when the in-store code represented by the barcode 612 is read by the barcode reading unit 201 and it is found that the number is not a vacant number, the user selects a button, which is an operation item, to number another in-store code. Touch 614. If the button 614 has been touched, the process proceeds to 413; otherwise, the process proceeds to S414.

In S413, for the in-store code acquired in S410de and displayed as the registration target number 611, it is recorded that it has been registered in the registered column of the registration information table, and the process proceeds to S410 to obtain another free in-store code number. is obtained again. By doing so, the number that has been decided not to be used once is recorded as registered and is no longer an object to be obtained in S410, so that it will not be automatically presented from the next time onwards.

In S414, the CPU 101 determines whether or not the detail setting button 613 has been touched. A detailed setting button 613 is an operation item for displaying a screen for inputting a number to be registered by the user as an arbitrary number instead of the presented registration target number 611 . If the detailed setting button 613 has been touched, the process proceeds to S420; otherwise, the process proceeds to S415.

In S415, the CPU 101 determines whether the forward button 615 has been touched. If touched, proceed to S401; otherwise, proceed to S416.

In S416, the CPU 101 determines whether or not the Next button 616 has been touched. A next button 616 is an operation item for accepting an instruction to confirm the displayed registration target number 611 as a registration target number associated with a product to be registered from now on. If the next button 616 is touched, the process proceeds to S417; otherwise, the process proceeds to S412.

In S417, the CPU 101 determines the in-store code held in the memory 102 as the number to be registered. Specifically, it continues to be held as a confirmed number.

In S420, the CPU 101 displays a detailed setting screen on the display 105. FIG. FIG. 6C shows a display example of the detailed setting screen on the display 105. As shown in FIG.

FIG. 6(c) is a screen for the user to set the registration target number to an arbitrary value.

A registration target number 611 is a label displaying a registration target number held in the memory 102, and a bar code 612 is a bar code image converted from the registration target number and displayed. The input field 621 is a field for inputting a two-digit number string indicating an in-store code, and the user selects the input field 621 and inputs any number from 20 to 29 from the software keyboard or pull-down menu. be able to. The input field 622 is a field for inputting an item code, and the user can input any 10-digit numeric string by selecting the input field 621 and operating the software keyboard. When the numbers are input to the input fields 621 and 622, the CPU 101 searches the registration information table to check whether the in-store code including the combination has already been registered. If the number is recorded as registered ("registered" is recorded in the registered column), a message 623 is displayed to notify the user that this number has already been registered and cannot be used. The registration target number 624 is an in-store code composed of numbers entered in the input fields 621 and 622 . A bar code 625 displays the registration object number 624 as a bar code. When the button 626, which is an operation item, is touched, the in-store code including the numbers entered in the input fields 621 and 622 is discarded, the screen returns to the screen shown in FIG. Display the numbered screen. When the previous button 627, which is an operation item, is touched, the process returns to S401 and the screen of FIG. 6A is displayed. When the next button 628, which is an operation item, is touched, the process proceeds to step S417, where the in-store code, which is a combination of the number and check digit entered in the input fields 621 and 622, is determined as the number to be registered and held in the memory 102. . In a situation where an in-store code including a combination of numbers entered in the input fields 621 and 622 has already been registered and a message 623 is displayed, the next button 628 is displayed in an inactive state and does not accept touch operations. and Even if the next button 628 in the inactive state is touched, the process does not proceed to S417.

At S421, the CPU 101 accepts input of a two-digit number indicating an in-store code and an item code. That is, input operations to the input fields 621 and 622 are accepted. The input number is held in memory 102 .

In S422, the CPU 101 determines whether or not the in-store code including the number input in S421 is recorded as registered in the registration information table. If the number has been registered, the process proceeds to S423; otherwise, that is, if the number is vacant, the process proceeds to S424.

In S423, the CPU 101 displays a message 623 on the display 105 as an error display.

At S424, the CPU 101 displays the registration target number 624 and the bar code 625 on the display 105 based on the in-store code including the number input at S421.

In S425, the CPU 101 determines whether or not the Next button 628 has been touched. If the next button 628 has been touched, the process proceeds to S417; otherwise, the process proceeds to S421.

In S430, the CPU 101 initializes the variable n held in the memory 102 to one. The variable n is a variable for counting the number of learning images (the number of shots) of the product to be registered.

In S<b>431 , the CPU 101 displays on the display 105 a shooting screen for acquiring the n-th learning image of the product to be registered. A display example at this time is shown in FIG.

FIG. 6D is a display example of a photographing screen for photographing products to be registered. An image 631 is a live view image (LV image) captured by the camera unit 112 . A product 632 is a product to be registered that appears as a subject in the image 631 . A guide line 633 is an auxiliary line indicating the position where the product is placed. The user (worker 10) frames the electronic device 100 so that the product to be registered is captured at the position indicated by the guide line, or registers at the position indicated by the guide line 633 within the fixed angle of view of the fixedly installed electronic device 100. Place the item you want. In the display item 635, the denominator indicates the number of images required for product registration (the number of shots), and the numerator indicates the number of images to be obtained in the next shot (the number of shots taken next). showing. The illustrated example indicates that a total of 10 shots are required, and the current shot screen is for the first shot (the next shot is the first shot). A shooting button 634, a previous button 636, and a next button 637 are operation items that can be touch-operated by the user. The next button 637 becomes touchable when the number of shots required for product registration (10 shots in the example shown) is performed, and until then it is in an inactive state that does not respond to touch. Is displayed.

In S432, the CPU 101 determines whether or not the shooting button 634 has been touched. If the shooting button 634 is touched, the process proceeds to S434, otherwise the process proceeds to S433.

In S433, the CPU 101 determines whether or not the forward button 636 has been touched, and proceeds to S411 if touched, otherwise proceeds to S432.

In S434, the CPU 101 causes the camera unit 112 to perform actual photography. That is, shooting control is performed. It is assumed that the main shooting in the present embodiment is still image shooting, but it may be moving image shooting.

In S435, the CPU 101 uses the image processing unit 104 to perform learning processing based on the image shot in the actual shooting in S434. Specifically, the portion indicated by the guide line 633 is extracted (trimmed) from the photographed image, input to the learning model, and the learned model is generated as the learning result. Alternatively, the feature amount is extracted from an image obtained by extracting (trimming) the portion indicated by the guide line 633 from the photographed image. Then, the CPU 101 uses the generated trained model and the extracted feature amount to recognize the row corresponding to the in-store code of the determined registration target number held in the memory 102 in the registration information table described above with reference to FIG. The conditions are recorded in the nonvolatile memory 103 . The more this information is, the more accurate image recognition can be expected (recognition accuracy is improved), so learning is performed not only from one image but also from a plurality of images (10 images in this embodiment). By this recording control, the recognition condition as related information acquired in response to a photographing instruction, which is a type of registration operation, is recorded in association with the in-store code of the registration target number.

In S436, the CPU 101 determines whether or not the variable n is 1. If n=1, proceed to S437; otherwise, proceed to S438. It is determined whether or not the value of the number of shots at the time of product registration held in the memory 102 is 1. If it is 1, the process proceeds to S437, and if it is not 1, the process proceeds to S438.

In S437, the CPU 101 records the determined registration target number held in the memory 102 in the non-volatile memory 103 in association with the product name as registered. Specifically, information indicating "registered" is recorded in the registered column in the row corresponding to the in-store code of the fixed registration target number held in the memory 102 in the registration information table, and the product name is registered. In the column, the product name input in S402 held in the memory 102 is recorded. Thus, the product name, in-store code, and recognition condition of the new product are registered. Note that the product name and information indicating that the product has been registered are not recorded until the process of S435 is performed at least once. This is because if the processing is interrupted before the learning process has been performed even once after entering the product name and determining the registration target number, the product information is not actually registered because there are no recognition conditions. This is to prevent the in-store code from being registered even though it exists.

In S438, the CPU 101 determines whether or not the variable n is equal to Nmax. Nmax is the denominator displayed in the display item 635 and is a predetermined number as the number of images required for product registration. It is 10 in this embodiment. If n is not equal to Nmax, the process advances to S439 to increment the variable n by 1, and to S431 to perform the processing for the next photographing. If n=Nmax, proceed to S450.

In S450, the CPU 101 initializes the variable m held in the memory 102 to one. The variable m is a variable for counting the number of times to check whether the registered product is detected correctly.

In S451, the CPU 101 displays on the display 105 a confirmation screen for acquiring the m-th image for confirming whether the registered product is correctly recognized. A display example at this time is shown in FIG.

FIG. 7A is a display example of a confirmation screen for acquiring the m-th image for confirming whether the registered product is correctly recognized. An area 708 is a display area for indicating whether the recognition result is correct and the recognized product name. In the illustrated example, the recognition result and product name are not displayed because the confirmation of the first page has not yet been performed. An image 701 is an LV image captured by the camera unit 112 . A product 702 is a product (the same individual as the product photographed in S434 or another individual of the same item) that is shown as a subject in the image 701 and that is desired to be confirmed. A guide line 703 is an auxiliary line indicating the position where the product is to be placed. The user (worker 10) frames the electronic device 100 so that the product to be confirmed is captured at the position indicated by the guide line, or confirms at the position indicated by the guide line 703 within the fixed angle of view of the fixedly installed electronic device 100. Place the item you want. In the display item 705, the denominator indicates a predetermined number of images (the number of times of photographing) for confirming the product, and the numerator indicates the number of images for confirmation obtained in the next photographing (the number of images for confirmation obtained in the next photographing). The number of shots taken for confirmation) is shown. In the illustrated example, a total of five images are scheduled to be confirmed, and among them, it is indicated that the current image capturing screen is for the first image capturing (the next image capturing is the first image capturing). A check button 704, a previous button 706, and a next button 707 are operation items that can be touch-operated by the user.

In S452, the CPU 101 determines whether or not the forward button 706 has been touched, and if so, proceeds to S430, otherwise proceeds to S453.

In S453, the CPU 101 determines whether or not the check button 704 has been touched, and if so, proceeds to S454, otherwise returns to S452.

In S454, the CPU 101 inputs the LV image held in the memory 102 to the learned model stored in the nonvolatile memory 103 and held in the memory 102, and determines the type (item) of the product placed on the tray 30. infer. This process is the same as the process of S304 described above, and is performed using the recognition conditions registered in the learning process of S435. Also, an image used for inference (still image for one frame of the LV image when the check button 704 is touched) is held in the memory 102 .

In S456, the CPU 101 refers to the inference result of the product type held in the memory 102, and determines whether or not the product with the registration target number recorded in S437 has been detected among the plurality of registered products (objects). judge. If the product with the registration target number is detected, proceed to S461; otherwise, that is, none of the registered products is detected, or the registered product is detected but does not match the product with the registration target number. is detected, the process proceeds to S457.

In S457, the CPU 101 displays an NG screen on the display 105. FIG. A display example at this time is shown in FIG.

FIG. 7B is a display example of the NG screen on the display 105. FIG. The NG screen is a screen output when the product recognition result is different from the registered contents. Note that the same reference numerals as in FIG. 7A are given to the same display elements as in FIG. 7A. An area 708 displays "Recognition result: NG", indicating that the product recognition result is different from the registered content. In addition, "product name: super spicy curry bread" is displayed, and it can be seen that it has been detected as a product different from the "crunchy juicy curry bread" registered by the user. An image 711 is an image used for image recognition processing in S454, and is a still image of one frame of the LV image when the check button 704 is touched. A product 712 is a product (the same individual as the product photographed in S434 or another individual of the same item) shown as a subject in the image 711 and to be confirmed. Guidance 710 is a text notifying the user that recognition accuracy can be expected to be improved by using image 711 as an image with the correct registration target number (in this example, an image of crunchy juicy curry bread) for learning (feature extraction). is. Guidance 710 indicates that learning processing (feature amount extraction processing) using image 711 can be instructed by pressing (touching) an additional registration button 714 .

In S458, the CPU 101 determines whether or not the additional registration button 714 has been touched. If the additional registration button 714 has been touched, the process proceeds to S459; otherwise, the process proceeds to S460.

In S459, the CPU 101 selects the portion indicated by the guide line 703 from the still image for one frame of the LV image when the check button 704 held in the memory 102 is touched (that is, the image determined as NG). Extract (trimming) and input to the learning model. Thus, a learned model is generated. Alternatively, the feature amount is extracted from the image obtained by extracting (trimming) the portion indicated by the guide line 703 . Then, the CPU 101 uses the generated trained model and the extracted feature amount to recognize the row corresponding to the in-store code of the determined registration target number held in the memory 102 in the registration information table described above with reference to FIG. The conditions are recorded in the nonvolatile memory 103 . This process is the same process as S435 except for the input image.

In S460, the CPU 101 determines whether or not the forward button 706 has been touched, and proceeds to S451 if touched, otherwise proceeds to S458.

In S461, the CPU 101 displays an OK screen on the display 105 for a predetermined period of time (for example, 3 seconds) to indicate that it has been correctly recognized. On the OK screen, "Recognition result: OK" is displayed in an area 708, and the recognized product name ("Crunchy Juicy Curry Bread" in this example) is displayed.

In S462, the CPU 101 determines whether or not the variable m is equal to Mmax. Mmax is the denominator displayed in the display item 705, and is a predetermined number as the number of images (the number of shots) for confirming the product. It is 5 in this embodiment. If m is not equal to Mmax, the process advances to S463 to increment the variable m by one, and the process advances to S451 to perform processing for the next photographing. If m=Mmax, the process proceeds to S464.

In S464, the CPU 101 displays a registration completion screen on the display 464. FIG. A display example at this time is shown in FIG.

FIG. 7C shows a display example of the registration completion screen. The product name 721 is the product name registered in association with the registration target number, and is the product name input in S402. A product image 722 is an image of a registered product, and is one of the images captured in S434 and used for the learning process in S435. Although this product image 722 has been held in the memory 102, it may be recorded in the non-volatile memory 103 in association with the in-store code of the registration target number in the registration information table. A barcode 723 is a barcode having information on the registration number generated based on the registration number determined in S417. A forward button 724 and a complete button 725 are operation items. The user (worker 10 ) reads the barcode 723 displayed on the display 105 with the barcode reading unit 201 and registers the product with the same in-store code in the register 200 . For example, after performing an operation to newly register an in-store code at the cash register 200 , the bar code 723 is read and the in-store code to be registered in the cash register 200 is input to the cash register 200 . Since the numeric string of the in-store code itself is also displayed in the bar code 723, instead of reading the bar code with the bar-code reading unit 201, the numeric keys of the cash register are operated by looking at the displayed numeric string to read the in-store code to be registered. You can enter the code. Then, the operator 10 operates the register 200 to register various information such as product name, product category, and price in association with the input in-store code. By doing so, if the same bar code as the bar code 723 is read at the time of payment, the payment can be made at the register. As described above, according to the product registration process of FIG. 4, code information (in-store code) is registered in association with an object by a device (electronic device 100) separate from the device (register 200) that reads barcodes. When doing so, display the barcode to be registered. Therefore, it can be easily linked and registered in a device (cash register 200) that reads barcodes.

In S465, the CPU 101 determines whether the completion button 725 has been touched. If the completion button 725 has not been touched, the process waits in S465, and if the completion button 725 has been touched, the product registration processing in FIG. 4 is terminated.

According to the product registration process described above, the recognition conditions for detecting the target object (product) are learned in association with the code information (in-store code) to be registered, and the recognition conditions are recorded in association with the code information. can be done.

In addition to registering an object and code information in association with the electronic device 100, an external device (cash register 200) that reads the code information also associates information about the object (price, name, etc.) with the code information. must be attached and registered. On the other hand, in the present embodiment, the barcode or number string of the code information to be registered is displayed on the display 105 of the electronic device 100 at a suitable timing in the product registration process, so the registration work on the external device can be easily performed. It can be carried out. Even if the electronic device 100 and the cash register 200 cannot communicate with each other and cannot share the registration information table, according to the present embodiment, the registration work can be easily performed in conjunction with each device.

According to the present embodiment described above, it is possible to more easily manage objects using detection results of objects (for example, products) from captured images. In the above-described embodiment, the target object is a product that is paid at the store, but the target object is not limited to this. Even objects other than products can be detected from captured images. It is applicable if it can be managed by reading the code image with an existing system. For example, it can be applied to items to be delivered, items to be mailed in physical distribution, and raw materials, materials, and parts in factories.
Note that the various controls described above that are performed by the CPU 101 may be performed by one piece of hardware, or a plurality of pieces of hardware (for example, a plurality of processors or circuits) may share the processing to control the entire apparatus. may be performed.
Moreover, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention can be applied to the present invention. included. Furthermore, each embodiment described above merely shows one embodiment of the present invention, and it is also possible to combine each embodiment as appropriate.
Further, in the above-described embodiment, the case where the present invention is applied to the electronic device 100 has been described as an example, but the present invention is not limited to this example, and can be applied to any device capable of acquiring a captured image. . That is, the present invention can be applied to personal computers, PDAs, mobile phone terminals, portable image viewers, game machines, electronic book readers, and the like.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, the software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads the program code. This is the process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

100... Electronic device 200... Cash register

Claims (18)

display control means for controlling display of code information to be registered;
a photographing control means for controlling to perform photographing processing of an object corresponding to the code information;
a control means for controlling to perform a learning process of the object using the image captured in the imaging process;
An electronic device, comprising: recording control means for controlling such that the learning result of the learning process and the code information to be registered are recorded in association with each other. display control means for controlling display of code information to be registered;
a photographing control means for controlling to perform photographing processing of an object corresponding to the code information to be registered;
Control means for controlling to perform a predetermined process of acquiring recognition information for image recognition of the object using the image captured in the image capturing process;
An electronic device, comprising: recording control means for performing control such that the recognition information obtained by the predetermined processing and the code information to be registered are recorded in association with each other. display control means for controlling display of code information to be registered;
receiving means for receiving a registration operation of information related to an object corresponding to the code information to be registered after displaying the code information to be registered;
and recording control means for controlling the code information to be registered and the related information to be recorded in association with each other. 4. The electronic device according to claim 1, wherein the code information is a code image of a one-dimensional code or a two-dimensional code. 5. The electronic device according to claim 4, wherein the code information is a bar code. 6. The electronic device according to any one of claims 1 to 5, wherein the code information is code information that can be arbitrarily numbered by at least one of a store, an organization, and a business. 7. The electronic device according to any one of claims 1 to 6, wherein the code information is an in-store code. 8. The method according to any one of claims 1 to 7, further comprising acquiring means for acquiring code information in which information relating to an object is not registered as the code information to be registered displayed by the display control means. The electronic device described in . The acquisition means is characterized in that, in accordance with a user's instruction, code information different from the code information displayed by the display control means and in which information related to an object is not registered, is acquired again. The electronic device according to claim 8. When the code information is acquired again by the acquisition means, the recording control means provides information to the effect that the code information displayed by the display control means has already been registered without being associated with information relating to an object. 10. The electronic device according to claim 9, wherein the electronic device controls to record the . 11. The electronic device according to any one of claims 1 to 10, further comprising reception means for receiving input of arbitrary code information as the code information to be registered. 12. The electronic device according to claim 1, wherein the display control means controls to display the code information to be registered after recording by the recording control means. 13. The code information to be registered displayed by the display control means can be used for registration in the external device by being read by a reading unit of the external device. 1. The electronic device according to 1. 14. The electronic device according to claim 13, wherein the external device is a cash register. a display control step for controlling display of code information to be registered;
a photographing control step of controlling to perform photographing processing of an object corresponding to the code information;
a control step of controlling to perform a learning process of the object using the image captured in the imaging process;
A control method for an electronic device, comprising: a recording control step of performing control so as to record the learning result of the learning process and the code information to be registered in association with each other. a display control step for controlling display of code information to be registered;
a photographing control step of controlling to perform photographing processing of an object corresponding to the code information to be registered;
a control step of controlling to perform a predetermined process of acquiring recognition information for image recognition of the object using the image captured in the image capturing process;
A control method for an electronic device, comprising: a recording control step of performing control such that the recognition information acquired by the predetermined processing and the code information to be registered are recorded in association with each other. a display control step for controlling display of code information to be registered;
a receiving step of receiving a registration operation of information related to an object corresponding to the code information to be registered after displaying the code information to be registered;
A control method for an electronic device, comprising: a recording control step of performing control such that the code information to be registered and the related information are recorded in association with each other. A program for causing a computer to function as each means of the electronic device according to any one of claims 1 to 14.

Images