TWI485627B - Scanning and identifying method for physical object and application program for the same - Google Patents

Description

Scanning identification method for physical objects and its application program

The invention relates to an image recognition method and an application program thereof, and more particularly to a method and an application program for instantly recognizing an object without performing a photographing action on a physical object.

With the advancement of technology, users are more and more convenient to inquire about items. For example, by scanning bar codes and QR codes, users no longer need to manually check items when they want to query information about items. The relevant text information is entered into the computer for inquiry, which is quite convenient.

In recent years, smart mobile devices, such as smart phones or tablets, have been flooded with the general public. Almost all of today's smart mobile devices are equipped with a camera lens that allows the mobile device to take photos of external images. Therefore, when the user installs the application for identifying the QR code in the mobile device, the camera lens on the mobile device can be used to scan the surrounding QR code, and the corresponding item information is obtained after the application is recognized.

The QR code is a box-shaped totem, and a square pattern of "back" is set in the upper left corner, the upper right corner, and the lower left corner of the box. The QR code can assist the positioning of the application and the camera through the above three square patterns, when the user uses the mobile device When the upper camera lens is scanned, no alignment is required, and the QR code can be read correctly regardless of scanning at any angle.

However, the above QR code needs to be made according to a standard format. When an item-related information is not made into a QR code, the user cannot read and recognize in the above manner. If the user wants to inquire about the above items, he still needs to manually input the text information related to the item in order to inquire about the details of the item. As a result, the application and the camera lens are virtually useless, which is a pity.

In view of this, some people have proposed a technique for image recognition in the market, which is to take a photo of a user's desired item and directly identify the picture. When the item in the photo is to be recognized, the server automatically queries the item for detailed information and returns the query result to the user. The advantage of this technology is that as long as the item to be inquired can be photographed, the user can identify and query any item through the photo without having to find the QR code.

As mentioned above, the bar code and the QR code have a standard format, so the user only needs to take the reader to scan, and the reader can automatically locate and identify. However, the above picture recognition technology requires the user to manually take a photo of the item, which cannot be directly obtained by scanning, mainly because the user is subject to any physical item, and the physical items are not standard. Format, so the application cannot automatically determine what the object to identify.

Cover, the above-mentioned technology of the camera action is not humanized, in addition, the completed photo will be stored in the mobile device, wasting limited storage space in the mobile device. After the identification is completed, if the user does not need the photo, it is also necessary to manually delete the photo in the photo album of the mobile device, which is quite troublesome.

Therefore, the market should propose a novel technology that allows users to directly identify physical objects by scanning barcodes and QR codes without the need to manually perform photographing actions. Image recognition technology is more user-friendly.

The main object of the present invention is to provide a method for scanning and identifying a physical object and an application program thereof, which can determine whether a physical object of interest to a user exists in a frame scanned by the image capturing unit, thereby triggering without taking a picture. Image recognition action.

In order to achieve the above objective, the present invention firstly scans the frame of the external image through the image capturing unit on the mobile device, and compares the scanned complex frames to determine whether the mobile device is in a moving state or a stagnant state. When the mobile device is in a stagnant state and meets the initiation condition of an identification action, the mobile device determines that there is a physical object that the user wants to identify in the current frame, so the current frame is transmitted to a backend server for image recognition. The identification result is then received by the server and displayed.

The utility of the present invention in comparison with the prior art is to directly compare and analyze the complex frames scanned by the image capturing unit to determine whether there is a physical object of interest to the user in the current frame, so that the user does not need to manually Take a picture of the physical object of interest. Moreover, when it is determined that there is a physical object of interest to the user in the current frame, the method of the present invention automatically transmits the frame to the backend server for image recognition, so the user does not need to manually upload the entity. A photo of the object.

With the technique of the present invention, the user does not need to manually take a photo action, so the action The photo of the physical object is not stored in the device, and after the identification is completed, the user does not need to manually delete the photo for identification. Therefore, not only can the storage space of the mobile device be effectively saved, but also the user can find a more convenient, faster and more intuitive identification service.

1‧‧‧Electronic device

11‧‧‧Image capture unit

12‧‧‧Display unit

13‧‧‧Microcontroller

14‧‧‧Transportation unit

15‧‧‧Input unit

16‧‧‧ storage unit

161‧‧‧Application

2, 21, 22, 23, 24‧‧‧ objects

3‧‧‧Server

4‧‧‧Database

S10~S20, S140~S148, S30~S38‧‧‧ steps

F1, F2, F3, F4‧‧‧ frames

I1‧‧‧Photo Information

R1‧‧‧ identification results

R2‧‧‧ query results

The first figure is a schematic view of the use of a preferred embodiment of the present invention.

Figure 2A is a first schematic view of a preferred embodiment of the present invention.

FIG. B is a second schematic diagram of a second embodiment of the present invention.

FIG. C is a third schematic view of a preferred embodiment of the present invention.

FIG. D is a fourth schematic view of a preferred embodiment of the present invention.

The third figure is a scanning flow chart of a preferred embodiment of the present invention.

The fourth figure is a flow chart of state determination according to a preferred embodiment of the present invention.

The fifth figure is a system architecture diagram of a preferred embodiment of the present invention.

Figure 6 is a flow chart of a query in accordance with a preferred embodiment of the present invention.

FIG. 7 is a schematic diagram of a first query result according to a preferred embodiment of the present invention.

FIG. 7B is a schematic diagram of a second query result according to a preferred embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION A preferred embodiment of the present invention will be described in detail with reference to the drawings.

Referring first to the first figure, a schematic diagram of the use of a preferred embodiment of the present invention is shown. The present invention is an identification method for scanning a frame of an external image by the mobile device 1 and automatically recognizing the physical objects 21 to 24 in the frame.

As shown in the first figure, the mobile device 1 of the present invention is mainly a smart mobile device, such as a smart phone or a tablet computer. In this embodiment, a smart phone is taken as an example, but not limited. The mobile device 1 has an image capturing unit 11, such as a camera lens. When the image capturing unit 11 is turned on, the external image can be scanned to continuously obtain a frame of the external image. Moreover, the frames are displayed on a display unit 12 on the mobile device 1 for viewing by the user. As shown in the first figure, when the image capturing unit 11 is aligned with a physical object 24, the physical object 24 will be included in the frame displayed on the display unit 12.

Continuing to refer to FIG. 2A to FIG. D, respectively, which are schematic diagrams of first to fourth operations of a preferred embodiment of the present invention. When scanning any of the physical objects 21 to 24, since it does not have a specific format such as a bar code or a QR code, the mobile device 1 cannot know what kind of frame is actually required by the user. In the method of the present invention, the continuous frame (the closest two) is subjected to the comparative analysis, thereby obtaining the displacement amount of the mobile device 1, and then the magnitude of the displacement amount determines whether the mobile device 1 is in a moving state or stagnated. status.

More specifically, if the mobile device 1 is in a moving state, the method can determine that the user is still searching for the physical object 2 to be identified; if the mobile device 1 is in a stagnant state, the method can determine that the image is currently located. The physical object 2 in front of the unit 11 is taken as the physical object 2 that the user wants to identify. In this case, the mobile device 1 automatically transmits the current frame (the frame displayed on the display unit 12) to the back end to identify the physical object contained in the frame.

First, as shown in FIG. 2A and FIG. 2B, the continuous frame F1 and F2 scanned by the image capturing unit 11 may have a difference. If the difference is greater than a threshold, the mobile device 1 may be determined. It has a certain amount of displacement and is in a moving state. In this case Next, the method determines that the physical objects 2 that the user wants to identify are not present in the frames F1 and F2, so the recognition action is not triggered.

As shown in the second figure C and the second figure D, if the continuous frames F3 and F4 scanned by the image capturing unit 11 are very similar, that is, when the difference between the frames F3 and F4 is less than the threshold value, It can be determined that the mobile device 1 has locked the target and is thus in a stagnant state. In this case, the method determines that there is a physical object 24 that the user wants to recognize in the frame F4 currently displayed on the display unit 12, so the recognition action will be triggered.

Referring to the third figure, a scanning flow chart of a preferred embodiment of the present invention is shown. As shown in the figure, to perform the method of the present invention, the image capturing unit 11 on the mobile device 1 is first turned on (step S10), and then the image capturing unit 11 continuously scans the image of the external image (steps). S12). Next, the mobile device 1 analyzes the scanned complex video frame by an internal application (such as the application 161 shown in the fifth figure) to determine whether the mobile device 1 is in a stagnant state (step S14). That is, it is determined whether or not the mobile device 1 is moving. It is worth mentioning that the image capturing unit 11 can be automatically turned on after the application 161 is executed, but is not limited.

After the step S14, if the application 161 determines that the mobile device 1 is in the moving state, the process returns to the step S12, and the image capturing unit 11 continues to scan the frame of the external image, and continuously determines whether the mobile device 1 is in the mobile device. Stalled state. If the application 161 determines that the mobile device 1 is in a stagnant state, it is further determined whether the stagnant state conforms to an initiation condition of an identification action (step S16).

After the step S16, if the stagnant state does not meet the initiation condition, return to the step In step S12, the image capturing unit 11 continues to scan the frame of the external image, and the application 161 determines whether the mobile device 1 continues to be in a stagnant state. If the stagnant state meets the initiation condition, the application 161 controls the mobile device 1 to perform image processing on the current frame (ie, the last frame displayed on the display unit 12) (step S18), and after processing The frame information (such as the frame information I1 in the fifth figure) is transmitted to a server at the back end (such as the server 3 shown in the fifth figure) for performing the recognition operation (step S20). It is worth mentioning that the frame includes at least one physical object 2 that the user wants to identify. The main operation of the server 3 is to receive the frame information I1 and identify the content included in the frame information I1. What is the physical object 2?

The step S18 is mainly to perform image processing on the frame, and reduce the capacity of the frame or change the data format of the frame to reduce the occupation of the frame by the mobile device 1 when the frame is transmitted to the server 3. The network bandwidth, which in turn increases the transmission speed. More specifically, the mobile device 1 can mainly perform a compression operation for the current frame, and in step S20, the compressed image is transmitted to the server 3 as the frame information I1. Furthermore, the mobile device 1 can also perform an encoding operation on the current frame, and in step S20, the text information generated after the encoding is transmitted to the server 3 as the frame information I1. After the server 3 receives the frame information I1, the character in the frame information I1 is decoded according to the corresponding algorithm to restore the text to the frame scanned by the image capturing unit 11. In this way, the size of the data can be effectively reduced to improve the transmission efficiency.

In the present invention, the user can only recognize one physical object 2 at a time, so that when a frame is selected, the image capturing unit 11 continuously scans the external image during image processing, transmission, and recognition of the frame. Frame, but the app 161 will not compare the frames.

In the method of the present invention, the application 161 determines whether the physical object 2 of interest to the user exists in the current frame based on whether the mobile device 1 is in a stagnant state. In the above step S14, the application 161 can mainly determine whether the mobile device 1 is in a stagnant state according to the flow shown in the fourth figure.

Please refer to the fourth figure, which is a flow chart of state determination according to a preferred embodiment of the present invention. In the above step S14, the application 161 mainly compares two consecutive frames (step S140), and takes out the difference portion of the two frames. Next, the displacement amount of the mobile device 1 is analyzed based on the difference portion of the two frames (step S142). Then, after analyzing the displacement amount of the mobile device 1, it is determined whether the displacement amount of the mobile device 1 is less than a predetermined threshold value (step S144).

As described above, if the displacement amount of the mobile device 1 is greater than the threshold value (ie, not less than the threshold value), the application 161 determines that the mobile device 1 is in a moving state (step S146), so the application 161 does not Perform any action. In this case, the application 161 returns to the above step S12, continuously scans the frame of the external image, and reanalyzes the displacement of the mobile device 1.

Furthermore, if the displacement amount of the mobile device 1 is less than the threshold value, the application 161 determines that the mobile device 1 is in a stagnant state (step S148). In this case, the application 161 will perform the step S16 described above to further determine whether the stagnant state meets the initiation condition of the identification action.

More specifically, the application 161 can continuously perform a comparison analysis on successive frames to determine whether the mobile device 1 is continuously in a stagnant state. In an embodiment, when the displacement of the mobile device 1 is less than the threshold value And reached a certain number of times At this time, the application 161 determines that the initiation condition of the identification action is met. For example, the application 161 compares a continuous first frame with a second frame to analyze the displacement of the mobile device 1. If the displacement is less than the threshold (first time), the action is determined. The device 1 is in a stagnant state. Next, the application 161 compares the second frame with the subsequent third frame to continuously analyze the displacement of the mobile device 1. If the displacement is still less than the threshold (second time), it is determined The mobile device 1 is still in a stagnant state. In this way, when the displacement of the mobile device 1 continues to be less than the threshold and reaches a certain number of times, the application 161 can determine that the stagnant state conforms to the initiation condition.

Furthermore, the different mobile device 1 and the image capturing unit 11 have different performances. Therefore, the number of frames that the image capturing unit 11 can scan per second is not fixed. In another embodiment, the application 161 may also determine that the stagnant state conforms to the initiation condition when the mobile device 1 is in a stagnant state and continues for a certain period of time.

For example, a first image capturing unit can scan ten frames per second, so that the amount of displacement of the mobile device can be analyzed nine times per second (first frame and second frame, second frame and third) The frame comparison, ..., the twenty-third frame and the twenty-fourth frame comparison). That is to say, whether the displacement of the mobile device is less than the threshold number of thresholds can be performed nine times per second; if a second image capturing unit can scan twenty-four frames per second, the mobile device can be operated every second. The amount of displacement is analyzed twenty-three times, so whether the amount of displacement is less than the number of judgments of the threshold value can be performed up to twenty-three times per second. As described above, the user can adjust the setting value of the application 161 to determine whether the "the number of times the displacement of the mobile device 1 is less than the threshold" is the initiation condition, or "the mobile device 1 continues to be stagnant. The time of the state" as the initiation condition should not be limited.

Continuing to refer to the fifth figure, a system architecture diagram of a preferred embodiment of the present invention is shown. As shown in the figure, the mobile device 1 mainly includes the image capturing unit 11, the display unit 12, a microcontroller 13, a transmission unit 14, an input unit 15, and a storage unit 16, wherein the microcontroller 13 The image capturing unit 11, the display unit 12, the transmitting unit 14, the input unit 15, and the storage unit 16 are electrically connected.

The microcontroller 13 integrates and controls all of the units 11, 12, 14, 15, 16 in the mobile device 1. The application 161 is stored in the storage unit 16, and the user can trigger execution of the application 161 by the manipulation of the input unit 15, and the identification method of the present invention is executed by the application 161. After the application 161 is executed, the image capturing unit 11 can be controlled to be turned on, the image capturing unit 11 scans the frame of the external image, and the scanned frame is displayed on the display unit 12.

When the user finds the physical object 2 to be recognized, the image capturing unit 11 can be aligned with the physical object 2, and the mobile device 1 is placed in a stagnant state. When the application 161 judges that the mobile device 1 is in a stagnant state by the scanned complex video frame and conforms to the initiation condition, the mobile device 1 is controlled to capture the video frame including the physical object 2. Then, the mobile device 1 can perform image processing on the frame by the microcontroller 13, and the processed frame information I1 will be transmitted to the server 3 at the back end through the transmission unit 14 to perform an identification operation.

The server 3 receives the frame information I1 transmitted by the mobile device 1 and decodes the frame information I1 to perform image recognition to obtain the physical object 2 included in the frame information I1, and generates A recognition result R1 is transmitted back to the mobile device 1.

Please refer to the sixth figure at the same time, which is a query flow chart of a preferred embodiment of the present invention. After the identification of the server 3 is completed, the mobile device 1 receives the identification result R1 replied by the server 3 through the transmission unit 14 (step S30), and determines whether the identification result R1 is a valid result (step S32). More specifically, if the server 3 can successfully recognize the physical object 2 included in the frame information I1, the server 3 generates a valid identification result R1 and replies to the mobile device 1; If the server 3 cannot recognize the physical object 2 included in the frame information I1, an invalid identification result R1 is generated, for example, a text message indicating "identification failure" or "unrecognizable" is returned, and the action is replied to the action. Device 1.

If it is determined in the step S32 that the identification result R1 is a valid result, the mobile device 1 can display the identification result R1 on the display unit 12, wherein the identification result R1 can only include the physical object. A brief description of 2. At this time, the user can operate the input unit 15 to issue a request for the server 3 to inquire the detailed object information of the physical object 2 (step S34). After receiving the request, the server 3 can send a corresponding data. The library 4 performs a query to obtain detailed information about the physical object 2.

After the query by the server 3 is completed, the mobile device 1 receives a query result R2 replied by the server 3 through the transmission unit 14 (step S36), and displays the query result R2 on the display unit 12. The item information about the physical object 2 (step S38).

Please refer to FIG. 7A and FIG. 7B respectively, which are schematic diagrams of a first query result and a second query result according to a preferred embodiment of the present invention. As shown in FIG. 7A, if the identification result R1 is a valid result, the object type and name of the physical object 2 can be simply described in the identification result R1. The mobile device 1 receives the identification result After the R1, the identification result R1 can be displayed on the display unit 12. For example, the display unit 12 displays the physical object 2 as a book and the book name according to the identification result R1. However, the above is only a preferred embodiment of the present invention and should not be taken as an example.

When the user sends a request for querying the object information of the physical object 2 to the server 3, the query result R2 can be received by the server 3 and displayed on the display unit 12. For example, as shown in FIG. 7B, the display unit 12 can display the object type of the physical object 2 as a book according to the query result R2, and display the name, author, publisher, pricing, and Details such as the purchase of links, but not limited.

In the mobile device 1 of the present invention, the application program 161 is mainly installed, and when the mobile device 1 is loaded into the application program 161 and executed, the scanning identification method of the physical object of the present invention can be executed by the application program 161. . In this embodiment, the method for scanning and identifying the physical object mainly includes the following steps: a) turning on the image capturing unit 11; b) scanning the external image by the image capturing unit 11; c) The plurality of frames determine whether the mobile device 1 is in a stagnant state; d) if the mobile device 1 is in a stagnant state, further determining whether the stagnant state meets an initiation condition of an identification action; e) if the stagnant state meets the initiation condition, Image processing, wherein the frame contains at least one image of a physical object; f) transmitting the processed frame information to the backend server 3 for image recognition; g) receiving the backend server 3 backhaul The identification result R1; and, h) displays the identification result R1 on the display unit 12.

The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention, so equivalent changes are all included in the application of the present invention. Within the scope of the present invention, it is combined with Chen Ming.

S10~S20‧‧‧Steps

Claims (8)

A method for scanning and identifying a physical object is applied to a mobile device, the mobile device having an image capturing unit, wherein the scanning and identifying method of the physical object comprises: a) turning on the image capturing unit; b) capturing the image through the image The unit scans the external image frame; c) judges whether the mobile device is in a stagnant state according to the complex video frame obtained by scanning; d) according to step c, if the mobile device is in a stagnant state, further determining whether the stagnant state conforms to an identification action Initiating condition; e) taking step d, if the stagnant state meets the initiating condition, performing image processing on the current frame to reduce the network bandwidth occupied by the mobile device when transmitting the frame, wherein the frame includes at least one An image of the physical object, the image processing is to compress the current image frame or perform image encoding operation; f) transmitting the processed image information to the back end server for image recognition, wherein the image information is the image frame a compressed image, or text information generated after the image is encoded; and g) receiving the backend server backhaul Identification results. The scanning identification method of the physical object according to claim 1, wherein the step c further comprises the steps of: c1) analyzing the displacement amount of the mobile device according to the complex video frame obtained by scanning; c2) determining whether the displacement of the mobile device is Less than one threshold; and c3) If the displacement of the mobile device is less than the threshold, the mobile device is determined to be in a stagnant state. The scanning identification method of the physical object according to claim 2, wherein in the step c1, two consecutive frames are compared to analyze the displacement amount of the mobile device by the image difference portion of the two frames. The scanning identification method of the physical object according to claim 2, wherein in the step e, when the displacement amount of the mobile device is determined to be less than the threshold value for a certain number of times, it is determined that the initiation condition is met. The method for scanning and identifying a physical object according to claim 2, wherein in the step e, when the mobile device is in a stagnant state for a certain time, it is determined to meet the initiation condition. The scanning identification method of the physical object according to claim 1, wherein the step g further comprises the steps of: g1) receiving an identification result of the physical object included in the frame by the backend server; and g2) determining the identification result. Whether it is a valid result; and g3) if the identification result is a valid result, a display unit on the mobile device displays the identification result. The method for scanning the physical object according to claim 6, wherein if the identification result is a valid result, the method further comprises the following steps: h) requesting the backend server to query the object information of the physical object; i) Receiving object information of the physical object returned by the backend server; and j) displaying the object information in the display unit. An application program (AP) stored in the mobile device, when the mobile device is loaded into the application and executed, performing a scanning and identifying method of a physical object, wherein the mobile device has an image capturing unit And a display unit, the scanning identification method of the physical object includes: a) turning on the image capturing unit; b) scanning the external image by the image capturing unit; c) judging whether the mobile device is in a stagnant state according to the scanned complex video frame; d) if the mobile device is in a stagnant state a state, further determining whether the stagnation state meets an initiation condition of an identification action; e) taking step d, if the stagnation state conforms to the initiation condition, performing image processing on the current frame to reduce the occupation required when the mobile device transmits the frame The network bandwidth, wherein the frame contains at least one image of a physical object, the image processing is to compress the current frame or perform image coding; f) transmit the processed frame information to the backend server Performing image recognition, wherein the frame information is the compressed image of the frame, or the text information generated after the frame is encoded; g) receiving the identification result returned by the backend server; and h) in the display unit The identification result is displayed.