KR20230171859A - Object identification apparatus and 3d image generation method using the same - Google Patents

Abstract

The present invention relates to an object identification device, comprising: at least one camera module that rotates or moves through a support module and photographs objects in the base module from various positions or angles; And the camera module generates a three-dimensional image of the object based on images taken from various positions or angles of the object, and uses the three-dimensional image of the object to generate a plurality of virtual 2D images of the object or to Characterized in that it includes a processor that identifies.

Description

Object identification device and three-dimensional image generation method using the same {OBJECT IDENTIFICATION APPARATUS AND 3D IMAGE GENERATION METHOD USING THE SAME}

The present invention relates to an object identification device and a method for generating a three-dimensional image using the same. More specifically, an object identification device capable of generating a three-dimensional image using a rotating or moving camera in a system that captures and identifies objects, and the same. This relates to a method for generating three-dimensional images.

Recently, unmanned payment counters have been installed in many businesses, and the number of stores that operate only with unmanned payment systems without managers is also increasing.

These unmanned payment systems were mainly used in the past where users had to recognize the barcodes of objects (e.g. products) one by one through a barcode reader, but recently, single or multiple objects (e.g. products) placed on a product stand have been used. There is a shift to a method of recognizing (identifying) objects at once by photographing them.

However, there is a problem in that the unmanned payment system's low identification power makes it unable to accurately identify objects placed on the product stand, which may cause errors in the amount to be paid, causing losses to the store or user.

Therefore, in order to prevent these problems, there is a need for technology that can improve the identification of objects.

The background technology of the present invention is disclosed in Korean Patent Publication No. 10-2023-0015618 (January 31, 2023).

The present invention is intended to solve the above problems, and provides an object identification device that allows a stereoscopic image to be generated using a rotating or moving camera in a system for photographing and identifying objects and a method for generating a stereoscopic image using the same. There is a purpose.

In one aspect of the present invention, an object identification device includes: at least one camera module that rotates or moves through a support module and photographs objects in the base module from various positions or angles; And the camera module generates a three-dimensional image of the object based on images taken from various positions or angles of the object, and generates a plurality of virtual 2D images of the object using the three-dimensional image of the object, and the object It is characterized by including a processor that performs learning to identify objects using a plurality of virtual 2D images.

In one aspect of the present invention, a method of generating a three-dimensional image using an object identification device includes the steps of: a processor photographing objects in a base module from various positions or angles using at least one camera module that rotates or moves through a support module; generating, by the processor, a three-dimensional image of the object based on images taken by the camera module from various positions or angles of the object; generating, by the processor, a plurality of virtual 2D images of the object using the stereoscopic image of the object; and a step of the processor performing learning to identify the object using a plurality of virtual 2D images of the object.

The present invention allows objects to be photographed from multiple angles or in three dimensions even when using a single camera by rotating or moving the camera.

In addition, the present invention allows a three-dimensional image to be generated using a rotating or moving camera in a system that photographs and identifies objects.

1 is an exemplary diagram showing the schematic configuration of an object identification device according to an embodiment of the present invention.
Figure 2 is a flowchart for explaining the learning mode operation of the object identification device according to an embodiment of the present invention.
Figure 3 is a flowchart for explaining the identification mode operation of the object identification device according to an embodiment of the present invention.
Figure 4 is an example diagram for explaining a method of rotating a camera module according to an embodiment of the present invention.
Figure 5 is an example diagram showing the rotation range of the camera module in Figure 4.
Figure 6 is an example diagram for explaining a method of rotating the camera module using a plurality of support modules in Figure 4.
Figure 7 is an example diagram for explaining a method of moving a camera module according to an embodiment of the present invention.
Figure 8 is an example diagram for explaining a method of moving a plurality of camera modules in Figure 7.
Figure 9 is an exemplary diagram showing the schematic form of an object identification device according to an embodiment of the present invention.
Figure 10 is an example diagram showing a photograph taken during a test process of an object identification device according to an embodiment of the present invention.
FIG. 11 is an example diagram showing a reference pattern formed on the base module and the inner side of the body or frame in FIG. 1.
Figure 12 is an example diagram for explaining a method of utilizing a reference pattern for object recognition according to the first embodiment of the present invention.
Figure 13 is an example diagram for explaining a method of utilizing a reference pattern for object recognition according to the second embodiment of the present invention.
Figure 14 is an example diagram for explaining a method of using a reference pattern for object recognition according to the third embodiment of the present invention.
Figure 15 is an example diagram for explaining a method of using a reference pattern for object recognition according to the fourth embodiment of the present invention.
Figure 16 is an example diagram for explaining a method of utilizing a reference pattern for object recognition according to the fifth embodiment of the present invention.
Figure 17 is an example diagram showing some of the images taken in all directions of 360 degrees of the same object based on a reference pattern in Figure 16.
FIG. 18 is an example diagram for comparing three-dimensional images generated when photographing the same object based on a reference pattern and when photographing the same object without using a reference pattern in FIG. 16 .
Figure 19 is an example diagram for explaining a method of using a reference pattern for object recognition according to the sixth embodiment of the present invention.
Figure 20 is an example diagram to explain a method of generating a three-dimensional image using a rotating or moving camera module instead of a stereo camera in the present invention.
FIG. 21 is an example diagram for explaining the method of generating a three-dimensional image using an image captured using structured light in FIG. 20.
Figure 22 is an example diagram to explain accuracy when measuring depth information using a rotating or moving camera in the present invention.
Figure 23 is an example diagram showing images taken of various objects with different depth information using a rotating or moving camera in the present invention.
FIG. 24 is an example diagram showing 2D and depth information that can classify characteristics of each shape of an object in FIG. 23.
Figure 25 is an example diagram shown to explain the problems of the existing method of generating a three-dimensional image based on an image taken of an object placed on a rotating turntable using a fixed camera.
Figure 26 is an example diagram showing a three-dimensional image generated based on an image of an object captured while rotating or moving the camera module at various angles and directions according to an embodiment of the present invention.
Figure 27 is an example diagram showing a photograph testing the photographing operation of the camera module of the object identification device according to an embodiment of the present invention.
Figure 28 is an example diagram for explaining a method of training an object identification device according to an embodiment of the present invention.
Figure 29 is an example diagram to explain a method of improving the identification time and identification performance of an object by sharing coordinate information of the object captured while the camera module rotates or moves along the support module according to an embodiment of the present invention.

Hereinafter, embodiments of an object identification device according to the present invention and a method for generating a three-dimensional image using the same will be described with reference to the attached drawings.

In this process, the thickness of lines or sizes of components shown in the drawing may be exaggerated for clarity and convenience of explanation. In addition, the terms described below are terms defined in consideration of functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, definitions of these terms should be made based on the content throughout this specification.

An object identification device that uses a camera to identify objects must photograph objects from various directions and learn from the captured images in order to improve identification, but problems may arise in which objects cannot be accurately identified depending on lighting or shooting angles. there is. Therefore, in order to solve this problem, a plurality of cameras capable of photographing objects from various angles can be fixedly installed. However, in this case, as the number of cameras increases, a problem may occur in which the manufacturing cost of the object identification device increases.

Accordingly, the present invention allows three-dimensional photography of objects from various angles even when using a single camera by rotating or moving the camera, thereby lowering the manufacturing cost of the object identification device and improving identification ability.

1 is an exemplary diagram showing the schematic configuration of an object identification device according to an embodiment of the present invention.

The object identification device according to this embodiment shown in FIG. 1 may include a camera module 120 and a processor 150. Alternatively, as shown in FIG. 1, the object identification device according to an embodiment of the present invention includes a support module 110, a camera module 120, a base module 130, a sensor module 140, a processor 150, It may include a communication module 160, a lighting module 170, and a database (DB).

The support module 110 is formed at a specified angle on one inner side of the body (or frame) of the object identification device, and the camera module 120 can be attached to it.

The support module 110 is formed with a track of a designated shape (e.g., circular, square, oval, C-shaped, U-shaped, etc.) corresponding to the rotation direction or moving direction of the camera module 120, or a track of a designated shape is attached. You can.

At least one camera module 120 may be attached to the support module 110.

The support module 110 can be rotated or moved in a specified direction at a specified speed while the camera module 120 is attached.

The support module 110 may include an electric motor (not shown).

A plurality of support modules 110 may be arranged at multiple heights based on the base module 130 in order to photograph objects placed on the base module 130 from various heights. When the support modules 110 are configured at multiple heights, a camera module 120 may be provided in each support module 110 correspondingly.

The camera module 120 may be fixedly attached to the support module 110 and may actively rotate or move along the orbit of the support module 110.

The camera module 120 may be equipped with an electric motor (not shown) to actively rotate or move along the trajectory of the support module 110.

The camera module 120 may be fixedly attached to the support module 110 and passively rotated or moved according to the rotation or movement of the support module 110.

The camera module 120 rotates or moves and can continuously photograph objects (eg, products, articles, etc.) placed on the base module 130 from various angles.

The camera module 120 may have its shooting angle adjusted inward or outward based on the center of the base module 130.

The camera module 120 may be arranged at multiple heights based on the base module 130 in order to photograph objects placed on the base module 130 from various heights.

The camera module 120 includes a 3D camera such as a 2D camera, a depth camera, and a stereo camera. The camera module 120 is capable of acquiring two-dimensional images or three-dimensional images including depth information, and rotates or moves to capture two-dimensional images or three-dimensional images including depth information taken from various angles of objects placed on the base module 130. Based on this, a depth image or a three-dimensional image can be generated.

In this example, one or two or more camera modules 120 that rotate or move are described, but depending on the implementation environment or purpose, a fixed camera module (not shown) may be further provided in addition to the camera module 120. Process 150 identifies objects from the images acquired by them.

The base module 130 performs the function of an item support.

The base module 130 extracts the position and pose of the camera module 120 or the position, size, and characteristics of the object, and extracts a pre-designated reference pattern (see FIG. 11) that improves object recognition performance to at least one of the base module 130. It can be provided on one side. These reference patterns enable more robust object recognition performance. In addition, it is preferable that the surface on which the reference pattern is formed, that is, the surface of the base module 130 or at least one side surface, is formed to be matte, and the background other than the reference pattern can be of various colors or textures, but embodiments of the present invention In fields, it is desirable to form a pre-specified texture (e.g. marble, etc.). This minimizes the influence from lighting and minimizes the influence of the reference pattern being obscured by objects or when a tray containing objects is placed on the base module 130.

The base module 130 may be rotated in a specified direction at a specified speed by an electric motor (not shown) and may also be vibrated at a specified intensity.

The sensor module 140 may include an encoder and a position sensor, and the processor 150 determines the shooting angle or shooting angle of the camera module 120 based on sensing information (e.g., position, angle, etc.) from the sensor module 140. Enables accurate detection and control of rotation or movement position.

The processor 150 identifies each object based on images taken continuously from various angles of the object placed on the base module 130 by the rotating or moving camera module 120. In addition, the processor 150 contains two-dimensional images, depth images (e.g., data with distance values between objects from the camera's viewpoint), and/or three-dimensional images (e.g., three-dimensional position information on the x-axis, y-axis, and z-axis). image data) can be generated, and objects are identified through pre-processing and post-processing of the generated images. In addition, since the camera module 120 is capable of acquiring two-dimensional images or three-dimensional images including depth information, three-dimensional images including two-dimensional images or depth information captured from various angles of objects placed on the base module 130 while rotating or moving Based on the dimensional image, the processor 150 may generate a depth image or a three-dimensional image and use it to identify objects.

When identifying an object placed on the base module 130, the processor 150 performs object identification using results learned in advance (e.g., pattern matching, machine learning, or deep learning learning). Alternatively, the processor 150 may change the position, shape, and form of the object placed on the base module 130 and the state conditions of the object (e.g., luminous intensity of the lighting module, influence of external light, etc.) based on the results of an experiment. Pattern matching, machine learning, or deep learning are performed and the results are stored in the database (DB).

The processor 150 uses a depth image or three-dimensional image generated from an image of an object placed on the base module 130, or an image of an object placed on the base module 130 to identify the object. Perform learning (e.g. pattern matching, machine learning, or deep learning).

At this time, the above pattern matching, machine learning, or deep learning may be performed within the object identification device by the processor 150, but the object identification device and the communication module 160 may be used depending on the implementation environment or implementation intention. It is performed on a server or a pre-designated cloud (not shown) connected by wired/wireless means, and the learning results are transmitted again and stored in a database (DB), and are used by the processor 150 to identify objects.

The processor 150 includes a learning mode for learning objects placed on the base module 130 and an identification mode for identifying objects placed on the base module 130.

The processor 150 controls electric motors (not shown) included in the support module 110, camera module 120, and base module 130 when performing the learning mode and identification mode.

When performing the learning mode and identification mode, the processor 150 determines the image captured through the camera module 120 and the position (i.e., rotation/movement position) of the camera module 120 detected through the sensor module 140. Or, angle information is matched and stored in a database (DB).

The database DB stores images captured through the camera module 120 (e.g., two-dimensional images, depth images, and stereoscopic images) in correspondence with location information and/or angle information detected through the sensor module 140. The database (DB) contains at least one of the programs for performing learning mode and identification mode, learning-related information, learning results, object characteristic information (e.g. length, volume, height, color, etc.), and object price information. It is saved.

The communication module 160 communicates with at least one external device (e.g., display module, user terminal, server, etc.) in a wired or wireless manner to input or output information related to performing the learning mode and identification mode.

For example, the display module can output a guidance message for performing learning in the learning mode, and the display module can also output the name and price information of the identified object in the identification mode. The user terminal can output the name, price information, and payment information of the identified object in identification mode. The server allows information learned from one object identification device to be shared with other object identification devices. For example, the learning information learned about the first object from the first object identification device is transmitted to the second object identification device so that learning information about the same object can be shared with each other. Additionally, the server allows updating operating programs or data for the object identification device.

At this time, in another embodiment, the display module may be formed by being integrally combined with the object identification device.

The lighting module 170 is attached to one inner side of the body (or frame) and illuminates. In embodiments of the present invention, the lighting module 170 is formed at a predetermined location on the upper side (eg, upper center, upper edge, etc.) opposite the base module 130.

The lighting module 170 may illuminate in at least one color, and the brightness may be adjusted when illuminated.

At this time, adjusting the color or brightness of the lighting is for clear photography of the object (i.e., improving identification), for example, the material of the object's packaging (e.g. vinyl, paper, aluminum, etc.) and the information printed on the packaging. Depending on the type of object (e.g. trademark, design, barcode, hologram, etc.), the color or brightness of the lighting can be adjusted for clear photography.

The lighting module 170 may include a visible light LED (LED), and may additionally include a plurality of lighting elements with different lighting characteristics (e.g., halogen, infrared, ultraviolet, etc.).

At this time, the selection of the lighting element is for clear photography of the object (i.e., improved identification), for example, the material of the object's packaging (e.g. vinyl, paper, aluminum, etc.) and the information printed on the packaging (e.g. the object) Depending on the type (brand, design, barcode, hologram, etc.), a lighting device may be selected for clear photography.

Figure 2 is a flowchart for explaining the learning mode operation of the object identification device according to an embodiment of the present invention.

Referring to FIG. 2, when the current mode of the object identification device is the learning mode (example of S101) and an object is mounted on the base module 130 (example of S102), the processor 150 operates the camera module 120. Rotates or moves in a designated direction at a designated speed, and photographs an object placed on the base module 130 (S103).

At this time, the processor 150 can control an electric motor (not shown) to actively rotate or move the camera module 120 according to the shape (e.g., track shape) of the support module 110, or the support module 110. By fixing the camera module 120 to the module 110, the camera module 120 may be manually rotated or moved according to the rotation or movement of the support module 110 (see FIGS. 4 to 8).

Accordingly, the processor 150 produces an image (i.e., two-dimensional image) captured of an object placed on the base module 130 while the camera module 120 rotates or moves (i.e., active/passive rotation or movement). Or, learning to identify (or recognize) an object using depth images and/or three-dimensional images generated from images of objects placed on the base module 130 (e.g., pattern matching, machine learning, or deep learning learning) ) is performed (S104), and the learning information (i.e., object identification or recognition information) and the corresponding sales information (e.g., price) are stored in the database (DB) (S105). In addition, if the camera module 120 is capable of acquiring not only a two-dimensional image but also a three-dimensional image including depth information, a two-dimensional image captured from various angles of an object placed on the base module 130 while rotating or moving or including depth information A depth image or stereoscopic image is generated based on the 3D image, and the processor 150 may learn to identify objects using the depth image or stereoscopic image. Alternatively, the learning may be performed on a server or cloud (not shown), so in this case, the processor 150 transmits the image (two-dimensional image) or the generated depth image and/or stereoscopic image to an external device. It is transmitted to a server or cloud (not shown).

Additionally, although not shown in the drawing, the processor 150 may output a guidance message for performing learning in the learning mode through the display module.

Figure 3 is a flowchart for explaining the identification mode operation of the object identification device according to an embodiment of the present invention.

Referring to FIG. 3, when the current mode of the object identification device is the identification mode (example of S201) and an object is mounted on the base module 130 (example of S202), the processor 150 operates the camera module 120. Rotates or moves in a designated direction according to a designated speed, and photographs an object placed on the base module 130 (S203).

At this time, as in the learning mode, the processor 150 controls an electric motor (not shown) to actively rotate or rotate the camera module 120 according to the shape (e.g., orbital shape) of the support module 110. It can be moved, or when the camera module 120 is fixedly attached to the support module 110, the camera module 120 can be manually rotated or moved according to the rotation or movement of the support module 110 (Figure 4 to 8).

Accordingly, the processor 150 captures an image (i.e., a two-dimensional image) of an object placed on the base module 130 while the camera module 120 rotates or moves (i.e., active/passive rotation or movement). , or using a depth image and/or three-dimensional image generated from images of an object placed on the base module 130, identify (or recognize) the object (S204), and attach the object to the identified (or recognized) object. Corresponding sales information (e.g. price) is loaded from the database (DB) (S205). In addition, if the camera module 120 is capable of acquiring not only a two-dimensional image but also a three-dimensional image including depth information, a two-dimensional image captured from various angles of an object placed on the base module 130 while rotating or moving or including depth information A depth image or three-dimensional image may be generated based on the three-dimensional image, and the processor 150 may identify (or recognize) an object using the depth image or three-dimensional image (S204), and the object thus identified (or recognized) may be generated. Sales information (e.g. price) corresponding to can be loaded from the database (DB) (S205).

In addition, although not shown in the drawing, the processor 150 outputs (i.e. transmits or displays) payment information summing up sales information (e.g. price) corresponding to the identified (or recognized) object to the display module or user terminal. )can do.

Hereinafter, a method of rotating or moving the camera module 120 will be described with reference to FIGS. 4 to 8.

Figure 4 is an example diagram for explaining a method of rotating a camera module according to an embodiment of the present invention.

As shown in FIG. 4, the support module 110 is formed at a designated angle on one inner side of the body (or frame), and the camera module 120 is attached to it.

For example, as shown in (a) and (b) of Figures 4, the support module 110 may be formed horizontally on the inner upper side of the body (or frame), specifically as shown in (a). One camera module 120 may be attached to the support module 110, or a plurality of camera modules 120 may be attached to one support module 110 as shown in (b).

Additionally, as shown in (c) and (d) of Figures 4, the support module 110 may be formed at a specified angle on one inner side of the body (or frame), and as shown in (c), the support module 110 may be formed at a specified angle. The support module 110 may be formed at a designated angle, or, as shown in (d), a plurality of support modules 110 may each be formed at a designated angle.

Figure 5 is an example diagram showing the rotation range of the camera module in Figure 4.

As shown in (a) of FIG. 5, the camera module 120 rotates 360 degrees and can capture objects placed on the base module 130 from 360 degrees in all directions.

Alternatively, as shown in (b) of FIG. 5, the camera module 120 rotates within a specified angle range (e.g., 180 degrees, 270 degrees, etc.) and detects an object placed on the base module 130 within the specified angle range (e.g. : You can also shoot at 180 degrees, 270 degrees, etc.)

Figure 6 is an example diagram for explaining a method of rotating the camera module using a plurality of support modules in Figure 4.

Referring to FIG. 6, a plurality of support modules 110 are formed at different heights (e.g., upper and lower sides) inside the body (or frame) of the object identification device, and are at different heights as shown in (a). A plurality of support modules 110 formed on the upper and lower sides (e.g., upper and lower sides) may all be formed in a form that allows the camera module 120 to rotate 360 degrees, or may be formed at different heights as shown in (b). Example: One of the plurality of support modules 110 formed on the upper and lower sides is formed in a shape that can rotate the camera module 120 360 degrees, and the other one supports the camera module 120 within a specified angle range (e.g. : It is formed in an arc shape that can be rotated within 270 degrees, or as shown in (c), a plurality of support modules 110 formed at different heights (e.g., upper and lower sides) are all connected to the camera module 120. ) can be formed in the form of an arc that can be rotated within a specified angle range (e.g. 270 degrees).

Figure 7 is an example diagram for explaining a method of moving a camera module according to an embodiment of the present invention.

As shown in (a) of FIG. 7, the camera module 120 moves in the horizontal direction (or X, Y direction) from the corner along the support module 110 formed at the inner corner of the body (or frame). The support module 110 can move in the vertical direction (or Z direction) at the corner and is formed horizontally with respect to one inner side (e.g., upper side) of the body (or frame) as shown in (b) and (c). ), it can move in the It can move within a specified angle range (e.g., 270 degrees) along the support module 110 formed in the shape, and as shown in (e), it moves from one point on one side of the body (or frame) to the other side of the body (or frame). It may also move in a straight direction (or diagonal direction) along the support module 110 formed in a straight line (or a straight line with an inclination) to one point.

Figure 8 is an example diagram for explaining a method of moving a plurality of camera modules in Figure 7.

As shown in (a) of FIG. 8, a plurality of camera modules 120 can move horizontally in the X-axis direction along the support module 110 formed horizontally with respect to one inner side of the body (or frame), As shown in (b), a plurality of camera modules 120 may move horizontally in the Y-axis direction along the support module 110 formed horizontally with respect to one inner side of the body (or frame).

In this embodiment, three-dimensional photography of objects is possible with only one camera module 120, but there is an effect of minimizing image acquisition and recognition time by using a plurality of camera modules 120, and the plurality of camera modules 120 ), the object recognition (identification) rate can be improved by configuring the angle of view of each camera module 120 differently.

Figure 9 is an exemplary diagram showing the schematic form of an object identification device according to an embodiment of the present invention. As shown in (a), the object identification device has an open front side and places an object on the base module 130. It can be formed into a form that can be placed and easily recovered. In addition, as shown in (b) and (c), the object identification device has a completely open front and left and right sides, so that objects can be more easily placed on or retrieved from the base module 130. . And, as in (c), the upper side may be formed in a circular shape, and the rear side may be formed to be narrower than the width of the base module 130 when viewed from the front. The object identification device according to the present invention can be formed in various forms. It goes without saying that it can be designed and is not limited to the examples illustrated.

Although not specifically shown in the drawing, the processor 150 can identify objects in the scan (photography) area on the base module 130 through the camera module 120, sensor module 140, or another camera module. User's actions (e.g. placing or removing an object, approaching the object identification device for payment, placing an object on the base module and waiting for payment, moving away from the object identification device after payment is completed, etc.) Changes can be detected in real time.

Additionally, the object identification device according to this embodiment is combined with a payment system (e.g., POS system), allowing immediate payment based on object identification (recognition) information.

Figure 10 is an exemplary diagram showing a photograph taken during a test process of an object identification device according to an embodiment of the present invention. In a state in which the body (frame) of the object identification device is opened, (a) is a photograph taken from the front. , (b) is a photo taken from the side, and (c) is a photo taken from the top inside the body (frame).

FIG. 11 is an example diagram showing a reference pattern formed on the base module and the inner side of the body or frame in FIG. 1.

In this embodiment, the reference pattern has the purpose of extracting and calibrating the position and pose of the camera module 120 and improving object recognition performance by extracting the position, size, and characteristics of the object placed on the base module 130. The purpose is to use the pattern to create uniform lighting even in various lighting environments. These reference patterns allow object recognition performance to be further improved.

The reference pattern may be formed using ink that is displayed so that the user can always recognize it, or it may be formed using ink that is displayed so that the user can recognize it by displaying it only under certain lighting conditions, or it may be formed using ink that is not displayed normally and is displayed under specific lighting conditions. It may be formed using ink that is displayed so that only the camera module 120 can be recognized in this state.

For example, the reference pattern divides the upper surface of the base module 130 into a first area (e.g., inner area) and a second area (e.g., border area), as shown in (a) of FIG. 11, and includes at least one part of the second area. A ruler is formed on the side, and in the first area, a pattern of black and white squares is formed in a checkered (or grid pattern) shape. As shown in (b) of FIG. 11, the upper surface of the base module 130 is formed in the first area (example). : It is divided into an inner area) and a second area (e.g. border area). In the second area, black and white squares are formed in the form of sequentially repeating lines, and in the first area, a color different from the background color (e.g. white) (e.g. : A pattern in which a designated figure (e.g., circle) of black color is formed in a checkered (or grid pattern) shape, as shown in (c) of FIG. 11, the upper surface of the base module 130 is formed as a first area (e.g., inner area). It is divided into a second area (e.g. border area), and in the second area, straight lines heading toward the center of the first area (i.e. straight lines that serve as a ruler) are arranged at specified intervals up to the border of the first area. In the first area, there is a pattern in which a background color (e.g. white) and a designated figure (e.g. a circle) of a different color (e.g. black) are formed in the form of a checkered pattern (or grid pattern), and a base module (as shown in (d) of Figure 11). 130) divides the upper surface into a first area (e.g. inner area) and a second area (e.g. border area), letters or numbers are arranged in the form of lines in the second area and a background color (e.g. marble) in the first area. A pattern in which a designated figure (e.g., a square) of a different color (e.g., black) and a texture are formed in a checkered (or grid pattern) form, as shown in (e) of FIG. 11, the upper surface of the base module 130 is formed as a first It is divided into an area (e.g., inner area) and a second area (e.g., border area), and only the first area has a checkered pattern (e.g., a square) with a background color (e.g., white) and a different color (e.g., black). or a grid pattern), as shown in (f) of FIG. 11, the upper surface of the base module 130 is divided into a first area (e.g., inner area) and a second area (e.g., border area), and the first area is divided into a second area (e.g., border area). A pattern in which black and white squares are formed in a checkered (or grid pattern) pattern only in the area, and the upper surface of the base module 130 is divided into a first area (e.g., inner area) and a second area, as shown in (g) of FIG. 11. It is divided into areas (e.g., border areas), and only the first area includes at least one type of pattern formed by alternating straight lines of different lengths directed from the outer border toward the center at specified intervals.

Additionally, the pattern may be formed on the inner side of the body (or frame) as shown in (h) of FIG. 11. Of course, various types of patterns may be formed on one inner side.

Additionally, the pattern may be formed on one inner side of the body (or frame) and the surface of the base module 130, as shown in (i) of FIG. 11. In the case of this example (i), text on how to use the object identification device is written on one inner side of the body (or frame) in a plurality of rows from a predetermined height (b) from the surface of the base module 130 at predetermined row intervals (a). ), and on the surface of the base module 130, the first area (e.g., inner area) and the second area (e.g., border area) are separated by a thick line, and a predetermined text is formed in the second area. These phrases are used as reference patterns. In this case, at least one of the phrases formed on the side surface and the edge of the base module 130 can be used as a reference pattern, so that the reference pattern is formed in a location area where it is difficult to be obscured by objects and serves as a standard for object recognition and the camera Processing such as module calibration and lighting brightness response becomes easier.

At this time, in the various examples of reference patterns described above, the first area and the second area are each formed by a border of a designated color (eg, white, black, etc.) and thickness, and is formed in a designated shape (eg, circular, square, etc.). The areas are separated. However, there is no need to be limited to these. That is, the white or black color or shape of the first and second areas may vary depending on the implementation environment or purpose and the setting standards for lighting and object recognition. For example, the background color may be gray rather than white, or it may be a texture background, and the color of the pattern may be light black rather than black.

Additionally, the width and position of the first area and the second area can be adjusted, and do not limit the information (e.g. letters, numbers, lines, shapes, images, characters, etc.) shown in the first area and the second area, respectively. .

Accordingly, the processor 150 can use the information of the reference pattern to measure the size of the object from the captured image, improve the performance of distinguishing between the object and the background through the reference pattern, and use the reference pattern to detect the sensor (e.g. : Encoder), accuracy can be improved when 3D modeling based on images acquired from multiple views through reference patterns, and lighting brightness can be equalized through reference patterns, and camera module It can be used for geometric calibration of (120).

Figure 12 is an example diagram for explaining a method of utilizing a reference pattern for object recognition according to the first embodiment of the present invention.

As shown in (a) and (b) of FIG. 12, the processor 150 stores size information (i.e., a′×a″) and/or border of each grid pattern of the reference pattern formed on the base module 130. Based on the area ruler and/or the number of grid patterns obscured by the object, the X*Y axis size (e.g. area) of the object can be measured or estimated.

As shown in (c) and (d) of FIGS. 12, the processor 150 includes size information (i.e., a′×a″) of each grid pattern of the reference pattern formed on the inner side of the body (or frame). /or the number of grid patterns obscured by rulers and/or objects in the border area ( ), you can measure or estimate the Z-axis size (e.g. height) of the object.

Figure 13 is an example diagram for explaining a method of utilizing a reference pattern for object recognition according to the second embodiment of the present invention.

Referring to FIG. 13, the processor 150 based on reference pattern information formed in the base module 130 (e.g., grid pattern information and coordinate information formed in the first area and ruler information formed in the second area), (a ), by extracting the difference image of the reference pattern information where the object is located as in (b) from the reference pattern information in which the object is not placed, as in (c), the position of the object placed on the base module 130 is measured or It can be estimated.

Figure 14 is an example diagram for explaining a method of using a reference pattern for object recognition according to the third embodiment of the present invention.

In this embodiment, the camera module 120 rotates and photographs objects placed on the base module 130 from various angles. Therefore, a sensor (eg, encoder sensor) can be provided to accurately detect the rotation angle when the camera module 120 rotates. However, including a sensor has the disadvantage of increasing cost and requiring an algorithm to detect sensor failure.

Therefore, the embodiment according to the present invention can be used to detect the rotation angle of the camera module 120 using the reference pattern as shown in FIG. 14 even without adding a separate sensor (e.g., encoder sensor).

The processor 150 uses, for example, the camera module 120 and The angle at which the line formed in the second area (i.e., the line corresponding to ruler information) becomes a straight line can be measured or estimated as the rotation angle of the camera module 120.

Figure 15 is an example diagram for explaining a method of using a reference pattern for object recognition according to the fourth embodiment of the present invention.

In general, object recognition performance of the camera module 120 may deteriorate due to changes in the surrounding lighting environment.

Therefore, in order to provide uniform lighting brightness, the present invention corrects the lighting brightness in real time in response to changes in the surrounding lighting environment, based on the image captured through the camera module 120, even without using an illuminance sensor, The processor 150 measures and corrects the lighting brightness of the object recognition device in real time.

For example, as shown in (a) of FIG. 15, when a reference pattern is photographed in a standard illumination brightness state and stored in advance as reference brightness information, the processor 150 records the entire reference pattern from the reference pattern captured image as shown in (b). Process of measuring brightness in real time, process of measuring brightness of the first grid (e.g. white grid) of the reference pattern in real time from the reference pattern shooting image as in (c), reference from the reference pattern shooting image as in (d) The process of measuring the brightness of the second grids (e.g. black grids) of a pattern in real time, the brightness of the first grids (e.g. white grids) and the brightness of the second grids (e.g. black grids) are equal to the pre-stored reference brightness. The process of adjusting the lighting brightness of the lighting module 170 and checking whether the brightness of the entire reference pattern is the same as the pre-stored reference brightness are repeatedly performed to provide uniform lighting brightness.

Figure 16 is an example diagram for explaining a method of utilizing a reference pattern for object recognition according to the fifth embodiment of the present invention.

As described with reference to FIG. 11, reference pattern information (e.g., grid pattern information and coordinate information formed in the first area and ruler information formed in the second area) according to the present embodiment is stored in the first area and the second area. By combining the formed information, the four sides of the reference pattern can be distinguished, and since grids of a certain size are repeated within the reference pattern, the coordinates can be specified by measuring the distance (or length) of the X and Y axes.

Therefore, even if the reference pattern is photographed at different positions using a plurality of camera modules 120, the coordinates can be specified in the same manner.

For example, even if a plurality of camera modules 120 located in different positions each photograph an object placed on the reference pattern, the coordinates of the area where the object is placed can be identically specified.

Accordingly, the processor 150 selects the same object (i.e., an object located on the same coordinates) photographed simultaneously by a plurality of camera modules 120, or the same object photographed while rotating or moving by one camera module 120 ( In other words, it is possible to integrate (or share) information about objects located on the same coordinates (that is, information related to images taken from different directions or angles of the same object).

When using multiple camera modules, objects can be recognized by extracting them from each camera, but sharing object location coordinates between multiple cameras is possible through geometric calibration. Therefore, by sharing the position coordinate information of an object extracted from one camera with images acquired by cameras other than the corresponding camera, there is no need to additionally extract the position coordinates of the object from multiple images acquired from angles other than the corresponding angle, so the object can be identified from multiple images. It has the characteristic of being able to shorten the position detection time. Additionally, in the case of certain objects, occlusion of objects may occur in cameras at specific locations, and since such occlusion can be estimated in advance based on geometric calibration, the possibility of misrecognition can be avoided by excluding recognition attempts in that area. There is an advantage.

Accordingly, the processor 150 can generate a stereoscopic image (or 3D image) by integrating (or sharing) information about the same object, thereby enabling more accurate object recognition.

Figure 17 is an example diagram showing some of the images taken in all directions of 360 degrees of the same object based on a reference pattern in Figure 16. In this example, for example, images taken of an object (eg, a milk carton product) from 36 different angles while the camera module 120 rotates/moves are shown.

Referring to FIG. 17, the processor 150 moves (or rotates) a plurality of camera modules 120 at designated angles to simultaneously capture the same object (i.e., an object located on the same coordinates) or one camera module. (120) Information about the same object (i.e., an object located on the same coordinates) photographed while rotating or moving 360 degrees (or a specified angle) (i.e., information related to images taken from different directions for the same object) information) can be integrated to create a three-dimensional image (or 3D image).

FIG. 18 is an example diagram for comparing three-dimensional images generated when photographing the same object based on a reference pattern and when photographing the same object without using a reference pattern in FIG. 16 .

Assuming that a reference pattern as shown in (a) of FIG. 18 is formed on the base module 130, when photographing the same object from different directions as shown in (b), the processor 150 By making it possible to specify the coordinates of an object placed on the base module 130 based on the reference pattern and the reference pattern marker formed around the reference pattern, a stereoscopic image (or 3D image) can be created using multiple images of objects with the same coordinates. ) can improve image consistency when generating.

For example, as shown in (c) of FIG. 18, when creating a three-dimensional image (or 3D image) using an image taken of an object placed on the base module 130 without a reference pattern and reference pattern marker, image registration is required. It can be seen that the degree is decreasing. In addition, as shown in (d) of FIG. 18, when creating a three-dimensional image (or 3D image) using an image taken of an object placed on the base module 130 with a reference pattern and a reference pattern marker, image registration is required. It can be seen that the degree is improving.

At this time, the reference pattern marker may be replaced with information on the second area among the first area (eg, inner area) and the second area (eg, border area) of the reference pattern.

Figure 19 is an example diagram for explaining a method of using a reference pattern for object recognition according to the sixth embodiment of the present invention.

As described above, the reference pattern is also formed on the upper surface of the base module 130 and the inner side of the body (or frame) (see (h) in FIG. 11), as shown in (a) to (d) of FIGS. As described above, when part or all of the reference pattern (or reference pattern marker) is obscured by an object, the processor 150 uses the reference pattern (or reference pattern marker) formed in other parts that are not obscured to place the object. By specifying the coordinates, the degree of image matching can be improved when generating a stereoscopic image (or 3D image).

As described above, the present invention according to one aspect can extract the position and pose of the camera module 120 and perform position control and calibration of the camera module 120 using a reference pattern even if it does not include a sensor, see This has the effect of further improving object recognition performance by using patterns to create a three-dimensional image (or 3D image) by integrating images of the same object based on specific coordinates.

Meanwhile, in the above embodiments, the camera module 120 rotates or moves to photograph one or more objects placed on the base module 130 and uses a reference pattern and/or reference pattern marker to further increase the performance of identifying objects from the captured image. Although the method of utilizing has been described, the method of utilizing the reference pattern and/or reference pattern marker of the present invention can also be applied to an object identification device in which one or a plurality of camera modules 120 are fixed.

Figure 20 is an example diagram to explain a method of generating a three-dimensional image using a rotating or moving camera module instead of a stereo camera in the present invention.

Figure 20 (a) is an example diagram showing a stereo camera, which is a camera that can capture three-dimensional images using two cameras arranged in parallel, but has the disadvantage of having to install two cameras.

Figure 20 (b) shows a rotating or moving camera module 120 according to the present invention, which allows three-dimensional images to be captured using a single camera module 120 that can rotate or move along the support module 110. .

For example, the camera module 120 when at position A plays the role of the A camera among the stereo cameras, and the camera module 120 when at the B position plays the role of the B camera among the stereo cameras.

Accordingly, the processor 150 of the present invention uses a single camera even without using a separate device (e.g., RGB-D, TOF, Stereo camera, etc.) for capturing three-dimensional images (i.e., for acquiring depth information). Using the module 120, images of the same object can be obtained from various angles and positions through rotation or movement, and the size and depth information of the object can be measured based on triangulation.

FIG. 21 is an example diagram for explaining the method of generating a three-dimensional image using an image captured with structured light in FIG. 20.

In (a) and (b) of Figure 21, in order to obtain more precise depth information, structured light (or structured light pattern) is radiated to an object and then rotated or moved using a camera module 120, respectively. This is an example showing images taken of the same object from different positions and angles.

At this time, structured light (or structured light pattern) may be mounted at a designated location within the object identification device or may be mounted on the camera module 120.

Referring to FIG. 21, after irradiating an object placed on the base module 130 as shown in (c) with structured light (or structured light pattern) as shown in (d) and (e), the object is irradiated with structured light (or structured light pattern). , images of objects can be captured at different positions and angles using a rotating or moving camera module 120, and depth information can be measured based on the two images as shown in (f).

Figure 22 is an example diagram to explain accuracy when measuring depth information using a rotating or moving camera in the present invention.

Referring to FIG. 22, the depth is obtained from an image captured using the camera module 120 rotating or moving while a first object (e.g., a cookie) is placed on the base module 130 as shown in (a). When the information was measured, 25 mm was measured as shown in (b), and the camera rotated or moved with a second object (e.g. bread) placed on the base module 130 as shown in (c). When depth information was measured from an image captured using the module 120, 80 mm was measured as shown in (d), and a second object overlapped on top of the first object as shown in (e). When depth information was measured from an image captured using the rotating or moving camera module 120 in a state (e.g., bread overlapping on top of a cookie), 105 mm was accurately measured as shown in (f). , It can be confirmed that even overlapping objects can be accurately identified based on depth information (or height information).

Figure 23 is an example diagram showing images taken of various objects with different depth information using a rotating or moving camera in the present invention, and Figure 24 is a 2D and This is an example diagram showing depth information.

By grouping 2D and depth information of various objects in the database and creating artificial intelligence (AI) models for each group, it is possible to lighten the learning model, reduce processing time, and improve recognition (identification) performance. do. At this time, different artificial intelligence (AI) models can be created and applied for each feature of the object.

In other words, by extracting the 2D and depth information of objects as features and using them as deep learning learning features, the identification of objects can be improved, or by learning all objects as one model and grouping them when making inferences. The final inference can be made by selecting only the candidates for the objects.

Referring to Figure 24, 2D and depth information that can classify features by shape of an object include image data (IMAGE DATA), color (COLORS), size (SIZE), shape (SHAPE), and position (POSITION). ), transaction history (TRANSACTION HISTORY), geometry (GEOMETRY), and brightness (BRIGHTNESS) information.

Figure 25 is an example diagram shown to explain the problem of a method of generating a three-dimensional image based on an image taken of an object placed on an existing rotating turntable using a fixed camera, and the turntable structure is similar to that of the object. There are restrictions on size and placement depending on the size and rotation speed of the turntable. For example, if the object exceeds the size (or width) of the turntable, as shown in (b) of Figure 25, it is difficult to photograph, and if the object is placed at the edge of the turntable, as shown in (c) of Figure 25, the turntable's When rotating, it may fall to the outside, making filming difficult.

On the other hand, in this embodiment, as shown in FIGS. 4 to 8, the base module 130 is not rotated, but the camera module 120 is rotated or rotated at various angles and directions depending on the shape of the support module 110. Since images of objects can be captured at various angles and positions while moving (see FIG. 17), there is an effect of creating more accurate three-dimensional images (i.e., 3D modeling images) than before.

Figure 26 is an example diagram showing a three-dimensional image generated based on an image of an object captured while rotating or moving the camera module at various angles and directions according to an embodiment of the present invention.

As shown in Figure 26, the present invention moves (or rotates) at least one camera module 120 by a specified angle according to the shape of the support module 110 to capture the same object (i.e., placed on the same coordinates). A three-dimensional image (or 3D modeling image) can be generated by integrating a plurality of information about an object (i.e., information related to images taken from different directions for the same object) (see FIG. 17).

Figure 27 is an example diagram showing a photograph testing the shooting operation of the camera module of the object identification device according to an embodiment of the present invention, (a) is an object identification device with an open body (frame) placed on the base module Pictures are taken of an object, and (b) to (d) are pictures taken of an object by the camera module 120 that moves (or rotates) at a specified angle along the support module 110 formed on the upper inner side of the body.

In this way, this embodiment allows to generate a three-dimensional image (or 3D modeling image) by integrating images taken from various directions and angles even when one camera module 120 is used for the same object.

Figure 28 is an example diagram for explaining a method of training an object identification device according to an embodiment of the present invention. More specifically, Figure 28 constructs 3D modeling from a plurality of images acquired from the rotating and moving camera module 120 in the object identification device according to an embodiment of the present invention, and objects are located in various positions through the 3D modeling. This is an example to explain how to artificially generate and learn data about things placed and placed in various poses.

As shown in (a) of FIG. 28, the processor 150 generates a depth image or three-dimensional image from an image of an object placed on the base module 130, or an image of an object placed on the base module 130. Create a video.

When a depth image or a three-dimensional image of an object is generated in this way, as shown in (b), the processor 150 uses the depth image or three-dimensional image of each object to place various positions on the base module 130. Generates a plurality of virtual 2D images (i.e., virtual images) positioned at an angle with .

At this time, when comparing the virtual 2D image (i.e., virtual image) generated as shown in (b) and the 2D image (i.e., real image) in which the object was actually photographed as shown in (c), they are virtually identical. Able to know. Therefore, when creating a virtual 2D image (i.e., virtual image) using a three-dimensional image of an object, it is much faster and easier to create multiple 2D images (i.e., virtual image) than shooting an actual 2D image (i.e., real image). ) can be created.

In this way, when a plurality of virtual 2D images (i.e., virtual images) are generated using a three-dimensional image of an object, as shown in (d), the processor 150 creates a plurality of virtual 2D images (i.e., virtual images). ) is used to perform learning (e.g. pattern matching, machine learning, or deep learning) to identify objects.

At this time, the above-described virtual 2D image generation, pattern matching, machine learning, or deep learning may be performed within the object identification device by the processor 150, but may be performed with the object identification device depending on the implementation environment or implementation intention. It may be performed on a server or a pre-designated cloud (not shown) connected wired/wirelessly through the communication module 160, and the results may be transmitted again and stored in a database (DB), and may be processed by the processor 150 as an object. It can be used for identification.

Figure 29 is an example diagram to explain a method of improving the identification time and identification performance of an object by sharing coordinate information of the object captured while the camera module rotates or moves along the support module according to an embodiment of the present invention.

Referring to FIG. 29, the processor 150 shares the coordinate information of the object captured according to the rotation or movement of the camera module 120, so that the same object can be identified without detecting the position of the object each time in the captured image. do.

The present invention provides a method of improving the identification time and identification performance of objects by sharing information about objects located on the same coordinates captured by at least one camera module 120 rotating or moving along the support module 110. This is an example for explanation.

The present invention makes it possible to generate a three-dimensional image (or 3D image) by integrating (or sharing) information about the same object taken from various directions, thereby preventing the second object from being obscured by the first object at a specific location. In this case, it has the effect of improving identification of objects located on the same coordinates.

The present invention has been described above with reference to the embodiments shown in the drawings, but these are merely illustrative, and various modifications and other equivalent embodiments can be made by those skilled in the art. You will understand the point. Therefore, the scope of technical protection of the present invention should be determined by the claims below. Implementations described herein may also be implemented as, for example, a method or process, device, software program, data stream, or signal. Although discussed only in the context of a single form of implementation (eg, only as a method), implementations of the features discussed may also be implemented in other forms (eg, devices or programs). The device may be implemented with appropriate hardware, software, firmware, etc. The method may be implemented in a device such as a processor, which generally refers to a processing device including a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices such as computers, cell phones, portable/personal digital assistants (“PDAs”) and other devices that facilitate communication of information between end-users.

110,111,112: Support module 120: Camera module
130: Base module 140: Sensor module
150: processor 160: communication module
170: Lighting module DB: Database

Claims (11)

At least one camera module that rotates or moves through the support module and photographs objects in the base module from various positions or angles; and
The camera module generates a three-dimensional image of the object based on images taken from various positions or angles of the object, and uses the three-dimensional image of the object to generate a plurality of virtual 2D images of the object or identify the object. An object identification device comprising a processor.
According to clause 1,
It further includes structured light to support precise image capture of the object,
The structured light is
An object identification device, characterized in that it is mounted at a designated location within the object identification device or mounted on the camera module.
The method of claim 1, wherein the processor:
Performs learning to identify the objects,
An object identification device characterized in that it determines whether the object overlaps another object based on the depth information of the object measured from the image captured using the camera module.
According to clause 1,
The processor,
Classifies the characteristics of each object's shape based on the object's 2D information and depth information.
The 2D information and depth information are,
At least one of the following information: IMAGE DATA, COLORS, SIZE, SHAPE, POSITION, TRANSACTION HISTORY, GEOMETRY, and BRIGHTNESS. An object identification device comprising:
According to clause 3,
Learning to identify the objects is,
It is performed on a server or pre-designated cloud connected wired or wirelessly through the object identification device and communication module,
An object identification device, wherein the learning results performed on the server or a pre-designated cloud are transmitted again and stored in a database, so that the processor uses the learning results to identify objects.
The method of claim 1, wherein the processor:
An object identification device that identifies the same object in a captured image based on the coordinate information by sharing the coordinate information of the object in the image captured by the camera module, without detecting the location of the object for each captured image.
A step of the processor photographing objects in the base module from various positions or angles using at least one camera module that rotates or moves through the support module;
generating, by the processor, a three-dimensional image of the object based on images taken by the camera module from various positions or angles of the object;
A stereoscopic image generation method using an object identification device, comprising the step of the processor generating a plurality of virtual 2D images of the object or identifying the object using the stereoscopic image of the object.
According to clause 7,
generating, by the processor, a plurality of virtual 2D images of the object using a stereoscopic image of the object and performing learning to identify the object using the virtual 2D images; and
The processor further includes determining whether the object overlaps another object based on depth information of the object measured from an image captured using the camera module. How to create a stereoscopic image.
According to clause 7,
To identify the object,
The processor,
Classifies the characteristics of each object's shape based on the object's 2D information and depth information.
The 2D information and depth information are,
At least one of the following information: IMAGE DATA, COLORS, SIZE, SHAPE, POSITION, TRANSACTION HISTORY, GEOMETRY, and BRIGHTNESS. A method of generating a three-dimensional image using an object identification device, comprising:
According to clause 8,
Learning to identify the objects,
When performed on a server or pre-designated cloud connected wired or wirelessly through the object identification device and communication module,
It further includes receiving the learning results performed from the server or a pre-designated cloud and storing them in a database,
A method of generating a three-dimensional image using an object identification device, wherein the processor uses the learning result to identify the object.
According to clause 7,
The processor,
By sharing the coordinate information of the object in the image captured by the camera module, the object identification device is used to identify the same object in the captured image based on the coordinate information without detecting the location of the object for each captured image. How to create a stereoscopic image.

Images