KR20210041055A - Multi-view imaging system and method for non-invasive inspection in food processing - Google Patents

Abstract

An inline vision-based system 400 and associated methods used for inspection and processing of food ingredients 408 are disclosed. The system includes a conveyor belt 402, a transparent plate 412 and an imaging system 422, the imaging system including a light source 424 and at least one camera. The imaging system produces image data from multiple views of light that passes through objects on the transparent plate and is captured by the camera. The image data corresponds to one of transmission imaging data, interaction imaging data, and reflection imaging data, and is transmitted to the processor 428. The processor processes the data using machine learning to create a three-dimensional model of the structure of the material part inside the object and determine the boundary of the part with respect to the surrounding material.

Description

Multi-view imaging system and method for non-invasive inspection in food processing

FIELD OF THE INVENTION The present disclosure relates to non-invasive inspection in food processing, and more particularly, to an imaging system and associated method for detecting internal objects in food by processing image data of food to determine a three-dimensional model of internal objects. It's D.

The food industry operates within narrow margins and quality control standards are rising. As such, the food processing industry has turned to automated systems to increase processing capacity while meeting higher quality control standards. Aspects of food processing include separation of primary products from secondary products and removal of foreign substances to increase the added value of the product. However, this aspect is difficult to automate because the boundary between the primary and secondary products and foreign matter is difficult to identify with current systems.

When there is a natural boundary between the food material, such as a solid and liquid interface, separating the primary product from the secondary product is often straightforward and requires little physical effort. Active physical intervention, such as cutting, is usually required when food ingredients are connected to each other or when there is a solid-to-solid interface. In order to guide and perform these tasks, it is advantageous to identify the exact boundary or the most probable boundary between solid and solid interfaces, which is often realized through vision-based systems in an automated environment. Even if separation between materials is not required, the ability to detect the presence and extent of certain defects or unwanted objects can be of great benefit to food inspection or sorting processes.

For example, as disclosed in U.S. Patent Nos. It was developed for a variety of industries, including grain. Conventional systems generally apply conventional imaging for rapid screening of food ingredients to obtain processing-related information. However, the information derived through this approach is usually limited to the visible surface of the material.

An alternative technique has been proposed to provide the internal details of the object. Whether data is captured from a surface or within an object, the information is usually presented in a two-dimensional format. For some applications, this may be sufficient because knowing the depth has no particular relevance, or because the geometry of the object is consistent to make certain assumptions. However, there are scenarios where 3D additional information is particularly important. For example, the 3D information is useful for deriving accurate information on the alignment of objects, or processing irregularly shaped objects according to geometric structures to enable precise separation or removal of irregularly shaped objects.

Some solutions aimed at the recovery of three-dimensional (3D) surface data are limited to a rich description of the external profile. Others are based on volumetric imaging techniques such as computed tomography (CT) or magnetic resonance imaging (MRI), which provide data on the internal features of the scanned object, but such a technical solution. The solution has a number of limitations. For example, volumetric imaging technology lacks speed, which is a particularly important limitation when considering the narrow margins of the food processing industry. Among other limitations, the lack of speed makes the current system more suitable as a random quality control inspection tool rather than an in-line solution for automated food processing and sorting.

The present disclosure is directed to fast data acquisition and reconstruction for industrial food processing applications in inline form that enable the capture of geometric details of food ingredient internal components. Such a system is particularly useful for applications where a full representation of the internal structure as provided by volumetric imaging techniques is not required, but rather the recovery and speed of a rough internal profile is essential. In particular, the systems and methods disclosed herein form an outer layer in which one material permits partial transmission of any light spectrum, in particular the second material or inner object of interest is at least partially surrounded by the outer layer. It relates to a surface-to-surface combination of two separate materials that allows penetration or absorption of different ranges of any light spectrum. In general, exemplary implementations of the present disclosure include the use of an imaging system including a light source and an imaging device to capture image data, and a computational step for reconstructing the data to determine the boundaries of an internal object.

An exemplary implementation of a system for capturing and processing image data of an object to determine a boundary of an inner portion of the object includes: a first conveyor; A second conveyor separated from the first conveyor by a gap; A transparent plate positioned within the gap and coupled to at least one of the first conveyor and the second conveyor; A support ring positioned at least partially within the gap and coupled to at least one of the first conveyor and the second conveyor, the support ring comprising at least one camera coupled to the support ring; And a first light source coupled to the support ring, wherein during operation, the first light source emits light directed toward the transparent plate.

Implementations may further include: an object placed on a transparent plate-during operation, the camera receives light passing through the object from a first light source; The object is a tuna fillet, and the first light source emits light at a wavelength of one of approximately 1260 nanometers, approximately 805 nanometers or approximately 770 nanometers; The control unit is in electronic communication with the camera, the camera captures light passing through the object and transmits a signal corresponding to the image data from the captured light to the control unit; The image data is one of transmittance image data, interaction image data or reflectance image data; And, the control unit is a processor, and the processor uses machine learning to detect a boundary between the first part of the object and the second part of the object within the first part based on the image data received from the camera,

Implementations may further include: the processor processes image data via a deep convolutional neural network; The deep convolutional neural network receives image data and outputs a plurality of silhouettes based on the image data corresponding to the second part of the object, and the processor projects the silhouettes into a plurality of projections, and the third part of the second part of the object. Analyzing the intersection between multiple projections to determine the dimensional shape; The support ring includes a plurality of cameras coupled to the support ring, each of the plurality of cameras capturing one of transmission imaging data, cross-cross imaging data or reflective imaging data from a first light source; And, the support ring includes a second light source coupled to the support ring, and during operation, the second light source emits light directed to the transparent plate.

An alternative and exemplary embodiment of an apparatus for capturing and analyzing image data of an object to determine the boundary of an inner portion of the object is a conveyor, the conveyor having a space between the first and second portions of the conveyor; A plate located in the space and coupled to the conveyor; A support ring located at least partially within the space and coupled to the conveyor, the support ring rotating at least between a first position and a second position during operation; At least one light source coupled to the support ring, during operation, the at least one light source emits light directed towards an object on the plate; An imaging device coupled to the support ring, the imaging device receiving light from at least one light source after the light passes through the object; And in electronic communication with the imaging device, receiving a first set of image data from the imaging device when the support ring is in a first position and receiving a second set of image data from the imaging device when the support ring is in a second position. A processor that during operation, the processor outputs a 3D model of the interior portion of the object from the first image data set and the second image data set.

Implementations may further include: The processor processes the first image data set and the second image data set into a plurality of silhouettes and utilizes machine learning to project the plurality of silhouettes into a plurality of projections-3D model. -Is based on the intersection between each of the multiple projections; A second light source coupled to the support ring-the imaging device captures a third set of image data from the second light source when the support ring is in the first position or the second position, and the processor is used to clarify the boundaries of the three-dimensional model. -Using the third image data set; The imaging device comprises a spectrograph-at least one light source emits light of a wavelength selected from one of approximately 1260 nanometers, approximately 805 nanometers or approximately 770 nanometers -.

An exemplary embodiment of a method for capturing and analyzing image data of an object to determine the boundary of an inner portion of the object comprises: emitting light from a light source, wherein the emitting step comprises an object having a first portion and a second portion. Directing light through, the second portion being enclosed within the first portion; Capturing light from the light source with an imaging device after the light passes through the object, the captured light corresponding to image data of the first portion and the second portion received by the imaging device; Transmitting the image data to the processor; And analyzing the image data using a processor to detect a boundary between the first part and the second part-the analyzing step includes utilizing machine learning to generate a three-dimensional representation of the second part. Ham-includes.

Embodiments may further include: emitting light from the light source comprises emitting light at a wavelength selected from one of approximately 1260 nanometers, approximately 805 nanometers or approximately 770 nanometers; The step of utilizing machine learning to generate a three-dimensional representation of the second part includes machine learning using a deep convolutional neural network to process image data; Analyzing the image data using the processor includes utilizing machine learning to output a plurality of two-dimensional silhouettes corresponding to the image data of the second portion; Analyzing the image data using the processor includes utilizing machine learning to generate a plurality of projections, each projection corresponding to a respective one of the plurality of two-dimensional silhouettes; And, the step of analyzing includes the step of including machine learning to generate a three-dimensional representation, and analyzing the intersection between each of the plurality of projections to output a three-dimensional representation of the second part of the object. It further includes.

In order to better understand the embodiments, reference will be made to the accompanying drawings by way of example only. In the drawings, the same reference numbers identify similar elements or actions. The sizes and relative positions of elements in the drawings are not necessarily drawn to scale. For example, various elements, shapes, and angles are not necessarily drawn to scale, and some of these elements may be enlarged and arranged to enhance the readability of the drawings. In addition, the specific shape of the illustrated element is not necessarily for conveying information on the actual shape of the specific element, and may be selected only for easy recognition in the drawings.
1 is a perspective view of an exemplary embodiment of a conveyor belt system according to the present disclosure having a gap between a first conveyor and a second conveyor of the system.
2 is a schematic diagram of an exemplary implementation of an imaging system according to the present disclosure illustrating a transmission imaging mode.
3 is a schematic diagram of an exemplary implementation of an imaging system according to the present disclosure illustrating an interactive imaging mode.
4 is a perspective view of an exemplary embodiment of an imaging system according to the present disclosure having a support ring and a plurality of imaging devices and light sources coupled to the support ring.
5 is a schematic diagram of a control unit of the imaging system of FIG. 4;
6 is a perspective view of an alternative exemplary embodiment of an imaging system according to the present disclosure having a support ring, a single imaging device coupled to the support ring and a light source, wherein the support ring rotates between at least a first position and a second position. do.
7 is a perspective view of an exemplary implementation of an outer housing according to the present disclosure for reducing ambient light to an imaging system within the outer housing.
8 is a schematic diagram of an exemplary implementation for reconstruction of a three-dimensional model from a projected silhouette according to the present disclosure.
9 is a flowchart of an exemplary implementation of a method according to the present disclosure for capturing and processing image data of an object to determine the boundary of an inner portion of the object.
10 is a flowchart of an alternative and exemplary implementation of a method according to the present disclosure for capturing and processing image data of an object to determine the boundary of an inner portion of the object.

In the following description, certain specific details are set forth in order to provide a thorough understanding of the various disclosed implementations. However, one of ordinary skill in the art will recognize that implementations may be practiced without one or more of these specific details or in conjunction with other methods, components, materials, and the like. In other instances, known structures associated with imaging systems, non-invasive testing in food processing, machine learning, and neural networks have not been shown or described in detail in order to avoid unnecessarily obscuring the description of the implementation.

Unless the context otherwise requires, throughout the specification and claims that follow, the word "comprises" and variations thereof such as "comprises" and "comprising" are in an open and inclusive sense, i.e., "includes However, it should be construed as "not limited thereto." Further, the words "first", "second", and words denoting a similar order are to be construed as interchangeable unless the context clearly dictates otherwise.

Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. Thus, appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same implementation. Moreover, certain features, structures or properties may be combined in any suitable manner in one or more embodiments.

As used in this specification and the appended claims, the singular forms “a”, “an” and “the” include plural referents unless the context clearly dictates otherwise. It should also be noted that the term "or" is generally used in its broadest sense, ie, meaning "and/or" unless the context clearly dictates otherwise.

The relative terms “approximately” and “substantially”, when used to describe a value, quantity, quantity, or dimension, are generally +/- of the stated value, unless the context clearly dictates otherwise. Represents a value, quantity, quantity, or dimension within 3%. Any specific dimensions of the components provided herein are for illustrative purposes only, with reference to the exemplary embodiments described herein, and thus the disclosure is intended to be in an amount greater or less than the stated dimensions, unless the context clearly dictates otherwise. It should be further understood that it includes.

The present disclosure provides a solution for a fast, non-invasive imaging system capable of acquiring visual data of food ingredients for processing and inspection. In particular, embodiments of the present disclosure capture a three-dimensional structure of an object of any shape surrounded by different layers of material. For example, in embodiments where the object to be analyzed is a fillet of fish, the imaging systems and methods disclosed herein provide a three-dimensional view of a dark flesh portion (e.g., another layer of material) contained within the outer white flesh layer. Capturing the structure, the dark flesh has an arbitrary three-dimensional shape that fluctuates between successive lean meats.

Embodiments of the present disclosure include various systems, devices, and related methods that can utilize the optical properties of different absorption, reflection, transmission and scattering for different hazy materials and spectral bands. The relative proportion and amount of each occurrence can vary depending on the chemical composition and physical parameters of the material. When light interacts with a material, certain portions of the photons are reflected through specularity, diffusion, or backscatter. The first two, specular and diffuse, depend on the roughness of the surface, and scattering or backscattering of light occurs due to phase shifts inside the material or multiple refractions at different interfaces. In addition, scattering may appear due to heterogeneity such as pore spaces or capillaries randomly distributed throughout the material, and the size, shape, and microstructure of different particles.

The remaining photons that are not reflected are either absorbed or transmitted through the material. The lower the absorption coefficient, the deeper the light can penetrate the material before the photons are absorbed, and thus the more likely the light will exit the other side of the material. Thus, for non-invasive imaging according to various implementations of the systems, devices, and related methods described herein, both scattered light and light passing through a material can provide useful information about internal properties. For example, such information, such as scattered light and light passing through a material, is captured through interaction images or transmittance imaging, respectively, whereas reflectance imaging is described with reference to exemplary embodiments. It focuses primarily on the light reflected directly from the surface as described in the specification.

The optical properties and results from the interaction with the cloudy material are different for each wavelength of light. Some wavelengths are absorbed quickly, while other wavelengths penetrate deeply into the material and may be completely or partially transmitted depending on the thickness. As described in more detail below, some of the implementations of systems, devices and related methods include multiple or hyperspectral phenomena for investigating optical property phenomena for different wavelengths of the ultraviolet (UV), visible and near infrared (NIR) light spectrum. (hyperspectral) imaging tools may be included.

In addition, some implementations described herein may include or utilize a diffraction grating in which light is scattered and the intensity of each wavelength is captured by various implementations of the sensor(s) described herein. In addition, implementations of the systems, apparatuses, and related methods described herein may have spatial data stored in two dimensions and a three-dimensional cube having three-dimensional spectral data, or obtain associated data that may include them. The choice of an individual suitable wavelength or combination of suitable wavelengths may depend on the processed food material and may be determined in advance. For example, the selection of the appropriate wavelength may be selected based on a database containing transmittance and reflectance information for a particular food to be scanned, and a database with spectral properties of light of a particular wavelength, or prior to processing, the system may be Based on the object or food, it can be calibrated to determine the wavelength corresponding to capturing the appropriate imaging data.

In various embodiments of the systems, devices, and methods described herein, an outer layer of material that has but has a low absorption coefficient and minimal scatter, and is completely different from the inner object of interest (e.g., the white and dark flesh of tuna meat, respectively). A specific wavelength is selected that has good penetration ability through flesh). Moreover, in various implementations of the systems, apparatus and methods described herein, in addition to selecting a suitable wavelength for food ingredient inspection, a suitable illumination source may also be selected as a system design parameter. For example, a light source may have a different intensity of emitted light for a particular spectrum. Thus, a light source having a wavelength suitable for the application can be selected in order to obtain optimal results. Several types of illumination sources are disclosed herein, including halogen, LED, and laser light of specific wavelengths.

Acquisition of high quality image data of inspected food ingredients is an aspect of the present disclosure. As mentioned above, although the material being processed is three-dimensional, traditional imaging sensors tend to lack means of determining the depth of the scene, limiting the ability to perceive the complexity of real objects. In order to construct a three-dimensional representation of an object, various implementations of the systems, devices, and related methods described herein can capture a collection of images conveying information from multiple views. Unlike the full 3D representation obtained through volumetric imaging technology, the 3D surface reconstruction technology of conventional imaging systems is the 3D coordinates of individual points located on the surface of the object, or, in this case, 3D of the boundary between the two materials of the object. It tends to capture only the coordinates. Thus, this method is often referred to as surface measurement, distance sensing, depth mapping, or surface scanning, which can be used interchangeably herein unless the context clearly dictates otherwise.

One prior art for 3D surface reconstruction is multi-view stereo (MVS), which matches corresponding feature points in an image, taking into account sufficient overlap between views. The result is a 3D cloud to which the surface can be fitted. This approach generally requires a feature-rich texture, which may or may not be available in certain types of foods. In addition, MVS lacks efficiency for certain applications in the food processing industry due to the small margins involved.

The structured-light 3D surface reconstruction technique is another technique for obtaining 3D information. The structured light method uses spatially variable 1D or 2D structured illumination projected onto an object. In a planar scene, the light pattern is projected equally onto the surface, whereas in a non-planar scene, the pattern viewed by the camera will be distorted. 3D information is usually extracted from the characteristics of distorted patterns obtained from the direct reflection of light at the surface boundary.

Both approaches can provide detailed descriptions of 3D surfaces, but none of them will be practical for non-invasive imaging data captured via interactive mode or transmission mode. On the other hand, the shape from the silhouette will depend on the contour of the object, from which the 3D representation can be restored. As described in more detail below, various implementations of systems, devices, and related technologies are operable to project the silhouette viewed by each camera back into a three-dimensional scene to extract 3D shapes from its intersection. The reconstruction is just an approximation of a true 3D structure, commonly known as a visual hull, since the concavity is generally not visible in the silhouette and will be discarded. However, in various implementations described herein, using interactive imaging together with transmission imaging, a precise 3D shape of an object can be extracted taking into account the concave area.

As will be described in more detail below, the aspect associated with 3D shape reconstruction and acquisition of multi-view image data is the positioning of the camera. In order to project the silhouette back into the scene, it is important to determine where the camera is positioned and how the scene was originally projected onto the image plane. The various systems, devices, and related methods described herein may operate to receive and/or process this information from camera calibration. This information may include how the light is projected through the lens onto the imaging sensor, internal parameters related to any distortion that occurs through this process, and external parameters related to the camera's position in reality. The various systems, devices, and related methods described herein can implement a calibration process, including the internal parameters discussed above, for each camera and position contributing to the acquisition of multi-view image data of the system, and This can be accomplished efficiently by using fiducial markers.

In some implementations, when obtaining image data, the silhouette of the target object may be determined before the three-dimensional model is generated. For example, aspects of the present disclosure may include recognizing which one represents a food ingredient in the image data, and then distinguishing between the different components forming the outer layer and the inner layer. For example, in embodiments where the object is lean fish, the outer layer or first portion corresponds to a first type of meat, and the inner layer or second portion is usually contained within the first type of flesh. Corresponds to the type of flesh. In some aspects, machine learning, more specifically artificial neural networks, can be implemented to perform these tasks.

For example, in a neural network, there are multiple nodes within different layers connected through connections associated with weights. These weights are generally adjusted and learned over multiple iterations by specifying which outputs of the node are predicted, taking into account known inputs. By collecting large data sets of labeled images representing food ingredients and internal defects or objects of interest, deep convolutional neural networks can be trained to learn how to recognize the location and precise boundaries of specific objects. have. With increasing training data sets and meticulous design of neural network architectures, even more complex tasks can be efficiently solved using the various systems, devices and related methods described herein. Moreover, in some implementations, a variety of different modes may be implemented for different specific applications.

As described herein, the present disclosure includes a conveyor belt system for detection and extraction of inspected defects or three-dimensional structures of processed products. Due to the continuously moving belt, it is desirable that the data acquisition and analysis is very efficient. In some embodiments, the wavelength of the applicable light spectrum can be predetermined based on the challenges of the food material and application. Certain spectral bands can be obtained through hyperspectral imaging, or through the use of specific filters or laser light, suggesting a line scan system.

As will be explained in more detail below, in order to obtain transmission data, a light source is positioned opposite the imaging sensor, which is a small gap in the conveyor belt that is filled by a transparent medium that allows the light to be transmitted and passed through the food material. gap). Another light source is placed parallel next to the imaging sensor for the interactive imaging mode. Both imaging modes alternate at high frequencies to avoid obscuring the image data captured by each mode.

As described in more detail below, in the present disclosure, (may be a component of an imaging device, as described in more detail herein, alternatively, generally referred to as an imaging device, and in some embodiments, an imaging device A combination of multiple light sources and camera sensors (which can be referred to as individual components coupled to) is mounted on or around a conveyor belt to collect image data from multiple views. Alternatively, in some implementations, a single camera sensor or imaging device may be used instead. However, in this implementation, a single camera may be placed on a rotating frame that allows relocating the light and camera around the conveyor belt. With a fast conveyor belt speed or reduced acquisition time due to relocation of the camera system, image data is acquired in a spiral arrangement. Thus, the helical image data is interpolated between acquisition points along a transversal path. The number of views may vary depending on the application and required details of the target object. The structure is constructed around the imaging device and blocks any light from outside to control the illumination during imaging and thus achieve better image quality.

The present disclosure, in some implementations, utilizes a deep convolutional neural network trained to detect the location and boundary of a target object. However, according to the review of the present disclosure, it should be understood that the application of a deep convolutional neural network is application dependent and may vary accordingly, and this model may be pre-trained to achieve this task. As described in more detail below, the extracted silhouette is used to generate a rough approximation to the three-dimensional shape of the target object. As should be understood, in accordance with the review of this disclosure, the resolution of the reconstructed model is a trade-off between the required speed and details for the intended application. For example, in some implementations, higher resolution requires a larger number of images and more computational resources for reconstruction, which in turn affects the application speed.

In other implementations, as described herein, the arrangement and number of cameras may vary. Position and camera parameters can be calibrated prior to capturing image data. Due to the movement of the food material on the conveyor belt, the transverse position of the camera is changed with reference to the material being transported. The transverse position can be maintained internally or, if necessary depending on the application, can be set with reference to the material or explicitly defined via a marker on the conveyor belt.

Turning now to the illustrated exemplary implementations, FIG. 1 is a perspective view of a conveyor system 100. It should be understood that the conveyor system 100 has been simplified for easy understanding of implementations of the present disclosure, and therefore, certain features associated with the conveyor system 100 have not been described. The conveyor system 100 includes a first conveyor 102 spaced from the second conveyor 104. In other words, a space or gap 114 separates the first conveyor 102 from the second conveyor 104. The first conveyor 102 may be generally referred to herein as the first part of the conveyor system 100, and similarly, the second conveyor 104 is generally referred to herein as the second part of the conveyor system 100. It can be called this. Moreover, the size and shape of the space or gap 114 and the plate 110 may vary depending on the particular application, so the present disclosure is not limited to the distance or distance between the conveyors 102 and 104 corresponding to the space or gap 114. It is not limited by the size or shape of the plate 110.

Plate 110 is positioned within or close to gap 114 to form a continuous conveyor line. Preferably, the plate 110 is transparent to allow light to pass through the plate 110 without interference. For example, the plate 110 may be formed of transparent plastic, polymer or glass, among other things, and in an alternative embodiment, the plate 110 is translucent, and is similar to, for example, a translucent plastic, polymer or glass. Is formed. The plate 110 is coupled to the conveyor system 100 or, more specifically, to at least one of the first conveyor 102 and the second conveyor 104. Moreover, each conveyor 102, 104 of the system 100 is supported by a support structure 106, and the conveyor surface 112 is translated by a roller 108 and a conventional drive mechanism (not shown). . In addition, the conveyor surface 112 may be solid, or may include a plurality of perforations 116 arranged in a row or evenly distributed over the conveyor surface 112, as shown in FIG. 1. I can. In another embodiment, the conveyor surface 112 is solid, which means that there are no such holes in the surface 112.

2 is a schematic diagram corresponding to the transmission imaging mode 200. The transmission imaging mode 200 includes a light source 202 and an imaging device 204. 2 further shows the object 206 to be scanned, the object 206 being placed on the plate 110. The light source 202 is directed towards the plate 110 and emits light 208 that propagates outward as it moves towards the plate 110. As light 208 passes through plate 110 and object 206, the light converges as indicated by a converging path or portion 210. After exiting object 206, light 208 is captured by imaging device 204. As described herein, the imaging device 204 receives transmission image data corresponding to the captured light 208 transmitted through the object 206, and the transmission image data is then signaled for further processing and reconstruction. To a processor or a control unit (eg, control unit 248 shown in FIG. 4) in electronic communication with the imaging device 204 via.

In the illustrated embodiment, the conveyor system 100 is generally translating the object 206 from right to left with respect to the orientation shown in FIG. 2, but the conveyor system 100 may also translate in other directions. It should be understood that you can. Moreover, the imaging device 204 and the light source 202 are preferably aligned along a vertical axis so that the light 208 output from the light source 202 propagates through the object 206 in a linear manner towards the imaging device, The imaging device 204 is aligned over the light source 202. In addition, due to small variations in the properties or alignment of the object 206 and the light 208, the alignment of the imaging device 204 and the light source 202 may not be strictly vertical, but may instead be within 10 degrees of vertical. , May be within 5 degrees of vertical, or substantially vertical (ie, within 3 degrees of vertical).

The light source 202 is, for example, a laser, a light emitting diode (LED), an LED array or panel, an incandescent lamp, a small fluorescent lamp, a halogen lamp, a metal halide lamp, a fluorescent tube, a neon lamp, a low pressure sodium lamp, or Like a high intensity discharge lamp (HID), it can be selected from one of a variety of sources in various implementations. In embodiments in which the light source 202 is a laser, embodiments of the present disclosure may further include a light source 202 including a solid state laser, a gas laser, an excimer laser, a dye laser, or a semiconductor laser. To the extent that the laser is characterized by the duration of laser emission, the laser may also be a continuous wave laser, a single pulse laser, a single pulse q-switched laser, a repetitive pulse laser or a mode locked laser.

Moreover, as discussed above, preferably, since different light sources 202 output light 208 having different penetrating properties relative to the object 206, the light source 202 may be specifically applied or scanned. It is selected for the object 206. In embodiments where the object 206 is lean fish, preferably, the light source 202 outputs light having a wavelength between 790 and 820 nanometers (nm) in the transmission imaging mode 200, but more preferably , The wavelength is 805 nm or approximately 805 nm (i.e., 800 nm to 810 nm), which corresponds to the wavelength of the infrared portion of the electromagnetic spectrum outside the visible portion of the spectrum, which is generally between approximately 400 and 750 nm. Moreover, this wavelength corresponds to the wavelength of light that allows deep passage into the lean meat while minimizing scattering, at least for tuna lean meat, since scattering reduces the accuracy of the image data corresponding to the transmission imaging mode 200, Scattering is generally undesirable in transmission imaging mode 200.

The near-infrared spectrum greater than 750 nm or at approximately 805 nm is useful for tuna processing, as water is still somewhat permeable to these wavelengths as hemoglobin, which contributes to a substantial part of the biological tissue in tuna fish lean. . Relatively, in the visible spectrum (i.e. 400 nm to 750 nm), hemoglobin absorbs most of the light. One possible explanation for why light passes through white meat at this wavelength but not dark flesh is that it is due to the different density of muscle fibers between the two materials, and the muscle fiber density for dark flesh is white. Much higher than flesh. In this wavelength range (i.e., near infrared or approximately 805 nm) the absorption coefficient for dark flesh is still very high, while the absorption coefficient for white flesh is lower. In addition to physical properties, these properties can be explained, for example, by differences in chemical composition. Thus, for white flesh, the penetration remains rather deep. This difference in behavior makes this particular choice of wavelength (ie, approximately 805 nm) desirable for applications where the object being scanned is a tuna fish lean.

In certain implementations, the imaging device 204 is one of a number of commercially available imaging devices 204 including, but not limited to, a spectrograph, camera, or sensor, among others. In embodiments where the imaging device 204 is a sensor, preferably, the imaging device 204 is a complementary metal oxide semiconductor (CMOS) sensor that captures wavelengths between 300 nm and 1000 nm. Alternatively, the sensor is a charged-coupled device (CCD) that captures similar wavelengths or an indium gallium arsenide (InGaAs) sensor that captures wavelengths between 900 and 1700 nm. In embodiments where the imaging device 204 is a camera or spectroscope, it should be further understood that the camera or spectroscope may include any of the above-described sensors, in addition to other electronic components and other types of sensors.

In calibrating the transmission imaging mode 200, or in changing the mode 200 for different applications, preferably to investigate which wavelength has the best transmission and captures properties for the object 206, Light 208 is divided into individual wavelengths. Although not specifically illustrated, a spectroscope with a diffraction grating may be used to split light 208 into individual wavelengths. Alternatively, since the spectrometer is sensitive and expensive, once the mode 200 is calibrated, only certain selected wavelengths of captured light are passed by the imaging device 204 corresponding to the application to increase efficiency and reduce cost. A cut-off filter can be used. In another alternative, preferably, a laser light source 202 having a specific wavelength corresponding to the desired wavelength selected through calibration or emitting only light within a specific wavelength range can be used, which also reduces cost and increases efficiency. Let it. By comparison, cut-off filters generally have a wider range of pass wavelengths, while lasers are very specific to a particular wavelength, so the choice between the two will depend on the desired operating wavelength for the material in question.

3 is a schematic diagram of an exemplary implementation of an interactive imaging mode 300. The interactive imaging mode 300 includes a light source 302 and an imaging device 304. In some implementations, the imaging device 304 may be different from the imaging device 204 in transmission mode; In some implementations, the imaging device 304 in the interactive imaging mode 300 may be the same as the imaging device 204 in the transmissive mode. In other words, the same imaging device can be used for both the transmissive imaging mode and the interactive imaging mode. Light source 302 and imaging device 304 may be any light source and imaging device described above with reference to light source 202 and imaging device 204, respectively. The object 306 being scanned is on the plate 110 and the light source 302 emits light 310. However, compared to the transmission imaging mode 200, in the interactive imaging mode 300, the light 310 passes through a converging lens 308 coupled to the light source 302 at the output 312 of the light source 302. do. Converging lens 308 may be any one of a number of known converging lenses having a principal axis, focal length, focal length and vertical plane selected according to a particular application. The converging lens 308 helps to clarify the image data captured by the imaging device 304, among other advantages.

In the interactive imaging mode 300 for an embodiment where the object 306 is lean fish and more specifically lean meat, preferably, the light source 302 is between approximately 740 and 800 nm, more preferably , 770 nm or approximately 770 nm (ie, between 765 and 775 nm). Further, preferably, the imaging device 304 is, and preferably includes, a CMOS sensor or a CCD sensor as described herein. This wavelength range has been found to be desirable for the interactive imaging mode 300 based on the analysis described above with respect to the preferred wavelength of the transmission imaging mode 200.

After light 310 is emitted by the light source 302 and passes through the converging lens 308, the light 310 passes through the object 306, as indicated by the portion 312 of the light 310. 306). However, in the interactive imaging mode 300, the imaging device 304 measures the light backscattered by the object 306. In other words, the portion 314 of the light 310 is within the object 306 due to the material composition of the object 306 and the light 310 before exiting the object 306 after entering the object 306. Corresponds to light that bends, bends, or changes direction. In other words, the light 310 emitted through the converging lens 308 propagates along the first direction 305 and the light 310 exiting the object propagates in the second direction 307, one embodiment In, the first and second directions are opposite to each other along a parallel axis. However, implementations of the present disclosure also include a first, which may be obliquely in the transverse direction with respect to each other, such as when the light source 302 is oblique with respect to the object 306, as described with reference to FIG. 4. And a second direction. The light 310 exits the object 306 and propagates towards the imaging device 304, while during propagation, the light diffuses as indicated by the dispersing portion 316. When the light 310 is received by the imaging device 304, the imaging device 304 sends interactive imaging data corresponding to the amount of captured light 310, as described herein, to a control unit or To the processor (e.g., control unit 428 shown in FIG. 5).

In the illustrated embodiment, the conveyor 100 generally translates the object 306 from right to left with respect to the orientation shown in FIG. 3, as indicated by arrow 318. Thus, the light source 302 is positioned upstream of the imaging device 304 relative to the translation direction of the conveyor system 100. In other words, the light source 302 is generally positioned proximate and preferably parallel to the imaging device 304. While it may be possible to have a light source 302 downstream from the imaging device 304, this arrangement may result in less accurate imaging data that may need to be corrected during processing. Moreover, it is also possible for the conveyor 100 to translate the object 306 opposite to the direction indicated by the arrow 318, in this case, preferably, the light source is to the left side of the imaging device 304 in the orientation shown ( That is, upstream). In addition, both the light source 302 and the imaging device 304 are positioned above the object 306 and, therefore, are in a transmission imaging mode 200 that captures a portion of light that translates directly through the object 206 along a substantially vertical axis. Conversely, the interactive imaging mode 300 captures a portion 314 of light 310 that scatters back toward the imaging device 304 after entering the object 306.

FIG. 4 is a first diagram of a conveyor system 402, a support ring 414 coupled to the conveyor system 402, a plurality of imaging devices 422 coupled to the support ring 414 and a support ring 414. Shows a perspective view of an exemplary embodiment of an imaging system 400 including at least one light source, such as light source 424.

Conveyor system 402 may include all or a substantial portion of the features described above with reference to conveyor system 100 of FIG. 1. However, briefly, the conveyor system 402 comprises a first conveyor or portion 404 and a second conveyor or portion 406, the first conveyor 404 being a second conveyor ( 406). A transparent plate 412 is preferably positioned within the gap 410 and coupled to the conveyor system 402 to form a continuous conveyor line.

As described above, the support ring or frame 414 is coupled to the conveyor system 402 with supports 416, 418, and preferably, the support ring 414 is during calibration of the imaging system 400. It is circular to facilitate rotation of the support ring 414. Preferably, the support 416 is coupled to and extends from the plate 420 to the conveyor system 402, more specifically to the plate 420 coupled to each of the first and second conveyors 404, 406. It is an adjustable collar. Preferably, the support 418 receives the support ring 414 coupled to the conveyor system 402 so that the support ring 414 can be manually rotated by adjusting the support collar 416 during calibration of the system. It is a base that has an open channel for it. While the support 416 is shown in collar and the support 118 is shown as a base having a channel for receiving the support ring 414, a number of other devices or arrangements include the support ring 414 in the conveyor system 402. It should be understood that may be considered in the present disclosure in order to bind to. For example, in other embodiments, the coupling comprises one or more centrally disposed spokes extending from the conveyor system 402 or other structures located within the space 410 and coupled to the conveyor system 402 or Alternatively, the support ring 414 may be coupled to and supported by a housing, such as the housing shown in FIG. 6.

The support ring 414 further includes a plurality of imaging devices 422 coupled to and extending from the support ring 414. Each imaging device 422 may be substantially similar, if not identical, to the imaging device 422 and any variations thereof described with reference to FIG. 2. Further, the support ring 414 includes a first light source 424, which may be at least any of the light sources discussed above with reference to light source 204 in FIG. 2. 4, a first light source 424 is located between the conveyor system 402, and the light emitted by the first light source 424 is plate 412 and a plate 412 to be imaged or scanned. It is arranged to be oriented towards the object 408 on the image. The light passes through the plate 412 and the object 408 and is received by at least one of the plurality of imaging devices 422, and data corresponding to the light received from the first light source 424 corresponds to transmission imaging data.

In the illustrated embodiment, the support ring further includes a second light source 426 coupled to and extending from the support ring 414 near the plurality of imaging devices 422. Preferably, the second light source 426 is used in the interactive imaging mode, and the second light source 426 is positioned close to and parallel to the imaging device 422. In another embodiment, the second light source 426 is positioned proximate to the imaging device 422, but is tilted at an angle that crosses the field of view of the imaging device 422, as shown in FIG. 4 and described herein. have. The second light source 426 may similarly be any of the aforementioned light sources discussed with reference to light source 202 in FIG. 2. The second light source 426 emits light corresponding to the interactive imaging mode 300 described with reference to FIG. 3. Accordingly, the light emitted by the second light source 426 is directed toward the object 408 in the first direction, changes the direction within the object 408, exits the object 408 in the second direction, and a plurality of Among the imaging devices 422 of, if not all, data corresponding to light received from at least one, and received from the second light source 426 corresponds to interactive imaging data. In the illustrated implementation, the angle between the first and second directions is less than 90 degrees, preferably less than 45 degrees, but the angle will vary depending on the particular application (i.e., the type of object 408 being scanned). It must be understood.

As shown in FIG. 4, the plurality of imaging devices 422 includes five imaging devices 422, and the imaging devices 422 are spaced equal to each other along the circumference, circumference, or inner edge of the support ring 414 Is spaced apart, and the input is directed towards the plate 412 and the object 408 to receive light from one of the light sources 424 and 426. Thus, each imaging device 422 will receive imaging data corresponding to a different view of the object 408. In other words, there will be differences between the views or data from each imaging device 422 due to the arrangement, which will help to generate the silhouette of the object 408 and the three-dimensional image data. Thus, in the illustrated implementation, there is no need to rotate the support ring 414 during normal operation, as the arrangement and selection of the plurality of imaging devices 422 provides multiple views that are input to the machine learning system. The learning system creates a silhouette that is the basis for determining a 3D model based on multiple views, as described herein. However, it should be understood that the specific number, arrangement, and orientation of the imaging devices 422 depend on the object 408 being scanned and the calibration of the system 400, as discussed herein.

Moreover, each imaging device 422 may receive reflectance imaging data from the second light source 426, and the reflective imaging data is between 1230 and 1290 nm, more preferably 1260 nm or approximately 1260 nm. nm (i.e. between 1255 nm and 1265 nm) corresponds to a second light source 426 that outputs light of a wavelength, and the light emitted at this wavelength is reflected from the outer surface of the object 408 to be reflected from the plurality of imaging devices 422 ) Received or captured. Although water is super absorbent for wavelengths above 1000 nm, this wavelength (i.e., approximately 1260 nm) is preferably in the reflective imaging mode, since dark flesh begins to reflect light at approximately 1260 nm, whereas white flesh does not. For. In the reflective imaging mode, each imaging device 422 may further include an InGaAs sensor, as described above, to capture this longer wavelength of light. Reflective imaging data would be particularly useful in situations where the object is only partially contained within the outer layer (i.e., a situation where a portion of the object extends from the outer layer), but in other implementations, the reflective imaging data is In addition, it can be used as a calibration criterion.

4 further shows a control unit 428 in electrical communication with system 400. 5 shows in detail the control unit 428 according to one exemplary and non-limiting implementation. In particular, control unit 428 is generally operable to provide power to system 400 and to process or transmit imaging data received from imaging device 422. 5 schematically depicts various control systems, modules, or other subsystems operative to control system 400 including data exchange between imaging device 422 and control unit 428.

The control unit 428 includes, for example, a controller 422 which is a microprocessor, a digital signal processor, a programmable gate array (PGA) or an application specific integrated circuit (ASIC). The control unit 428 may be one or more non-transitory storage media, such as read only memory (ROM) 440, random access memory (RAM) 438, flash memory (not shown), or other physical computer or processor. Includes a readable storage medium. The non-transitory storage medium may store instructions and/or data used by the controller 422, for example, an operating system (OS) and/or an application. The instructions, when executed by the controller 422, perform the functions of the various implementations of the systems 400 and 500 described herein, including, but not limited to, capturing and processing data from the imaging device 422. You can run the logic to do it.

In embodiments in which the system 500 (see FIG. 6) includes a rotating support ring or frame 504, the controller 428 is connected to one or more actuators (not shown) to control the rotation of the ring 504. Can be communicatively combined. Alternatively, the controller 428 may be communicatively coupled to one or more belts (not shown) to rotate the ring 504. Moreover, the controller 442 can be used to automatically rotate the support ring 504 according to a predetermined manufacturing speed or conveyor speed at a specific position transmitted to the actuator or belt (i.e., the first position and the first position discussed with reference to FIG. 2) may include a command corresponding to the position.

The control unit 428 is a user interface 436 that allows an end user to operate the systems 500, 600 or provide input to the systems 500, 600 regarding the operating state or conditions of the systems 500, 600. ) Can be included. User interface 436 may include a number of user selectable controls accessible from systems 500 and 600. For example, user interface 436 may include a number of switches or keys operable to turn systems 400, 500 ON and OFF, and/or set various operating parameters of systems 400, 500. .

Additionally or alternatively, the user interface 436 may include a display, eg, a touch panel display. Touch panel displays (eg, LCDs with touch sensitive overlays) can provide both input and output interfaces for end users. The touch panel display provides a graphical user interface having various user selectable icons, menus, check boxes, dialog boxes, or other components or elements selectable by the user to set the operating state or condition of the systems 400 and 500. I can. In addition, user interface 436 may include one or more acoustic transducers, eg, one or more speakers and/or microphones. This allows an audible warning notification or signal to be provided to the end user. This, in addition or alternatively, allows the end user to provide an audible command or instruction. The user interface 436 may include additional components and/or other components other than those illustrated or described, and/or may omit some components.

Switches and keys or graphical user interfaces may include, for example, toggle switches, keypads or keyboards, rocker switches, trackballs, joysticks or thumbsticks. Switches and keys or graphical user interfaces can be used, for example, by the end user turning on systems 400, 500, initiating or ending transmissive imaging mode or interactive imaging mode, and communicatively coupling to remote accessories and programs, or It allows you to disassociate, access, transmit or process imaging data, activate or deactivate motors or audio subsystems, initiate or terminate the operating state of the conveyor system, and so on.

The control unit 428 includes a communication subsystem 444 that may include one or more communication modules or components that facilitate communication with various components of one or more external devices, such as a personal computer or process. The communication subsystem 444 may provide wired or wireless connection to one or more external devices. The communication subsystem 444 may include a wireless receiver, wireless transmitter, or wireless transceiver to provide wireless signal paths to various remote components or systems of one or more paired devices. The communication subsystem 444 is, for example, short-range wireless communication (e.g., via Bluetooth or field communication (NFC), frequency identification (RFID) components and protocols) or (e.g., wireless LAN, LPWAN ( Low-Power-Wide-Area Network), may include long-distance wireless communications (via satellite or mobile phone networks), and may include one or more modems, one or more Ethernet, or other types of communication cards or components to do so. . The communication subsystem 444 may include one or more bridges or routers suitable for handling network traffic, including switched packet type communication protocols (TCP/IP), Ethernet, or other networking protocols. In some implementations, wired or wireless communication with an external device can provide access to lookup tables representing various material properties and light wavelength properties. For example, an end user may select a material from a variety of materials displayed in the user interface 436 that may be stored in a lookup table or similar on an external device.

The control unit 428 is a power interface manager that manages the supply of power from a power source (not shown) to the various components of the controller 428, e.g., the controller 428 integrated or attached to the systems 400, 500. Including 432. The power interface manager 432 may be coupled to the controller 442 and a power source. Alternatively, in some implementations, the power interface manager 432 may be integrated into the controller 442. The power source may include an external power source, among other things. The power interface manager 432 may include power converters, rectifiers, buses, gates, circuits, and the like. In particular, the power interface manager 432 may control, limit, or limit power supply from the power source based on various operating states of the systems 400 and 500.

In some embodiments or implementations, instructions and/or data stored on a non-transitory storage medium that can be used by a controller, such as, for example, ROM 440, RAM 438, and flash memory (not shown) , An application program interface (API) that provides programmatic access to one or more functions of the controller 428. For example, these APIs include, but are not limited to, one or more functions of the user interface 436, or processing of imaging data received from the imaging device or devices 422. It may include a program interface for controlling operation characteristics. This control can be invoked either by another program, another remote device or system (not shown), or some other module. In this way, the API facilitates the development of third-party software such as various and different user interfaces and control systems for different devices, plug-ins and adapters, etc. Can be facilitated.

In an exemplary embodiment or implementation, components or modules of controller 428 and other devices within systems 400 and 500 may be implemented using standard programming techniques. For example, logic for performing the functions of the various embodiments or implementations described herein may be "native" running on a controller, e.g., microprocessor 442, along with one or more static or dynamic libraries. It can be implemented as an executable instruction. In other embodiments, various functions of the controller 428 may be implemented as instructions processed by a virtual machine in which the instructions are executed as one or more programs stored in ROM 440 and/or RAM 438. In general, various programming languages known in the art may be employed to implement these exemplary embodiments, and object-oriented languages (eg, Java, C++, C#, Visual Basic.NET, Smalltalk, etc.), functional languages (E.g. ML, Lisp, Scheme, etc.), procedural language (e.g. C, Pascal, Ada, Modula, etc.), scripting language (e.g. Perl, Ruby, Python, JavaScript, VBScript, etc.) or It includes representative implementations of various programming language paradigms including, but not limited to, narrative languages (eg, SQL, Prolog).

In software or firmware implementations, instructions stored in memory configure one or more processors of control unit 428, such as microprocessor 422, to perform the functions of control unit 428 when executed. Instructions are received from one or more imaging device(s) 422 so that some other processor, such as a microprocessor 422 or an I/O controller/processor, provides functions and operations to construct a 3D model based on the imaging data. Process the information and let it act on it.

Further, the embodiments or implementations described above are well known, but other synchronous or asynchronous client-server computing techniques may be used. However, the various components may be implemented using, for example, more monolithic programming techniques as well as executable technologies running on a single microprocessor, or alternatively, each one or more central processing units. (central processing unit (CPU)) or multiprogramming, multithreading, client-server or peer-to-peer ( Various structuring known in the art, including but not limited to, for example, Bluetooth®, NFC, or RFID technology that provides a communication channel between devices within the system 400, 500, mesh networks, etc. It can be decomposed using the (structuring) technique. Some embodiments may be realized asynchronously at the same time and communicate using message passing techniques. Further, although different functions may be implemented and/or performed by different components/modules in different orders, the functions of the control unit 428 may still be achieved.

Further, programming interfaces for the data stored in the control unit 428 and the functions provided by it may include, for example, C, C++, C# and Java APIs; Libraries for accessing files, databases, or other data stores; Scripting language; Or it can be used by standard mechanisms, such as through a web server, FTP server, or any other kind of server that provides access to stored data. The data stored and utilized by the control unit 428 and the overall imaging system may be any of the above, including database systems, file systems, any other technology for storing such information, or implementations using distributed computing technology. It can be implemented as a combination.

Other configurations and locations of programs and data are contemplated for use with the techniques described herein. Various distributed computing technologies are suitable for implementing the components of the illustrated embodiments in a distributed manner, including but not limited to TCP/IP sockets, RPC, RMI, HTTP and web services (XML-RPC, JAX-RPC, SOAP, etc.) Do. Other variations are possible. In addition, different functions may be provided by each component/module, or existing functions may be distributed among components/modules in systems 400, 500 in different ways, but control unit 428 and imaging system 400 , 500) can still be achieved.

Further, in some embodiments, some or all of the components of the control unit 428 in the systems 400, 500 and the components of other devices may be implemented or provided in different ways, for example, one or more ASICs (application -specific integrated circuit), standard integrated circuit, controller (for example, by executing suitable instructions and including microcontrollers and/or embedded controllers), field-programmable gate array (FPGA), complex programmable logic device (CPLD) It may be implemented or provided at least partially in firmware and/or hardware including, but not limited to, etc. In addition, some or all of the system components and/or data structures may be provided in a computer-readable medium (eg, hard Disk; memory; computer network, cellular phone wireless network, or other data transmission medium; or portable media article read by a suitable drive such as a DVD or flash memory device or via a suitable connection) and/or one or more associated computer systems or devices Content on a computer-readable medium to enable the execution or otherwise use or provision of content (e.g., as executable or other machine-readable software instructions or structured data) to perform at least a portion of the described technology. Can be stored as.

4 and 5, the control unit 428 includes a plurality of imaging devices through a wiring 430 that may be located inside or outside with respect to the conveyor system 402, the support 418 and the support ring 414. In electrical communication with each of the 422 and support rings 414. Alternatively, the control unit 428 may be in wireless communication with the system 400 to wirelessly receive imaging data from the imaging device 422, as described above with reference to FIG. 5. Further, the control unit 428 may be coupled to the system or may be located external to the system. In one implementation, control unit 428 provides power to system 400 and also receives imaging data from imaging device 422. The control unit 428 may include at least one processor, such as in a standard computer, to process the imaging data, or alternatively, the control unit 428 may be an additional external processor not specifically illustrated for the sake of clarity. Alternatively, imaging data can be transferred to a computer.

6 shows a conveyor system 402, a frame or ring 504 coupled to the frame 504, an imaging device 510 coupled to and extending from the frame 504, and first and second light sources 512. , 414, an alternative and exemplary implementation of an imaging system 500 is shown. Certain features of an implementation of system 500 are similar or identical to those described above with reference to system 400, and thus, these features are not repeated for efficiency considerations.

In this embodiment, the frame 504 is coupled to the conveyor system 502 using supports 506, 508, and the support 506 has a base and frame 504 having a channel for receiving the frame 504. Is at least one collar 508 surrounding it. However, since this implementation utilizes a single imaging device 510, the system 500 provides imaging data of the object 516 from multiple perspectives, angles, or views to facilitate 3D reconstruction by the imaging device 510. It further includes a mechanism for rotating the frame 504 about the conveyor system 502 so as to be able to capture. For example, the base 506 may include a belt that rotates within the channel, and the belt rotates the frame 504 according to input received from an external control device, which may be a control unit 428 (see FIG. 4). Contact with the frame 504 to allow it to occur. However, other commercially available mechanisms for rotating the frame 504 may be specifically contemplated herein. Moreover, although it is preferred that the frame rotates automatically, it is possible that the rotation is based on manual rotation by manipulating the collar 508, the collar 508 being in a closed position preventing rotation and the frame 504 rotating. It is adjustable between open positions that can be used.

Thus, in this embodiment, the frame 504 rotates at least between a first position and a second position, and in the first position, the imaging device 510 is from the first light source 512 or the second light source 514. Capture a first set of imaging data, respectively, corresponding to transmission or interaction imaging data. Then, the frame 504 is rotated to a second position, and the capture process for the second imaging data is repeated. This process can take as many views from as many orientations as needed for a particular application (i.e., 3rd, 4th, 5th, 6th, or more views based on different positions relative to object 516). Can be repeated to produce. Moreover, in embodiments in which frame 504 is automatically rotated, rotation of frame 504 can be performed efficiently according to the position set during calibration of system 500, but with the use of fewer imaging devices 510. This reduces the cost of the system 500.

FIG. 7 shows an exemplary view of a system 600 that may be substantially similar or identical to systems 400, 500, wherein the system 600 includes an outer housing or cover 604, and the outer housing 604 ) Wall 612 is solid and opaque. The inlet portion or opening 610 of the housing 604 comprises a cover 608 comprising a strip of opaque material extending over at least 80% of each inlet portion 610 such that light cannot enter the housing 604. Includes. In addition, although not specifically illustrated, the support ring or frame 414, 504 may be connected to and supported by the housing 604, and the control unit 606 is coupled to the outer wall 612 of the housing 604. The control unit 606 may, in various implementations, provide power to the system 600, provide coordinates corresponding to the location of the rotating frame 504, or rotate the rotating frame 504. Or a processor for generating a 3D model based on the imaging data received from the system 600.

8 is a schematic diagram of a reconstruction method or system 700 utilized by a machine learning system or deep convolutional neural network to generate a 3D model 702 from one-dimensional (1D) imaging data and a two-dimensional (2D) silhouette.

In general, machine learning and convolutional neural networks (CNNs) can generally be implemented as a series of operational layers. One or more convolutional layers may be followed by one or more pooling layers, and optionally one or more pooling layers may be followed by one or more normalization layers. The convolutional layer creates a plurality of kernel maps, otherwise called filtered images, from a single unknown image. A large amount of data in the plurality of filtered images is reduced using one or more pooling layers, and the amount of data is further reduced by one or more rectified linear unit layers (ReLUs) that normalize the data. Preferably, implementations of the present disclosure rely on semantic segmentation, and the parameters for training CNNs are application dependent and need to be adjusted according to the complexity of the image data depending on the food to be inspected.

In other words, the kernel is chosen from known images. Not all kernels of known images need to be used for neural networks. Instead, a kernel that is judged to be an "important" feature may be selected. After the convolution process generates a kernel map (ie, feature image), the kernel map is passed through a pooling layer and a normalization layer (ie, ReLU). All values in the output map are averaged (i.e., added and divided), and the output value from the averaging calculation is used as a prediction as to whether an unknown image contains a specific feature found in a known image.

In an exemplary case, the output value is that the unknown image is the first portion of the object, such as dark flesh (i.e., the second portion) surrounded by the white flesh (i.e., the first portion) of a tuna lean, in one embodiment. It is used to predict whether it contains an important feature, which is the second part of the object that is located inside. Then, using this output, the CNN can generate a silhouette corresponding to the identified region of interest from the image.

In the illustrated system 700, a machine learning program or deep convolutional neural network will, as input, receive image data 704 from multiple views captured from systems 400, 500, each set of image data being It corresponds to a photograph of a tuna lean meat with a first portion 706 surrounding two portions 708. Although a camera or spectroscope may use line scans to acquire 1D data, the 1D data is combined into 2D image data before being used in the CNN to recover the silhouette, as described herein.

In embodiments where the scanned object is a tuna lean meat, the first portion 706 corresponds to a first outer layer of flesh having a first set of properties, and the second portion 708 corresponds to a second portion of flesh having a different second set of properties. Corresponding to the inner layer of 2, the second portion 708 is located within the first portion 706. As shown in Fig. 7, each image data set 704 has imaging data, preferably, a brighter pixel is assigned to the first portion 706, and a darker portion is assigned to the second portion 708. Corresponds to transmission imaging data that can represent an intensity value to be obtained. Thus, in the sense of a single line of a 2D image, or in other words, in the sense that each pixel or kernel analyzed by the CNN corresponds to an intensity value, the image data 704 is 1D.

In the highest level machine learning programs or deep convolutional neural networks, the CNN is based on a pool of thousands of representative sample images to identify the general appearance of the second part 708 (i.e. dark flesh in tuna lean meat). To be trained; For example, the second portion 708 passes parallel to its major axis through the center of the lean meat based on a large pool of reference images, which may contain thousands of images of tuna lean meat. At another level, the CNN will acquire information about edges, lines and curves based on representative sample images, and the accuracy of the CNN will improve as more images are scanned. Thus, based on the difference in the intensity value, the CNN will identify the portion of the picture that corresponds to the second portion 708. Once these portions are identified, the CNN will create a plurality of silhouettes 710, each silhouette corresponding to a second portion 708 identified in each view represented in 2D.

For example, a CNN is made up of many layers, and the layer between the input and the output is called a "hidden layer." Each layer has a number of neurons fully connected between the layers. This connection corresponds to the weights that are learned based on the reference image. A neuron or node is a computational unit that takes an input value, multiplies it with an associated weight, and passes it through an activation function (e.g., ReLU as described herein), passing the output value. This output is connected through another connection and then forms the input of the neuron. In addition, CNNs are used similarly to neurons, but may include other layers such as convolution, pooling, normalization and dropout, which have different functions.

Before the network is trained, the weights or connections between the nodes are randomly assigned. When training the network, labeled or annotated data is used. For example, input data (eg, image data) is correlated with predicted output data (eg, silhouette). By providing input data (e.g., image data) through an input node in the first layer and knowing the predicted value (e.g., labeled silhouette) of the output layer, adjusting the weight of the connection through several iterations. It is possible, so for any input data, the CNN returns the predicted output. This is basically an optimization process using a large number of parameters. Whenever one weight is changed, it will affect the entire network, so training a CNN may involve tens, hundreds or thousands of iterations.

The convolutional layer of the CNN reduces the size of the image, which determines the area that can be viewed or evaluated. For example, a small window of 9×9 pixels is moved over the entire image being analyzed. In that window, the observer will see a small portion of the entire object (eg, lines and corners). Since the size of the image is reduced but the window size remains the same, more object features are recognized. If the image is very small and fits almost within the window, the observer will see the whole lean meat as well as dark flesh in a single step.

This is an example of what the neural network refers to. In the initial layer of the network, weights will be learned that allow to detect the lines and corners involved in identifying dark flesh, for example in tuna fish lean. Until the entire dark flesh is recognized as an object and is related to its periphery as a reference (e.g., the dark flesh is between the boundaries of white flesh and passes along the main axis), in later layers this line will form a curve . These features are learned at the same time, but all are related. Without lines and corners, the boundaries of dark flesh are difficult to locate based on the overall appearance of dark flesh. Similarly, there are many lines and corners corresponding to dark flesh, but if the overall appearance of the dark flesh is not known, it is difficult to distinguish which lines and corners are involved. However, knowing both of these high and low level features makes it possible to detect dark flesh in image data.

The computational system then backprojects each silhouette 710 into a plurality of projections 712 by using an algorithm, and in one implementation, as shown in FIG. 7, the algorithm Lines corresponding to the outer borders are extended to a higher-dimensional scene. By backprojecting each silhouette 710 and analyzing the intersections between the projections 712, a 3D model 702 of the second portion 708 can be determined. In one embodiment, the back projection is based on a cone-shaped field of view. Imaging data corresponding to an interactive imaging mode such as the imaging mode 300 may be utilized to improve a model based on the depth of the object of interest. For example, if the properties of the object of interest are known based on the corresponding information database, the amount of scattered light to be captured in the interactive imaging mode will, in one embodiment, depend on the depth of the dark killing object of interest in tuna lean. Moreover, the interactive imaging data helps correct concavities in the surface of the scanned object as the captured light will vary with the depth of the object as above. Thus, if an object has a concave area, the interactive imaging data will be different for the portion of the object whose material has a thinner concave area as opposed to the portion of the object where the material does not have a thicker concave area (i.e., a lower value. The captured intensity of is corresponding to a thinner material because less light is scattered and captured, and a higher value of the captured intensity is more light because more light is scattered and captured when it cannot penetrate or penetrate thicker materials. Corresponds to thick materials).

9 is a flow chart illustrating an exemplary method 800 of generating a 3D model of an object of interest based on image data captured by an imaging system (eg, imaging systems 400, 500). Method 800 begins at 802, at 804 a conveyor is activated, and objects being scanned are loaded onto the conveyor. The activation of the conveyor can take place via an external switch or via a control unit or program. Similarly, at 806, a light transmitting system, which may be substantially similar to the transmissive imaging mode 200 described with reference to 200, is activated either manually through an external switch or through a control unit or processor. At 808, the imaging device that is part of the light transmission system determines whether transmission image data corresponding to the light passing through the object on the conveyor is received by the imaging device, either on its own or through a program associated with a control unit in electronic communication with the transmission system. And, in one embodiment, the imaging device is a spectroscope, a camera, or a sensor.

If image data is not received, the process returns to 806 and repeats until image data is received. When transmission image data is received, it is transmitted to the processor, and the method 800 proceeds to 810 where the transmission mode is deactivated and the interactive imaging mode is activated. The interactive imaging mode is substantially similar to the interactive mode 300 described with reference to FIG. 3. Again, at 812, it is determined whether the imaging device or spectroscope is receiving interactive imaging data. If not, the process repeats back to 810 until the imaging device receives the interactive imaging data. If the interactive imaging data is detected, at 814, the interactive imaging data is sent to the processor. In implementations with multiple imaging devices, the process described above may be repeated for each unique imaging device to create multiple views. Alternatively, in embodiments in which the imaging device is rotated, this process is repeated each time the imaging device is placed in a unique position to generate a plurality of views or a plurality of transmission imaging data and a plurality of interactive imaging data.

The processor includes a machine learning program or CNN, and at 816, the CNN receives as input transmitted image data corresponding to each view. Then, at 818, the CNN creates, in each set of transmission images, a plurality of silhouettes corresponding to the feature of interest, which in one embodiment is an object positioned within a piece of food or a second portion of the creation positioned within a first portion of a fish. Generate. At 818, each of the silhouettes is backprojected into a plurality of projections, and a common intersection portion of each projection is analyzed to determine a 3D structure based on the intersection portion. Then, at 822, the process outputs this 3D structure and, either manually or through an additional processor step, is determined whether the object of interest is near the surface of the scanned object. If not nearby, a 3D structure is output based on the transmission imaging data, and at 828 the method 800 ends.

If the object of interest is near the surface, the process continues to 824, and the CNN uses the interactive imaging data to correct or clarify the 3D structure based on the transmission imaging data. If the 3D structure is corrected at 824, the processor or CNN outputs the corrected 3D structure at 826, and the process ends at 828.

10 is an alternative and exemplary implementation of a method 900 for generating a 3D model of an object of interest based on image data captured by an imaging system. The method 900 begins at 902 and the conveyor and light transmission system 904 is activated. At 906, the imaging device or spectroscope determines whether imaging data corresponding to the transmission imaging data is received or captured by the imaging device. If not received or captured, the processor returns to 904 until data is received. If received or captured, the method 900 continues to 908, where transmission imaging data is transmitted to the processor. Then, at 910, the processor determines a plurality of 2D silhouettes from the 1D transmission image data via a convolutional neural network. At 912, each silhouette is backprojected and the intersection is analyzed. Then, at 914, the processor outputs a 3D structure based on the common intersection between each projection.

Then, at 916, a processor or CNN is used to determine whether the object of interest is near the surface of the scanned object. If not, at 926 the method 900 ends, and the 3D model is based on the transmission imaging data. If so, at step 918, the method 900 continues to activate the interactive imaging system, and at 920, the imaging device determines whether imaging data corresponding to the interactive imaging data is received by the imaging device or spectroscope. If not, the method 900 returns to 918 until such data is received or captured. If received, the method 900 proceeds to 922 where the interactive imaging data is transmitted to the processor, and if necessary, the 3D is corrected based on the interactive imaging data. Finally, the corrected 3D model is output at 924, and the method 900 ends at 926.

Through the acquisition of three-dimensional information, the present disclosure makes it possible to more accurately determine the volume and shape of internal defects, which may affect the quality control inspection as to whether or not a particular food material needs to be defective. . In addition, by knowing the three-dimensional structure of the object, processing and removal of the secondary product can be performed more accurately, and thus the loss of the primary product can be minimized.

The foregoing description of the illustrated embodiments, including those described in the Abstract, is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Although specific implementations of the examples are used for illustrative purposes herein, various equivalent modifications may be made without departing from the spirit and scope of the present disclosure, as recognized by those skilled in the relevant art. The teachings disclosed herein for various embodiments may apply beyond the context of food processing, not necessarily the exemplary imaging systems and methods described above.

For example, the foregoing detailed description has described various implementations of devices and/or processes through block diagrams, schematic diagrams, and examples. As long as such block diagrams, schematics and examples contain one or more functions and/or operations, each function and/or operation within such block diagrams, flow charts or examples may be applicable to a wide range of hardware, software, firmware, or any combination thereof. It will be understood by one of ordinary skill in the art that it may be implemented individually and/or collectively by. In one implementation, the present disclosure may be implemented through an application specific integrated circuit (ASIC). However, one of ordinary skill in the art would understand that the implementations disclosed herein are, in whole or in part, as one or more computer programs executed by one or more computers (e.g., as one or more programs executed on one or more computer systems), One or more programs executed by one or more controllers (e.g., microcontrollers), as one or more programs executed by one or more processors (e.g., microprocessors), as firmware, or Any combination may be implemented equally in a standard integrated circuit, and designing the circuit and/or writing code for software and/or firmware is the skill of a person skilled in the art in light of the teachings of this disclosure. You will recognize that you will be within.

When logic is implemented as software and stored in memory, the logic or information may be stored on a computer-readable medium for use by or in connection with any processor-related system or method. In connection with the present disclosure, a memory is a computer-readable medium that is an electronic, magnetic, optical or other physical device or means containing or storing a computer and/or processor program. Logic and/or information is an instruction execution system, apparatus, or device, such as a computer-based system, or a system containing a processor, or any other capable of fetching instructions from an instruction execution system, apparatus, or device to execute instructions associated with the logic and/or information. It may be embodied in any computer-readable medium for use by or in connection with the system.

In connection with this specification, a “computer-readable medium” may be any element capable of storing a program associated with logic and/or information for use by or in connection with an instruction execution system, apparatus, and/or device. The computer-readable medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device, but is not limited thereto. More specific examples (non-exhaustive list) of computer-readable media may include: portable computer diskettes (magnetic, compact flash card (CFC), secure digital (SD), etc.), random access memory (RAM), Read-only memory (ROM), removable programmable read-only memory (EPROM, EEPROM, or flash memory), portable compact disc read-only memory (CDROM), digital tape, and other non-transitory media.

Many of the methods described herein can be performed in variations. For example, many methods may include additional operations, some operations may be omitted, and/or may perform operations in a different order than illustrated or described.

The various implementations described above can be combined to provide other implementations. Unless contradicted by the specific teachings and definitions herein, U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications and non- Patent publications are incorporated herein by reference in their entirety. Aspects of the implementations may be modified as necessary to employ the concepts of various patents, applications, and publications to provide yet another implementation.

These and other modifications may be made to the implementation in light of the foregoing detailed description. In general, in the claims that follow, the terms used should not be construed as limiting the claims to the specification and the specific embodiments disclosed in the claims, including all possible embodiments, along with the full scope of equivalents entitled to such claims. It should be interpreted as doing. Accordingly, the claims are not limited by the present disclosure.

Claims (17)

A first conveyor;
A second conveyor separated from the first conveyor by a gap;
A transparent plate positioned within the gap and coupled to at least one of the first conveyor and the second conveyor;
A support ring located at least partially within the gap and coupled to at least one of the first conveyor and the second conveyor;
At least one imaging device coupled to the support ring;
A first light source coupled to the support ring; And
A control unit in electronic communication with the support ring and the at least one imaging device
Including,
During operation, the first light source emits light directed towards an object on the transparent plate, and the control unit receives imaging data from the at least one imaging device, and the control unit is within a first portion of the object. A system for constructing a 3D model of the second portion of the contained object. The method of claim 1,
Wherein the at least one imaging device transmits imaging data to the control unit, the imaging data comprising interaction imaging data and transmittance imaging data. The method of claim 1,
Wherein the object is a tuna lean meat, and the first light source emits light at a wavelength of one of approximately 1260 nanometers, approximately 805 nanometers or approximately 770 nanometers. The method of claim 1,
The system, wherein the processor uses machine learning in the form of a convolutional neural network to process the imaging data. The method of claim 4,
The convolutional neural network receives the image data and outputs a plurality of silhouettes based on the image data corresponding to the second part of the object, and the processor projects the silhouette into a plurality of projections, and the A system for constructing the 3D model by analyzing an intersection between a plurality of projections. The method of claim 1,
Wherein the support ring includes a plurality of cameras coupled to the support ring, each of the plurality of cameras capturing one of transmission imaging data, cross-cross imaging data or reflective imaging data from the first light source. The method of claim 6,
Wherein the support ring includes a second light source coupled to the support ring, and during operation, the second light source emits light directed to the transparent plate. A conveyor, the conveyor having a space between the first portion and the second portion of the conveyor;
A plate located in the space and coupled to the conveyor;
A support ring located at least partially within the space and coupled to the conveyor;
At least one light source coupled to the support ring;
An imaging device coupled to the support ring; And
Processor in electronic communication with the imaging device
Including,
During operation, the support ring rotates between at least a first position and a second position, the at least one light source emits light directed towards the object on the plate, and the imaging device is Receive light after the light passes through the object,
The processor is configured to receive a first set of image data from the imaging device when the support ring is in the first position and a second set of image data from the imaging device when the support ring is in the second position. Receiving and outputting a three-dimensional (3D) model of an interior portion of the object from the first image data set and the second image data set. The method of claim 8.
The processor uses machine learning to process the first image data set and the second image data set into a plurality of silhouettes and project the plurality of silhouettes into a plurality of projections, and the 3D model Based on the intersection between each of the projections of. The method of claim 8.
A second light source coupled to the support ring, wherein the imaging device captures a third set of image data from the second light source when the support ring is in the first position or the second position, and the processor Utilizes the third image data set to demarcate the three-dimensional model. The method of claim 8.
Wherein the imaging device is a spectrograph, and the at least one light source emits light of a wavelength selected from one of approximately 1260 nanometers, approximately 805 nanometers or approximately 770 nanometers. Emitting light from a light source, the emitting step comprising directing the light through an object having a first portion and a second portion, the second portion enclosed within the first portion;
Capturing light from the light source with an imaging device after the light passes through the object, the captured light corresponding to image data of the first portion and the second portion received by the imaging device;
Transmitting the image data to a processor; And
Analyzing the image data using the processor to detect a boundary between the first portion and the second portion, wherein the analyzing step involves machine learning to generate a three-dimensional representation of the second portion. Including the steps to utilize-
Including, the method. The method of claim 12,
Emitting light from the light source comprises emitting light at a wavelength selected from one of approximately 1260 nanometers, approximately 805 nanometers or approximately 770 nanometers. The method of claim 12,
The step of utilizing machine learning to generate a three-dimensional representation of the second part comprises machine learning utilizing a deep convolutional neural network to process the image data. The method of claim 12,
Analyzing the image data using the processor comprises utilizing machine learning to output a plurality of two-dimensional silhouettes corresponding to the image data of the second portion. The method of claim 15,
Analyzing the image data using the processor includes utilizing machine learning to generate a plurality of projections, each projection corresponding to each one of the plurality of two-dimensional silhouettes How to. The method of claim 15,
The analyzing step includes the step of including machine learning to generate a three-dimensional representation, and an intersection between each of the plurality of projections to output a three-dimensional representation of the second portion of the object. ), the method further comprising the step of analyzing.

Images