JP7324271B2 - Multi-view imaging system and method for non-invasive inspection in food processing - Google Patents

Description

TECHNICAL FIELD This disclosure relates to non-invasive inspection in food processing, and more particularly, to identifying internal objects within food by processing image data of the food to determine a three-dimensional model of the internal objects. Imaging systems and related methods for detection.

The food industry operates on narrow profit margins and is subject to improved quality control standards. As such, the food processing industry is moving to automated systems to improve processing performance while meeting higher quality control standards. Aspects of food processing include, among other things, the separation of the primary product from the secondary product and the removal of foreign matter in order to increase the added value to the product. However, these aspects are difficult to automate because the boundaries between primary and secondary products and foreign matter are difficult to discern in current systems.

In instances where there is a physical boundary between food ingredients, such as a solid-liquid interface, separation of the primary product from the secondary product is often simple and requires little physical effort. In instances where the food material is interconnected, ie there is a solid-solid interface, active physical intervention such as cutting is usually required. To guide and perform such actions, it is advantageous to identify the exact or best possible boundaries between solid-solid interfaces, which can be accomplished through visual systems in an automated environment. many. Even where separation between materials is not required, the ability to detect the presence as well as the extent of certain faulty or undesirable objects can provide significant advantages in food inspection or sorting processes.

For example, disclosed in US Pat. No. 5,352,153, US Pat. No. 6,563,904, US Pat. No. 9,551,615, US Pat. As such, several systems have been developed for various industries including fish, meat, poultry, fruit, vegetables and grains. Conventional systems generally apply conventional imaging for rapid screening of food ingredients to obtain information regarding processing. However, information derived through such approaches is generally limited to only the visual surface of the material.

Alternative techniques have been proposed to provide details of the interior of the object. Information is generally provided in a two-dimensional format, regardless of whether the data is captured from the surface or interior of the object. This may be sufficient for some applications, either because depth perception is particularly irrelevant, or because the geometry of the object is so consistent that certain assumptions can be made. However, there are scenarios in which additional information in the third dimension is particularly important. For example, information in the third dimension can be used to derive accurate information about the placement of objects or the geometry of irregularly shaped objects to enable accurate separation or removal of irregularly shaped objects. Useful for handling dependencies.

Some of the adapted solutions towards recovery of three-dimensional (“3D”) surface data are limited to detailed description of the external profile. Others are based on volumetric imaging techniques, such as computed tomography (“CT”) or magnetic resonance imaging (“MRI”), which provide data regarding the internal composition of the scanned object, although such technical solutions have several limitations. For example, volumetric imaging techniques lack speed, which is a particularly significant limitation given the narrow profit margins of the food processing industry. The lack of speed, among other limitations, makes the current system more suitable as an irregular quality control inspection tool rather than as an in-line solution for automated food processing and sorting.

The present disclosure is directed to rapid data acquisition and reconstruction for in-line industrial food processing applications that enable capture of geometric details of food material internal components. Such systems are particularly useful in applications where complete representation of internal structures is not necessary, such as those provided by volumetric imaging techniques, but rather coarse internal profile reconstruction and speed are essential. In particular, the systems and methods disclosed herein are directed to surface-to-surface combinations of two different materials, the first material forming an outer layer that allows partial penetration of any light spectrum, and in particular the A second material, the inner body, which is the object, is at least partially surrounded by the outer layer, allowing different ranges of penetration or absorption by any light spectrum. In general, exemplary embodiments of the present disclosure involve use of an imaging system that includes a light source and an imaging device that captures image data, and computational steps for reconstructing the data to determine boundaries of inner objects.

An exemplary embodiment of a system for capturing and processing image data of an object to determine the boundaries of interior portions of the object comprises a first conveyor and a second conveyor separated from the first conveyor by a gap. a conveyor, a transparent plate positioned in the gap and coupled to at least one of the first conveyor and the second conveyor, and a transparent plate positioned at least partially in the gap and at least the first conveyor and the second conveyor a support ring coupled to one side, the support ring including at least one camera coupled to the support ring; and a first light source coupled to the support ring, directed toward the transparent plate during operation. and a first light source that emits attached light.

An embodiment is an object positioned on a transparent plate, the camera receiving light passing through the object from a first light source during operation, the object being a fillet of tuna, and the first light source being about 1260 a controller in electronic communication with an object that emits light at a wavelength equal to one of nanometers, about 805 nanometers, or about 770 nanometers, the camera capturing light passing through the object; sending a signal corresponding to image data from the captured light to a controller, the image data being one of transmittance image data, interactance image data or reflectance image data, the controller including a processor; a controller using machine learning to detect a boundary between a first portion of the object and a second portion of the object within the first portion based on image data received from the camera; can further include

Embodiments include a processor passing image data through a deep convolutional neural network, the deep convolutional neural network receiving the image data and outputting a plurality of silhouettes based on the image data corresponding to the second portion of the object. , a processor projects the silhouette into a plurality of projections, analyzes intersections between the plurality of projections to determine a three-dimensional shape of a second portion of the object, and a support ring comprises a plurality of cameras coupled to the support ring. each of the plurality of cameras captures one of transmittance, interactance or reflectance imaging data from a first light source; the support ring includes a second light source coupled to the support ring; The two light sources may further include emitting light directed to the transparent plate during operation.

An alternative exemplary embodiment of a device for capturing and processing image data of an object to determine the boundaries of interior portions of the object has a space between the first portion and the second portion of the conveyor. a conveyor, a plate positioned in the space and coupled to the conveyor, and a support ring positioned at least partially in the space and coupled to the conveyor, wherein during operation, at least a first position and a second position; a support ring rotating between and at least one light source coupled to the support ring for emitting light directed toward an object on the plate during operation and coupled to the support ring and a processor in electronic communication with the imaging device, wherein the support ring extends from the imaging device to the first position. receiving a first set of image data when the support ring is in the second position and a second set of image data when the support ring is in the second position from the imaging device; a processor that outputs a three-dimensional model of the interior portion of the object from the two sets of image data.

Embodiments include a processor utilizing machine learning to process a first set of image data and a second set of image data into a plurality of silhouettes, project the plurality of silhouettes onto a plurality of projections, and generate a three-dimensional model. is based on the intersection between each of the plurality of projections and may further include a second light source coupled to the support ring, wherein the imaging device is configured to move the support ring from the second light source to the first or second projections. capturing a third set of image data when in position, the processor utilizing the third set of image data to demarcate the three-dimensional model, the imaging device comprising a spectrograph, and at least one light source; emits light at a wavelength selected from about 1260 nanometers, about 805 nanometers or about 770 nanometers.

An exemplary embodiment of a method for capturing and processing image data of an object to determine boundaries of interior portions of the object comprises emitting light from a light source, the step of emitting light from a first directing light through an object having a portion and a second portion, the second portion enclosed within the first portion; and directing light from a light source by an imaging device after the light passes through the object. wherein the captured light corresponds to image data of the first and second portions received by the imaging device; transmitting the image data to a processor; analyzing the image data by a processor to detect a boundary between the first portion and the second portion, the analyzing step utilizing machine learning to generate a three-dimensional representation of the second portion; and a step comprising:

Embodiments include emitting light from the light source at a wavelength selected from about 1260 nanometers, 805 nanometers, or 770 nanometers, wherein the three-dimensional representation of the second portion is The step of utilizing machine learning to generate includes the machine learning utilizing a deep convolutional neural network to process the image data, and the step of analyzing the image data with a processor comprises: utilizing machine learning to output a plurality of two-dimensional silhouettes corresponding to the data, wherein analyzing the image data with a processor comprises utilizing machine learning to create a plurality of projections; Each projection corresponds to a respective one of the plurality of two-dimensional silhouettes, the analyzing step including utilizing machine learning to generate a three-dimensional representation, and outputting a three-dimensional representation of the second portion of the object. The method may further include analyzing intersections between each of the plurality of projections to do so.

For a better understanding of the embodiments, reference is now made, by way of example only, to the accompanying drawings. In the drawings, identical numbers identify similar elements or acts. Sizes and relative positions in the drawings are not necessarily drawn to scale. For example, the shapes of various elements and angles are not necessarily drawn to scale, and some of these elements may be enlarged and positioned to improve drawing legibility. Additionally, the particular shapes of the elements shown are not necessarily intended to convey information regarding the actual shape of the particular elements, and may be chosen merely for ease of understanding the drawings.

1 is a perspective view of an exemplary embodiment of a conveyor belt system according to the present disclosure having a gap between a first conveyor and a second conveyor of the system; FIG. FIG. 2 is a schematic representation of an exemplary embodiment of an imaging system according to the present disclosure showing a transmission imaging mode; FIG. 3 is a schematic representation of an exemplary embodiment of an imaging system according to the present disclosure showing an interactance imaging mode; FIG. 4 is a perspective view of an exemplary embodiment of an imaging system according to the present disclosure having a support ring and multiple imaging devices and light sources coupled to the support ring; 5 is a schematic representation of the controls of the imaging system of FIG. 4; FIG. FIG. 6 is a perspective view of an alternative exemplary embodiment of an imaging system according to the present disclosure having a support ring and one imaging device and light source coupled to the support ring, the support ring in at least a first position and a first position; Rotate between two positions. FIG. 7 is a perspective view of an exemplary embodiment of an outer housing according to the present disclosure for reducing ambient light to an imaging system within the outer housing; FIG. 8 is a schematic representation of an exemplary embodiment of reconstruction of a three-dimensional model from projected silhouettes according to the present disclosure; FIG. 9 is a flow diagram of an exemplary embodiment of a method according to the present disclosure for capturing and processing image data of an object to determine boundaries of interior portions of the object. FIG. 10 is a flow diagram of an alternative exemplary embodiment of a method according to the present disclosure for capturing and processing image data of an object to determine boundaries of interior portions of the object.

In the following description, certain specific details are set forth in order to provide a thorough understanding of the various disclosed embodiments. One skilled in the art will recognize, however, that the embodiments can be practiced without one or more of these specific details, or with other methods, components, materials, and so on. In other instances, well-known structures associated with imaging systems, non-invasive inspection in food processing, machine learning and neural networks have been described in detail in order to avoid unnecessarily obscuring the description of the embodiments. It is neither shown nor described.

Unless the context requires otherwise, throughout the specification and claims that follow, the word "comprise" and variations thereof such as "comprises" and "comprising" are used in an open and inclusive sense, i.e., "including shall be construed as "not limited to Furthermore, the terms "first", "second" and similar designations for sequences should be interpreted as indistinguishable unless the context clearly indicates otherwise.

References to "one embodiment" or "an embodiment" throughout this specification mean that at least one embodiment includes the particular configuration, structure, or feature described in connection with the embodiment. Thus, the appearances of the phrases "in (in) one embodiment" or "in (in) an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the specific configurations, structures or features may be combined in any suitable combination in one or more embodiments.

As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. It should also be noted that the term "or" is generally used broadly, ie, in the sense of "and/or" unless the context clearly indicates otherwise.

The related terms “about” and “substantially”, when used to describe a value, quantity, quantity or dimension, unless the context clearly indicates Generally indicates a value, amount, quantity or dimension that is within /minus 3%. Further, it should be understood that any specific dimensions of components provided herein are for illustration purposes only, with reference to the exemplary embodiments described herein, and as a result of the present disclosure. includes amounts greater than or less than the stated dimension, unless the context clearly dictates otherwise.

The present disclosure provides a solution for a fast, non-invasive imaging system capable of acquiring visual data of food ingredients for processing and inspection. In particular, embodiments of the present disclosure capture the three-dimensional geometry of arbitrarily shaped objects surrounded by different layers of material. For example, in embodiments in which the object to be analyzed is a fish fillet, the imaging systems and methods disclosed herein provide a three-dimensional geometry of portions of dark meat (e.g., layers of different materials) contained within outer layers of white meat. Dark meat has an arbitrary three-dimensional shape that changes between successive fillets.

Embodiments of the present disclosure include various systems, devices and related methods that can exploit the different optical properties of absorption, reflection, transmission and scattering of different turbid materials and spectral bands. The relative proportions and amounts of each occurrence may depend on the chemical composition and physical parameters of the material. When light interacts with matter, a certain portion of the photons are reflected either through specularity, diffusion or backscattering. The first two, specularity and diffusion, depend on surface roughness, while scattering or backscattering of light is the result of phase changes within the material or multiple refractions at different interfaces. Scattering can also appear due to inhomogeneities such as randomly distributed pores or capillaries throughout the material, and different particle sizes, shapes and microstructures.

The remaining photons that are not reflected are either absorbed or transmitted through the material. The lower the absorption coefficient, the deeper the light can penetrate into the material before the photons are absorbed, and the more likely the light will exit the opposite side of the material. Thus, for non-invasive imaging by various embodiments of the systems, devices and related methods described herein, both scattered light and light passing through materials provide useful information of internal properties. be able to. Such information, e.g., scattered light and light passing through materials, is captured through interaction imaging or transmission imaging, respectively, while reflectance imaging is described herein with reference to exemplary embodiments. We are primarily interested in the light reflected directly from the surface, as described in .

Each wavelength of light results in different optical properties and interaction with turbid materials. Some wavelengths are readily absorbed, while others can penetrate deeply into the material and depending on thickness can be completely or partially transmitted. As described in more detail below, some of the embodiments of systems, devices and related methods investigate the phenomenon of optical properties for different wavelengths of the ultraviolet (UV), visible and near-infrared (NIR) light spectrum. may include multi- or hyperspectral imaging tools for

Additionally, certain embodiments described herein may include or utilize diffraction gratings, where the light is dispersed and the intensity of each wavelength is captured by various embodiments of the sensors described herein. Still further, embodiments of the systems, devices and related methods described herein may include or comprise a three-dimensional cube having spatial data stored in two dimensions and spectral data stored in a third dimension. can be obtained. The selection of suitable wavelengths individually or in combination can vary depending on the processed food material and can be predetermined. For example, the selection of appropriate wavelengths may be selected based on a database containing transmittance and reflectance information for the particular food item to be scanned and a database having spectral characteristics of certain wavelengths of light, or may be processed. Previously, the system may be calibrated to determine the wavelengths corresponding to capturing the appropriate imaging data based on the particular object or food being scanned.

In various embodiments of the systems, devices and methods described herein, materials of low absorption coefficient and minimal scattering while distinguishing from the inner object of interest (e.g., white meat and dark meat of fresh tuna fillets, respectively) A wavelength is selected that has good penetration through the outer layer of the . Further, in various embodiments of the systems, devices and related methods described herein, in addition to selecting appropriate wavelengths suitable for food material inspection, an appropriate illumination source may also be selected as a system design variable. For example, light sources may differ in emission intensity for a particular spectrum. Therefore, a light source of suitable wavelength for the application can be selected to achieve optimum results. Several types of illumination sources are disclosed herein including wavelength specific halogen, LED and laser sources.

Obtaining high quality image data of inspected food ingredients is one aspect of the present disclosure. As noted above, although the materials being processed are three-dimensional, conventional imaging sensors tend to have no means of capturing the depth of the scene, thereby increasing their ability to perceive the complexity of real-world objects. is limited. To construct a three-dimensional representation of an object, various embodiments of the systems, devices and related methods described herein can capture a collection of images that convey information from multiple views. Unlike full 3D representations obtained through volumetric imaging techniques, 3D surface reconstruction techniques of conventional imaging systems rely on the 3D coordinates of individual points located on the surface of an object, or in this case the boundary between two materials of the object. tend to incorporate only Accordingly, such methods are often referred to as surface measurements, range sensing, depth mapping or surface scanning, which may be used interchangeably herein unless the context clearly indicates otherwise.

One conventional technique directed to 3D surface reconstruction is multi-view stereo ("MVS"), which matches corresponding feature points in images given sufficient overlap between views. . The result is a 3D point cloud to which surfaces can be matched. Such approaches generally require feature-rich textures, which may not be available or present in certain types of foods. Additionally, MVS lacks effectiveness for certain applications in the food processing industry due to the low margins involved.

Structured light 3D surface reconstruction techniques are another technique for obtaining three-dimensional information. Structured light methods use spatially-modifying 1D or 2D structured lighting that is projected onto the object. In a flat scene the illumination pattern is projected onto the surface as well, but in a non-flat scene the pattern as seen by the camera is distorted. 3D information is extracted from distorted pattern features, which are usually obtained from the direct reflection of light at surface boundaries.

Both approaches can provide detailed descriptions of 3D surfaces, but neither of them are practical for non-invasive imaging data acquired through either interactance or transmittance modes. On the other hand, the shape by silhouette depends on the contour of the object, so that the three-dimensional representation can be reconstructed. As described in more detail below, various embodiments of the systems, devices and associated methods backproject the silhouettes seen by each camera into a three-dimensional scene and extract 3D shapes from their intersection. can operate as Concave shapes are generally ignored as they are not visible in silhouette, so the reconstruction is only an approximation of the true 3D geometry commonly known as a visual hull. However, according to various embodiments described herein, using interaction imaging in conjunction with transmission imaging, the exact 3D shape of the object can be extracted by considering the concave shape.

As described in more detail below, an aspect in the context of 3D shape reconstruction and acquisition of multi-view image data is camera positioning. In order to project the silhouette back into the scene, it is important to determine where the camera was positioned and how the scene was originally projected onto the screen plane. Various systems, devices and related methods described herein are operable to receive and/or process this information through camera calibration. Such information can include intrinsic parameters, which describe how light is projected through the lens onto the imaging sensor, as well as extrinsic parameters that describe the position of the camera in the real world, and any parameters that occur during this process. related to the distortion of The systems, devices and associated methods described herein are capable of performing a calibration process including the above intrinsic parameters for each camera and position that contributes to the system's multi-view image data acquisition, effectively through the use of binary fiducial markers. can be implemented in

In some embodiments, when acquiring image data, the silhouette of the target object may be determined before the three-dimensional model is generated. For example, aspects of the present disclosure may include recognizing what indicates food ingredients in the image data and distinguishing between different components forming the outer and inner layers. For example, in embodiments in which the object is a fish fillet, the outer layer or first portion corresponds to a first type of meat and the inner layer or second portion is typically enclosed within the first type of meat. Corresponds to the second type of body. In certain aspects, machine learning, and more specifically artificial neural networks, may be implemented in performing such tasks.

For example, a neural network has several nodes in different layers that are connected through weight-related connections. These weights are typically adjusted and learned through several iterations by specifying what the expected output of a node is given given known inputs. By collecting large datasets of labeled images showing food ingredients and internal defects or objects of interest, deep convolutional neural networks are trained to learn how to recognize the location and boundaries of specific objects. It is possible to learn With increasing training data sets and careful design of neural network architectures, even complex tasks can be efficiently solved using the various systems, devices and related methods described herein. Moreover, in certain embodiments, a variety of different models are possible for different specific applications.

As described herein, the present disclosure includes a conveyor belt system for detection and extraction of three-dimensional geometric shapes of inspection defects or processed products. Due to the continuously moving belt, data acquisition and analysis is preferably very efficient. In certain embodiments, the wavelengths of the applicable light spectrum can be predetermined based on the food material and application objectives. Specific spectral bands can be obtained either through hyperspectral imaging or through the use of specific filters or laser light involved in a line scan system.

As described in more detail below, to obtain transmittance data, the light source is positioned opposite the imaging sensor, which is a conveyor bridged by a transparent medium that allows the light to pass through and through the food material. Requires a slight gap in the belt. Another light source is positioned in parallel next to the imaging sensor for interactance imaging mode. Both imaging modes are alternated frequently to avoid obscuring the image data captured by each mode.

As described in more detail below, in the present disclosure, the combination of multiple light sources and camera sensors (which may be components of an imaging device, as described in more detail herein, or generally , an imaging device, or in some embodiments a separate component coupled to an imaging device), mounted on or about a conveyor belt to collect image data from multiple views. Alternatively, in some embodiments, a single camera sensor or imaging device may be used instead. However, in such embodiments, one camera may be mounted on a rotating frame that allows repositioning of the light and camera around the conveyor belt. At high conveyor belt speeds or reduced acquisition times due to repositioning of the camera system, image data is acquired in a helical arrangement. As such, helical image data is interpolated between acquisition points along a traverse path. The number of views can vary depending on the application and required detail of the target object. The structure is built around the imaging device and blocks any light from outside in order to control the illumination and achieve good image quality during imaging.

The present disclosure, in one embodiment, uses a deep convolutional neural network trained to detect the position and boundaries of target objects. However, according to the summary of this disclosure, it should be understood that the applications of deep convolutional neural networks are application dependent and can vary based on the application, such a model accomplishing this task can be pre-learned for As described in more detail below, the extracted silhouettes are used to generate a rough approximation of the three-dimensional shape of the target object. As should be appreciated, according to the summary of this disclosure, the resolution of the reconstructed model is a trade-off between speed and detail required for the intended application. For example, in some embodiments, higher resolution may require more images and more computational resources for reconstruction, which likewise affects the speed of the application.

In other embodiments, the placement and number of cameras may vary as described herein. Position and camera parameters may be calibrated prior to capturing image data. Due to movement of the food material on the conveyor belt, the transverse position of the camera changes with respect to the conveyed material. The traverse position can be maintained internally or set on the material or clearly defined through markers on the conveyor belt if required by the application.

In the exemplary embodiment described herein, FIG. 1 is a perspective view of conveyor system 100 . It should be appreciated that the conveyor system 100 has been simplified to facilitate understanding of the example embodiments of the present disclosure, and as such, specific configurations associated with the conveyor system 100 have not been described. Conveyor system 100 includes a first conveyor 102 spaced apart from a second conveyor 104 . In other words, a space or gap 114 separates the first conveyor 102 from the second conveyor 104 . First conveyor 102 may generally be referred to herein as a first portion of conveyor system 100 , and likewise second conveyor 104 may generally be referred to herein as a second portion of conveyor system 100 . Further, the size and shape of the space or gap 114 and the plate 110 can vary according to specific applications, whereby the present disclosure describes the distance between the conveyors 102 and 104 corresponding to the space or gap 114 as well: It is also not limited by the size or shape of plate 110 .

Plates 110 are positioned at or near gaps 114 to form a continuous conveyor line. Preferably, plate 110 is transparent so that light can pass through plate 110 without obstruction. For example, the plate 110 may be formed from a transparent plastic, polymer or glass, among others, while in alternative embodiments the plate 110 is translucent, as well as for example a translucent plastic, polymer or glass. Formed from glass. Plate 110 is coupled to conveyor system 100 , or more specifically to at least one of first conveyor 102 and second conveyor 104 . Additionally, each conveyor 102, 104 of system 100 is supported by a support structure 106, with conveyor surface 112 translated via rollers 108 and a conventional drive mechanism (not shown). Moreover, the conveyor surface 112 may be solid or may include a plurality of perforations 116 either arranged in a line as shown in FIG. 1 or evenly distributed across the conveyor surface 112. In other embodiments, conveyor surface 112 being solid means that surface 112 is free of such perforations.

FIG. 2 is a schematic representation corresponding to transmission imaging mode 200 . Transmission imaging mode 200 includes light source 202 and imaging device 204 . FIG. 2 further shows an object 206 to be scanned, which is positioned on plate 110 . Light source 202 emits light 208 that is directed toward plate 110 and propagates outward as it travels toward plate 110 . As light 208 passes through plate 110 and object 206 , the light converges as indicated by convergence path or portion 210 . As light 208 exits object 206 , light 208 diverges or disperses as indicated by diverging path or portion 212 . Light 208 is captured by imaging device 204 after exiting object 206 . As described herein, the imaging device 204 receives transmission image data corresponding to the captured light 208 transmitted through the object 206, which is sent to the imaging device for further processing and reconstruction. 204 is subsequently transmitted via a signal to a processor or controller (eg, controller 428 shown in FIG. 5) that is in electronic communication with 204 .

In the illustrated embodiment, conveyor system 100 generally translates object 206 from right to left relative to the orientation shown in FIG. I am letting Further, imaging device 204 and light source 202 are preferably arranged along a vertical axis, and imaging device 204 is positioned such that light 208 output by light source 202 propagates through object 206 in a straight line toward the imaging device. It is above the light source 202 . Due to slight variations in the placement or characteristics of the object 206 and light 208, the placement of the imaging device 204 and light source 202 may not be exactly vertical, but rather within 10 degrees of vertical, and within 5 degrees of vertical. or approximately vertical (ie, within 3 degrees of vertical).

Light source 202 is, in various embodiments, for example, a laser, a light emitting diode (“LED”), an array or panel of LEDs, an incandescent lamp, a compact fluorescent lamp, a halogen lamp, a metal halide lamp, a fluorescent tube, a neon lamp, a low pressure sodium lamp. or from a variety of light sources such as high intensity discharge lamps. In embodiments where light source 202 is a laser, embodiments of the present disclosure further include light source 202 comprising a solid-state, gas, excimer, die or semiconductor laser. The laser may be a continuous wave, single pulse, single pulse q-switched, repetitive pulse or mode-locked laser, as long as the laser is characterized by the duration of the laser emission.

Further, as discussed above, the light source 202 is preferably selected specifically for the application or object 206 being scanned, as different light sources 202 output light 208 with different penetration characteristics to the object 206 . In embodiments in which object 206 is a fish fillet, light source 202 preferably outputs light in transmission imaging mode 200 at wavelengths between 790 and 820 nanometers (“nm”), but more preferably at wavelengths 805 nm or about 805 nm (ie, 800-810 nm), which corresponds to wavelengths in the infrared portion of the electromagnetic spectrum that is generally outside the visible portion of the spectrum between about 400-750 nm. Moreover, this wavelength corresponds, at least for tuna fillets, to a wavelength of light that can penetrate deep into the fillet while minimizing scattering, which generally determines the accuracy of the image data corresponding to the transmittance imaging mode 200. is disadvantageous in transmission imaging mode 200 because it tends to reduce .

The near-infrared spectrum above 750 nm or about 805 nm is useful for tuna processing because water is still sensitive to these wavelengths, such as hemoglobin, which contributes to a substantial portion of the biological tissue in tuna fish fillets. This is because it is somewhat transparent. In the relatively visible spectrum (ie, 400 nm-750 nm), hemoglobin absorbs most of the light. One possible explanation for why light at these wavelengths penetrates white meat but not dark meat is due to the different density of muscle fibers between the two materials, with the density of muscle fibers for dark meat being similar to that of white meat. much higher than The absorption coefficient for dark meat in this wavelength range (ie near-infrared or about 805 nm) is much higher, while the absorption coefficient for white meat is lower. Besides physical properties, these characteristics can also be explained by differences in chemical composition, for example. Thus, for white meat, the penetration remains fairly deep. This difference in behavior makes this particular choice of wavelength (ie, about 805 nm) suitable for applications where the object to be scanned is a tuna fish fillet.

In some embodiments, imaging device 204 is one of many commercially available imaging devices 204 including, but not limited to, spectrographs, cameras, or sensors. In embodiments where the imaging device 204 is a sensor, the imaging device 204 is preferably a complementary metal oxide semiconductor (“CMOS”) sensor that captures wavelengths between 300 nm and 1000 nm. Alternatively, in other embodiments, the sensor is a charge-coupled device (“CCD”) sensor that captures similar wavelengths, or an indium gallium arsenide (“InGaAs”) sensor that captures wavelengths between 900 and 1700 nm. It should further be appreciated that in embodiments in which imaging device 204 is a camera or spectrograph, the camera or spectrograph may include any of the sensors described above, along with other electronic components and other types of sensors. .

In order to investigate which wavelengths have the best transmission and capture properties for the object 206 when calibrating the transmission imaging mode 200 or changing the mode 200 for different applications, light 208 is preferably split into individual wavelengths. Although not specifically shown, a grating spectrograph can be used to split the light 208 into individual wavelengths. Alternatively, since spectrographs are sensitive and expensive, a blocking filter that allows only certain selected wavelengths of light to pass through and be captured by the imaging device 204 corresponding to the application, once the mode 200 is corrected, can be used efficiently. can be used to increase and reduce costs. In yet a further alternative, a laser light source 202 can be used that emits only light at specified wavelengths or in specified wavelength ranges, preferably corresponding to preferred wavelengths selected through calibration, thereby also reducing cost and increasing cost. Bring efficiency. By comparison, blocking filters typically have a broader range of pass wavelengths, but since lasers are very specific for particular wavelengths, the choice between the two depends on the desired operating wavelength for the material in question. will depend on

FIG. 3 is a schematic representation of an exemplary embodiment of an interactance imaging mode 300. As shown in FIG. Interactance imaging mode 300 includes light source 302 and imaging device 304 . In some embodiments, the imaging device 304 may be different than the imaging device 204 in transmission mode, and in some embodiments, the imaging device 304 in interaction imaging mode 300 may be the same as the imaging device 204 in transmission mode. good too. In other words, the same imaging device can be used for both transmission and interaction imaging modes. Light source 302 and imaging device 304 may be any of the light sources and imaging devices described above with reference to light source 202 and imaging device 204, respectively. An object 306 to be scanned is above plate 110 and light source 302 emits light 310 . However, as compared to transmission imaging mode 200 , in interactance imaging mode 300 light 310 passes through a converging lens 308 coupled to light source 302 at output 312 of light source 302 . Converging lens 308 can be any of a number of known converging lenses having a principal axis, focus, focal length and vertical plane selected according to the particular application. Converging lens 308 , among other benefits, helps clarify the image data captured by imaging device 304 .

In the interactance imaging mode 300 for embodiments in which the object 306 is a fish fillet, more specifically a tuna fillet, the light source 302 is between 740 and 800 nm, more preferably 770 nm or about 770 nm (i.e., 765 nm and 775 nm). emits light with a wavelength of Further, imaging device 304 preferably is or includes a sensor, CMOS or CCD sensor, as described herein. This wavelength range was found to be suitable for the interactance imaging mode 300 based on the above analysis of the preferred wavelengths for the transmittance imaging mode 200 .

After light 310 is emitted by light source 302 and passes through converging lens 308 , light 310 contacts object 306 as shown by portion 314 of light 310 passing through object 306 . However, in the interactance imaging mode, imaging device 304 measures light backscattered by object 306 . In other words, portion 314 of light 310 corresponds to light that enters object 306 and then bends, curves, or rotates within object 306 due to the material composition of object 306 and light 310 before exiting object 306 . . In other words, light 310 emitted through the converging lens 308 propagates along a first direction 305 and light 310 exiting the object propagates in a second direction 307, in embodiments the first and second directions. are mutually opposite along parallel axes. However, embodiments of the present disclosure also allow the first and first beams to be at sideways angles to each other, such as when the light source 302 is at an angle to the object 306, as described with reference to FIG. It should be noted that it also includes two directions. Light 310 exits object 306 and propagates toward imaging device 304 , and during propagation the light is dispersed as indicated by dispersion portion 316 . When light 310 is received by imaging device 304, imaging device 304 outputs interactance imaging data corresponding to the amount of light 310 captured, as described herein, to a controller or processor (eg, FIG. 5). to the control unit 428 shown in FIG.

In the illustrated embodiment, conveyor 100 is generally translating object 306 from right to left relative to the orientation shown in FIG. As such, the light source 302 is positioned upstream of the imaging device 304 with respect to the direction of translation of the conveyor system 100 . In other words, the light source 302 is generally located near and preferably parallel to the imaging device 304 . While it may be possible to have the light source 302 downstream from the imaging device 304, this arrangement results in less accurate imaging data that must be corrected during processing. Additionally, the conveyor 100 can also translate the object 306 in the opposite direction indicated by arrow 318, in which case the light source is to the left (ie, upstream) of the imaging device 304 in the orientation shown. is preferred. Also, both the light source 302 and the imaging device 304 are positioned above the object 306 such that the interactance imaging mode 300 captures a portion of the light that translates directly through the object 206 along a substantially vertical axis. In contrast to index imaging mode 200 , a portion 314 of light 310 that is scattered back toward imaging device 304 after being incident on object 306 is captured.

FIG. 4 illustrates at least one of a conveyor system 402, a support ring 414 coupled to the conveyor system 402, a plurality of imaging devices 422 coupled to the support ring 414, and a first light source 424 coupled to the support ring 414. 4 shows a perspective view of an exemplary embodiment of imaging system 400 including a light source.

Conveyor system 402 may substantially include all or any of the configurations described above with reference to conveyor system 100 of FIG. Briefly, however, the conveyor system 402 includes a first conveyor or portion 404 and a second conveyor or portion 406 , the first conveyor 404 being separated from the second conveyor 406 by a gap or space 410 . there is Plate 412, preferably transparent, is positioned in gap 410 and coupled to conveyor system 402 to form a continuous conveyor line.

As noted above, support ring or frame 414 is coupled to conveyor system 402 by supports 416, 418, and support ring 414 is circular to facilitate rotation of support ring 414 during calibration of imaging system 400. Preferably. The support 416 is preferably an adjustable collar coupled to and extending from a plate 420 coupled to the conveyor system 402, and more specifically to each of the first and second conveyors 404, 406. . Support 418 is a base having an open channel for receiving support ring 414 coupled to conveyor system 402 such that support ring 414 is manually rotatable by adjusting support collar 416 during calibration of the system. is preferably Although support 416 is shown as a collar and support 418 is shown as a base having a channel for receiving support ring 414 , numerous other arrangements may be used in the present disclosure to couple support ring 414 to conveyor system 402 . It should be understood that any device or arrangement is possible. For example, in other embodiments, coupling may involve the use of one or more centrally disposed spokes extending from conveyor system 402 or another structure located in space 410 and coupled to conveyor system 402. Alternatively, support ring 414 can be coupled to and supported by a housing such as the housing shown in FIG.

Support ring 414 further includes a plurality of imaging devices 422 coupled to support ring 414 and extending therefrom. Each of imaging devices 422 may be substantially similar, if not consistent, to imaging device 204 and any variations thereof described with reference to FIG. Additionally, support ring 414 includes at least a first light source 424, which may be any of the light sources described above with reference to light source 204 in FIG. As shown in FIG. 4, a first light source 424 is positioned between the conveyor system 402 and directs light emitted by the first light source 424 onto the plate 412 and an object 408 on the plate 412 to be imaged or scanned. positioned so as to be oriented toward the Light passes through the plate 412 and the object 408 to be received by at least one of the plurality of imaging devices 422, and the data corresponding to the light received from the first light source 424 corresponds to transmission imaging data. do.

In the illustrated embodiment, the support ring further includes a second light source 426 coupled to and extending from the support ring 414 near the plurality of imaging devices 422 . Preferably, the second light source 426 is used in an interactance imaging mode, where the second light source 426 is positioned near the imaging device 422 in parallel. In yet a further embodiment, the second light source 426 is located near the imaging device 422 but at a sideways angle to the field of view of the imaging device 422 as shown in FIG. 4 and described herein. Second light source 426 may similarly be any of the light sources described above with reference to light source 202 in FIG. A second light source 426 emits light corresponding to the interaction imaging mode 300 described with reference to FIG. As such, light emitted by the second light source 426 is directed toward the object 408 in a first direction, rotates within the object 408, and rotates at least if not all of the plurality of imaging devices 422. Data corresponding to light leaving object 408 in a second direction as received by one and received from second light source 426 corresponds to interactance imaging data. In the illustrated embodiment, the angle between the first and second directions is less than 90 degrees, preferably less than 45 degrees, although the angle depends on the particular application (i.e. the type of object 408 being scanned). It should be understood that the

As shown in FIG. 4, the plurality of imaging devices 422 includes five imaging devices 422 that are evenly spaced from each other along the perimeter, perimeter, or inner edge of the support ring 414 and one of the light sources 424, 426. An input is directed toward plate 412 and object 408 to receive light from. As such, each of imaging devices 422 will receive imaging data corresponding to a different view of object 408 . In other words, there is a difference between the views or data from each of the imaging devices 422 due to their placement, which helps generate the silhouette of the object 408 and the three-dimensional image data. Thus, in the illustrated embodiment, the selection and placement of multiple imaging devices 422 provide multiple views that are input to the machine learning system, thus eliminating the need to rotate support ring 414 during normal operation and allowing the machine learning system to generates a silhouette as a basis for determining a 3D model based on multiple views, as described here. However, it should be appreciated that the specific number, placement and orientation of the imaging devices 422 will depend on the scanned object 408 and the calibration of the system 400, as described herein.

Additionally, each of the imaging devices 422 can receive reflectance imaging data from a second light source 426, which is between 1230 and 1290 nm, or more preferably between 1260 nm or about 1260 nm (ie, between 1255 and 1265 nm). The light emitted at this wavelength is reflected off the outer surface of the object 408 and is received or captured by the plurality of imaging devices 422 . At this wavelength (i.e., about 1260 nm), water will have a high absorption for wavelengths of about 1000 nm, whereas dark meat will begin to reflect light at about 1260 nm, while white meat will not. Suitable for reflectance imaging mode. In the reflectance imaging mode, each of imaging devices 422 may further include an InGaAs sensor to capture this longer wavelength light, as described above. Reflectance imaging data is particularly useful in situations where the object is only partially contained within the outer layer (i.e., a portion of the object extends out from the outer layer), but in other embodiments reflectance imaging The data can be used as correction criteria in addition to the interactance imaging data.

FIG. 4 further shows controller 428 in electrical communication with system 400 . FIG. 5 shows the control unit 428 in detail according to one example non-limiting embodiment. In particular, controller 428 is generally operable to power system 400 and process or transmit imaging data received from imaging device 422 . FIG. 5 schematically illustrates various control systems, modules or other subsystems that operate to control system 400 including data exchange between imaging device 422 and controller 428 .

Control unit 428 includes a controller 442, such as a microprocessor, digital signal processor, programmable gate array (PGA), or application specific integrated circuit (ASIC). The controller 428 may be stored in one or more non-transitory storage media, such as read only memory (ROM) 440, random access memory (RAM) 438, flash memory (not shown), or other physical computer- or processor-readable storage medium. Including media. Non-transitory storage media may store instructions and/or data used by controller 442, such as an operating system (OS) and/or applications. The instructions, as executed by controller 442, implement the logic that implements the functions of the various embodiments of systems 400, 500 described herein, including, but not limited to, acquiring and processing data from imaging device 422. can run.

In embodiments where system 500 (see FIG. 6) includes a rotating support ring or frame 504, controller 428 may be communicatively coupled to one or more actuators (not shown) to control rotation of ring 504. . Alternatively, controller 428 may be communicatively coupled to one or more belts (not shown) to rotate ring 504 . Additionally, the controller 442 may include instructions corresponding to specific positions (i.e., the first position and second position described with reference to FIG. 6), which are predetermined production speeds or conveyor speeds. is sent to an actuator or belt to automatically rotate the support ring 504 according to.

The controls 428 may include a user interface 436 to allow an end user to operate the system 400,500 or provide input to the system 400,500 regarding the operating state or status of the system 400,500. User interface 436 may include a number of user movable controls accessible from system 400 , 500 . For example, the user interface 436 may include a number of switches or keys operable to turn the system 400,500 on and off and/or set various operating parameters of the system 400,500.

Additionally or alternatively, user interface 436 may include a display, eg, a touch panel display. A touch panel display (eg, an LCD with a touch sensitive overlay) can provide both an input and output interface for the end user. The touch panel display is graphical with various user-selectable icons, menus, checkboxes, dialog boxes, and other components and elements selectable by the end-user to set the operating state or status of the system 400,500. A user interface can be shown. User interface 436 may also include one or more auditory transducers, such as one or more speakers and/or microphones. An audible warning notification or signal may thereby be provided to the end-user. It additionally or alternatively allows the end-user to provide audible commands or instructions. User interface 436 may include additional and/or different components than those shown or described and/or may omit some components.

Switches and keys or graphical user interfaces may include, for example, toggle switches, keypads or keyboards, rocker switches, trackballs, joysticks or thumbsticks. Switches and keys or a graphical user interface, for example, instruct the end user to turn on the system 400, 500, enter or exit transmission or interaction imaging modes, and communicatively couple or release remote accessories and programs. , access, transmit or process imaging data, activate or deactivate motors or audio subsystems, initiate or terminate conveyor system operational states, and the like.

Controller 428 includes communication subsystem 444, which may include one or more communication modules or components that facilitate communication with various components of one or more external devices, such as personal computers or processors. Communication subsystem 444 may provide wireless or wired communication to one or more external devices. Communications subsystem 444 may include a wireless receiver, wireless transmitter, or wireless transceiver to provide wireless signal paths to various remote components or systems of one or more paired devices. The communication subsystem 444 may include, for example, short-range wireless communication (eg, via Bluetooth, Near Field Communication (NFC) or Radio Frequency Identification (RFID) components and protocols) or (eg, wireless LAN, low power wide area network). may include components capable of long-range wireless communication (on LPWAN), satellite or cellular networks, and one or more modems or one or more Ethernet or other types of communication cards or components for implementing the same may include Communications subsystem 444 may include one or more bridges or routers suitable for handling network traffic, including switched packet-type communications protocols (TCP/IP), Ethernet, or other networking protocols. In some embodiments, wired or wireless communication with an external device may provide access to lookup tables indicating various material properties and optical wavelength properties. For example, an end user may select materials from a variety of materials displayed on user interface 436, which may be stored in a lookup table or the like on an external device.

The controller 428 includes a power interface manager 432 that manages the supply of power from a power source (not shown) to the controller 428 , eg, various components of the controller 428 integrated or attached to the system 400 , 500 . A power interface manager 432 is coupled to the controller 442 and the power supply. Alternatively, in some embodiments, power interface manager 432 can be integrated into controller 442 . The power source may include, among other things, an external power supply. Power interface manager 432 may include power converters, rectifiers, buses, gates, circuits, and the like. In particular, the power interface manager 432 can control, limit, and limit the delivery of power from the power supply based on various operating conditions of the system 400,500.

In some implementations or embodiments, instructions and/or data stored in non-transitory storage media that may be used by the controller, such as, for example, ROM 440, RAM 438, and flash memory (not shown), may be used by controller 428. Contains or provides an application program interface (“API”) that provides programmatic access to one or more functions. For example, such APIs may include, but are not limited to, one or more functions of the user interface 436 or one or more functions of the system 400, 500, including processing imaging data received by one or more imaging devices 422. A programmatic interface may be provided to control operational features. Such control may be invoked by one of other programs, other remote devices or systems (not shown), or some other module. In this way, the API facilitates the development of third party software, such as various different user interfaces and control systems for other devices, plug-ins and adapters, etc., to operate and control the operation of devices within the system 400, 500. Interactivity and customization can be facilitated.

In the implementation or example embodiment, the components or modules of the controller 428 and other devices within the systems 400, 500 are implemented using standard programming techniques. For example, the logic to perform the functionality of the various implementations or embodiments described herein may be implemented as "native" executable running on a controller, e.g., microprocessor 442, along with one or more static or dynamic libraries. can be implemented. In other implementations, the various functions of controller 428 may be implemented as instructions processed by a virtual machine executing as one or more programs stored in ROM 440 and/or random RAM 438 . In general, the range of programming languages known in the art include, but are not limited to, object-oriented (e.g., Java, C++, C#, Visual Basic.NET, Smalltalk, etc.), functional (e.g., ML, Lisp, Scheme, etc.). ), procedural (e.g. C, Pascal, Ada, Modula, etc.), scripting (e.g., Perl, Ruby, Python, lavaScript, VBScript, etc.) or declarative (e.g., SQL, Prolog, etc.) may be employed to implement such specific examples, including representative embodiments of.

In a software or firmware embodiment, the instructions stored in memory, when executed, configure one or more processors of controller 428 , such as microprocessor 442 , to perform the functions of controller 428 . The instructions direct microprocessor 442 or some other processor, such as an I/O controller/processor, to process and act on information received from one or more imaging devices 422 to reproduce a 3D model based on the imaging data. Provides configuration functions and behaviors.

The implementations or embodiments described above may also use known or other synchronous or asynchronous client-server computing technologies. However, the various components may also be implemented using more monolithic programming techniques, such as, but not limited to, running on a single microprocessor. Various known in the art, including multiprogramming, multithreading, client-server or peer-to-peer (e.g., Bluetooth®, NFC or RFID wireless technology, mesh networks, etc. that provide a communication channel between devices in the system 400, 500) architecture, and may be isolated running on one or more computer systems, each having one or more central processing units (CPUs) or other processors. Some implementations may execute concurrently and asynchronously and communicate using message-passing techniques. Also, other functions may be implemented and/or performed by each component/module, and the functions of control unit 428 may also be implemented by different components/modules in different orders.

Further, the programming interface to the data stored in controller 428 and the functions provided by controller 428 can be in standard mechanisms, such as via C, C++, C# and Java APIs, files, databases or other data repositories. It may be made available by a library, scripting language, or web server, FTP server, or other type of server that provides access to stored data. The data stored and utilized by the control unit 428 and the overall imaging system may be stored in one or more database systems, file systems, or any other means for storing such information, including embodiments using distributed computing technology. or any combination of the above.

Different configurations and locations of programs and data are envisioned for use with the techniques described herein. A variety of distributed computing technologies can be implemented in the illustrated form in distributed form, including, but not limited to, TCP/IP sockets, RPC, RPC, RMI, HTTP and Web Services (XML-RPC, JAX-RPC, SOAP, etc.). Suitable for implementing components. Other variations are also possible. Other functionality may also be provided by each component/module, and existing functionality may be distributed among the components/modules within the system 400, 500 in different ways, yet still control 428 and Imaging systems 400, 500 may be implemented.

Further, in some embodiments, some or all of the components of the controller 428 and components of other devices in the system 400, 500 may be, but are not limited to, one or more application specific integrated circuits (" ASIC"), standard integrated circuits, controllers (e.g., by executing appropriate instructions and including microcontrollers and/or embedded controllers), field programmable gate arrays ("FPGA"), complex programmable logic devices (“CPLD”), etc., may be implemented or provided in other forms, such as at least partially in firmware and/or hardware. Some or all of the system components and/or data structures may also enable or configure a computer-readable medium and/or one or more associated computing systems or devices to perform or configure at least some of the described techniques. by a suitable device (e.g., hard disk, memory, computer network, cellular wireless network or other data transmission medium, or DVD or flash memory device, etc.) or by a suitable It may also be stored as content (eg, as executable or other machine-readable software instructions or structured data) on a computer-readable medium (such as a portable media article read over a connection).

Referring to FIGS. 4 and 5, controller 428 controls multiple imaging devices 422 through support ring 414 and wires 430 that may be either internal or external with respect to conveyor system 402 , support 418 and support ring 414 . are in electrical communication with each other. Alternatively, controller 428 may be in wireless communication with system 400 to wirelessly receive imaging data from imaging device 422, as described above with reference to FIG. Additionally, controller 428 may be coupled to the system or located external to the system. In an embodiment, controller 428 powers system 400 and receives imaging data from imaging device 422 . The controller 428 may include at least one processor, such as in a standard computer, for processing the imaging data, or the controller 428 may include additional processors not specifically shown for clarity. The imaging data can be transmitted to any external processor or computer.

FIG. 6 illustrates an imaging system 500 including a conveyor system 402, a frame or ring 504 coupled to the frame 504, an imaging device 510 coupled to and extending from the frame 504, and first and second light sources 512, 514. Figure 3 shows an alternative exemplary embodiment; Certain configurations of embodiments of system 500 are similar or consistent with configurations described above with reference to system 400, and thus those configurations are not repeated for efficiency.

In this embodiment, frame 504 is coupled to conveyor system 502 by supports 506 , 508 , support 506 being a base having a channel for receiving frame 504 and at least one collar 508 surrounding frame 504 . However, since this embodiment utilizes a single imaging device 510, system 500 allows imaging device 510 to capture imaging data of object 516 from multiple perspectives, angles, or views to facilitate 3D reconstruction. It further includes a mechanism for rotating the frame 504 about the conveyor system 502 so that the For example, base 506 may include a rotating belt in a channel that contacts frame 504 to rotate frame 504 according to input received from an external control, which may be control 428 (see FIG. 4). ing. However, other commercially available mechanisms for rotating frame 504 are not specifically discussed here. Further, although the frame is preferably automatically rotated, rotation can be based on manual rotation by manipulating collar 508, collar 508 inhibiting rotation in the closed position and frame 504 in the closed position. Adjustable between rotatable open positions.

As such, in this embodiment, the frame 504 rotates between at least a first position and a second position, in which the imaging device 510 rotates the first position corresponding to the transmittance or interactance imaging data. A set of imaging data is captured from first light source 512 or second light source 514, respectively. Frame 504 then rotates to a second position and repeats the acquisition process for a second set of imaging data. This process is repeated as needed for a particular application (i.e., third, fourth, fifth, sixth or more views based on imaging device 510 at different positions with respect to object 516). You can generate as many views from as many orientations as possible. Further, in embodiments where the frame 504 is automatically rotated, the rotation of the frame 504 is built in during calibration of the system 500 while reducing the cost of the system 500 due to the use of fewer imaging devices 510. can be efficiently done according to the selected position.

FIG. 7 shows an exemplary representation of a system 600, which may be substantially similar or identical to systems 400, 500, system 600 includes an outer housing or cover 604, with walls 612 of outer housing 604 Solid and opaque. The inlet portions or openings 610 of the housing 604 include covers 608 comprising strips of opaque material extending over at least 80% of the area of each inlet portion 610 such that light cannot enter the housing 604 . Additionally, although not specifically shown, the support ring 414 or frame 504 may be coupled to and supported by the housing 604, the controls 606 may be coupled to the outer wall 612 of the housing 604, and in various embodiments: The controller 606 powers the system 600 , provides coordinates corresponding to the position of the rotating frame 504 and controls the rotation of the rotating frame 504 , or creates a 3D model based on imaging data received from the system 600 . Including a processor to produce.

FIG. 8 illustrates a reconstruction method or system utilized by a machine learning system or deep convolutional neural network to generate a 3D model 702 from one-dimensional (“1D”) imaging data and a two-dimensional (“2D”) silhouette. 700 is a schematic representation.

In general, machine learning and convolutional neural networks (“CNNs”) can generally be implemented as a series of behavioral layers. One or more convolutional layers may follow one or more pooling layers, and one or more pooling layers may optionally follow one or more normalization layers. A convolutional layer creates multiple kernel maps from a single unknown image, also called a filtered image. The large amount of data in the multiple filtered images is reduced with one or more pooling layers, and the amount of data is further reduced by one or more normalization linear unit layers (“ReLU”) that normalize the data. Advantageously, embodiments of the present disclosure rely on semantic segmentation, and the parameters for training the CNN are application dependent and are adjusted according to the complexity of the image data, which in turn depends on the food under inspection. need to

In other words, kernels are selected from known images. Not all kernels for a known image need be used by the neural network. Instead, kernels determined to be "critical" configurations may be selected. After the convolution process produces the kernel map (ie, constituent images), the kernel map is passed through a pooling layer and a normalization (ie, ReLU) layer. All of the values in the output map are averaged (ie summed and divided) and the output value from that average is used as a prediction of whether the unknown image contains a particular structure found in the known image.

In an exemplary case, the output value is the first portion of the object, such as dark meat (ie, second portion) surrounded by white meat (ie, first portion) of a tuna fillet in an embodiment. is used to predict whether an unknown image contains the composition of interest, the second part of the object located inside the part of . This output then allows the CNN to generate a silhouette corresponding to the specific area of interest from the image.

In the illustrated system 700, a machine learning program or deep convolutional neural network will receive as input image data 704 from a plurality of views captured by the system 400, 500, each image data set having a second portion A first portion 706 surrounding 708 corresponds to a photograph of a tuna fillet. A camera or spectrograph may acquire 1D data using line scans, but the 1D data is synthesized into 2D image data before being used by the CNN to reconstruct the silhouette, as described herein.

In an embodiment where the scanned object is a fillet of tuna, the first portion 706 corresponds to the first flesh outer layer of the first set of features, and the second portion 708 corresponds to a different second set of features. A second portion 708 is located within the first portion 706, corresponding to the inner layer of the second body of the. As shown in FIG. 7, each of image data sets 704 corresponds to imaging data, preferably transmission imaging data, in which lighter pixels are assigned to a first portion 706 and darker pixels to a second portion. may represent the intensity value assigned to the portion 708 of . As such, the image data 704 is 1D in the sense that it is a single line of a 2D image, or in other words, each pixel or kernel analyzed by the CNN corresponds to an intensity value.

At the highest level of machine learning programs or deep convolutional neural networks, the CNN is trained based on a pool of thousands of representative sample images to obtain a general representation of the second portion 708 (i.e. dark meat of tuna fillets). For example, the second portion 708 passes through the center of the fillet parallel to its major axis based on a large pool of reference images, which may include thousands of tuna fillet images. At another level, the CNN will acquire knowledge about edges, lines and curves based on representative sample images, and the accuracy of the CNN will improve as more images are scanned. As such, the CNN will identify the portion of the photograph corresponding to the second portion 708 based on the difference in intensity values. Once these portions are identified, the CNN will formulate a plurality of silhouettes 710, each corresponding to a second portion 708 identified in each of the 2D represented views.

For example, a CNN consists of many layers, and the layers between the input and output are called "hidden layers". Each layer has a large number of neurons, which are fully connected between layers. These connections correspond to weights learned based on reference images. A neuron or node is a computing unit that takes an input value, multiplies it with the associated weight, looks it up through an activation function (eg, ReLU as described herein), and delivers an output. This output forms the input of the next neuron coupled through other connections. In addition, CNNs may include other layers such as convolution, pooling, normalization and dropout that are used similarly to neurons but have different functions.

Before the network is trained, connections or weights between nodes are randomly assigned. Training the network uses labeled or annotated data. For example, input data (eg, image data) is correlated with expected output data (eg, silhouettes). By providing the input data (e.g., image data) through the input nodes in the first layer and recognizing the expected value of the output layer (e.g., labeling silhouettes), the CNN expects whatever the input data is. A connection's weight can be adjusted through several iterations to return an output that This is basically an optimization process with a large number of parameters. Training a CNN can involve tens, hundreds, or even thousands of iterations, since if one weight is changed, it will affect the entire network.

The CNN's convolutional layers reduce the size of the image, which determines the visible or evaluated area. For example, a small window of 9x9 pixels is moved across the full image under analysis. In that window, the observer can see small sections (eg, lines and corners) of the entire object. As the image size is reduced, the window size remains the same, but more of the object configuration is recognized. If the images are very small and nearly aligned in the window, the observer can see the whole fillet and also the dark meat in a single step.

This is an example of what the neural network is seeing. In the former layer of the network, weights will be learned that allow detection of relevant lines and corners in identifying dark meat in, for example, tuna fish fillets. In the latter layer, however, these lines are displaced until the entire dark meat is recognized as an object and related to its surroundings as a reference (e.g., the dark meat lies between the borders of the white meat and is aligned with the principal axis). will form a curve. These constructs are learned in parallel, but are all related. Without lines and corners, the boundaries of the dark meat are difficult to locate based on the overall appearance of the dark meat. Similarly, there are many lines and corners corresponding to dark meat, but it is difficult to tell which lines and corners are relevant when the overall appearance of the dark meat is unknown. . However, having knowledge of both these high-level and low-level configurations allows the detection of dark meat in image data.

The computing system then backprojects each of the silhouettes 710 onto a plurality of projections 712 by using an algorithm that draws a line corresponding to the outer boundary of each silhouette 710 higher, as shown in FIG. Let it expand into a dimensional scene. By backprojecting each of the silhouettes 710 and analyzing the intersection 714 between the projections 712, a 3D model 702 of the second portion 708 can be determined. In an embodiment, the back projection is based on a cone-shaped field of view. Imaging data corresponding to an interaction imaging mode, such as mode 300, can be used to refine the model based on the depth of the object of interest. For example, if the properties of the object of interest are known based on a database of information corresponding to it, the amount of light scattered to be captured in the interactance imaging mode is, in an embodiment, the dark meat of a tuna fillet. will change depending on the depth of the object. In addition, the interactance imaging data helps correct for concave topography in the surface of the scanned object, as the captured light will vary according to the depth of the object, as described above. As such, if the object has a concave shape, the interactance imaging data will be different for the portion of the object with the concave shape, the material being thicker than the portion of the object without the concave shape. In contrast, a thinner (i.e., lower intensity value captured) corresponds to a thinner material as less of the light will be scattered and captured; (corresponds to thicker materials, as they will be scattered if they cannot penetrate or pass through).

FIG. 9 is a flow diagram representing an exemplary method 800 for generating a 3D model of an object of interest based on image data captured by an imaging system (eg, imaging systems 400, 500). The method 800 begins at 820, the conveyor is activated at 804, and an object to be scanned is loaded onto the conveyor. Operation of the conveyor can be performed through an external switch or through a controller or program. Similarly at 806, the light transmission system, which may be substantially equivalent to transmission imaging mode 200 indicated at 200, is activated either manually via an external switch or through a controller or processor. be done. An imaging device that is part of the light transmission system, either by itself or through a program associated with a controller in electronic communication with the transmission system, responds 808 to the light passing through the objects on the conveyor. A determination is made as to whether transmittance image data is received by an imaging device, which in embodiments is a spectrograph, camera, or sensor.

If no image data is received, the process returns to 806 and repeats until image data is received. Once the transmittance image data is received, it is sent to the processor and method 800 proceeds to 810 where transmittance mode is deactivated and interactance imaging mode is activated. The interactance imaging mode may be substantially equivalent to the interactance mode 300 described with reference to FIG. Again at 812, it is determined whether the imaging device, ie, the spectrograph, receives the interactance imaging data. If not, the processor repeats by returning to 810 until the imaging device receives the interaction imaging data. Once the interactance imaging data is detected, the interactance imaging data is sent to the processor at 814 . In embodiments where there are multiple imaging devices, the above process may be repeated for each unique imaging device to generate multiple views. Alternatively, in embodiments where the imaging device rotates, this process may be repeated each time the imaging device is positioned at a unique position to generate multiple views or multiple transmission imaging datasets and multiple interactance imaging datasets. is repeated to

The processor includes a machine learning program, or CNN, which receives as input the transmittance image data corresponding to each view at 816 . The CNN then, at 816, determines the desired configuration in each transmittance image data set, which in embodiments is an object located inside a piece of food, or a second portion of a fish located within a first portion of a fish. Generate multiple silhouettes corresponding to . Each of the silhouettes is backprojected 818 into multiple projections and the common intersection of each projection is analyzed to determine 3D geometry based on the intersection. The processor then outputs this 3D geometry at 822 and it is determined, either manually or through additional processor steps, whether the object of interest is near the surface of the scanned object. If not, the 3D geometry is output based on the transmittance imaging data and method 800 ends at 828 .

If the object of interest is near the surface, processing continues at 824 where the CNN uses the interactance imaging data to correct or clarify the 3D geometry based on the transmittance image data. Once the 3D geometry is corrected at 824, the processor, or CNN, outputs the corrected 3D geometry at 826 and the process ends at 828.

FIG. 10 is an alternative exemplary embodiment of a method 900 for generating a 3D model of an object of interest based on image data captured by an imaging system. The method 900 starts at 902 where the conveyor and light transmission system 904 are activated. The imaging device, ie, the spectrograph, determines 906 whether imaging data corresponding to the transmission imaging data is received or captured by the imaging device. If neither received nor captured, the processor returns to 904 until data is received. In such cases, the method 900 continues to 908 where the transmission imaging data is sent to the processor. The processor then determines a plurality of 2D silhouettes from the 1D transmittance image data at 910 via a convolutional neural network. Each of the silhouettes is backprojected and the intersection analyzed at 912 . The processor then outputs 3D geometry at 914 based on the common intersection between each of the projections.

It is then determined at 916 by the processor, or CNN, whether the object of interest is near the surface of the scanned object. If not, the method 900 ends at 926 and the 3D model is based on the transmission imaging data. If so, the method 900 continues operating the interactance imaging system at 918 and the imaging device determines at 920 whether imaging data corresponding to the interactance imaging data is received by the imaging device or spectrograph. If not, method 900 returns to 918 until such data is received or captured. If so, method 900 proceeds to 922 where the interactance imaging data is sent to a processor and, if necessary, the 3D model is corrected based on the interactance imaging data. Finally, the corrected 3D model is output at 924 and the method 900 ends at 926.

Through the acquisition of three-dimensional information, the present disclosure enables more accurate determination of the volume and shape of internal defects that can affect quality control inspections as to whether a particular food ingredient should be discarded. Furthermore, by knowing the three-dimensional geometry of the object, processing and removal of secondary products can be performed more accurately, thus minimizing loss of primary products.

The above description of the illustrated embodiments includes what is described in summary and is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Although specific example embodiments and examples are described herein for purposes of illustration, various equivalent changes can be made without departing from the spirit and scope of the disclosure, as recognized by those skilled in the art. The teachings provided herein of various embodiments are applicable outside the context of food processing, not necessarily the exemplary imaging systems and methods outlined above.

For example, the above detailed description has set forth various embodiments of devices and/or processes through the use of block diagrams, schematic diagrams, and examples. To the extent such block diagrams, schematic diagrams and examples include one or more functions and/or acts, each function and/or act within such block diagrams, flowcharts or examples may be individually and/or collectively Moreover, it should be understood by those skilled in the art that it can be substantially implemented by a wide variety of hardware, software, firmware, or any combination thereof. In one embodiment, the present subject matter may be implemented via an application specific integrated circuit (ASIC). However, one skilled in the art will appreciate that the embodiments disclosed herein can be implemented in whole or in part as one or more computer programs executed by one or more computers (e.g., one program running on one or more computer systems). as one or more programs executed by one or more controllers (e.g., microcontrollers); as one or more programs executed by one or more processors (e.g., microprocessors); as firmware; Substantially any combination thereof is equally implementable on a standard integrated circuit, and the design of the circuits and/or the writing of code for software and/or firmware are fully relevant in light of the teachings of this disclosure. You will recognize that you are within the skill of the trader.

When the logic is implemented as software and stored in memory, the logic or information may be stored on any computer-readable medium for use by or in connection with any processor-related system or method. In the context of this disclosure, a memory is an electronic, magnetic, optical or physical device or computer readable medium that is a means for containing or storing a computer and/or processor program. Logic and/or information is an instruction execution system, such as a computer-based system, a processor-embedded system, or other system capable of fetching instructions from an instruction execution system, apparatus, or device, and executing instructions associated with the logic and/or information. It may be embodied in any computer readable medium for use by or in connection with a system, apparatus or device.

In the context of this specification, a "computer-readable medium" is any element capable of storing a program related logic and/or information for use by or in connection with an instruction execution system, apparatus and/or device. could be. A computer readable medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (non-exhaustive enumeration) of computer readable media are: portable computer diskettes (magnetic cards, compact flash cards, secure digital cards, etc.), random access memory (RAM), read only memory (ROM) , erasable programmable read-only memory (EPROM, EEPROM or flash memory), portable compact disc read-only memory (CDROM), digital tape and other non-transitory media.

Many of the methods described herein can be implemented with variations. For example, many of the methods may include additional acts, omit some acts, and/or perform acts in a different order than shown or described.

The various embodiments described above can be combined to provide further embodiments. US patents, US patent application publications, US patent applications, foreign patents, foreign patents, US patent applications, US patent applications, foreign patents, foreign patents, referenced herein and/or listed in application data sheets to the extent they are consistent with the specific teachings and definitions herein. All patent applications and non-patent publications are hereby incorporated by reference in their entirety. Aspects of the embodiments can be modified, if necessary, to employ the teachings of the various patents, specifications and publications to provide still further embodiments.

These and other changes can be made to the embodiments in light of the above detailed description. In general, the terms used in the following claims should not be construed to limit the scope of the claims to the specific embodiments disclosed in the specification and claims; Such claims should be interpreted to include all possible embodiments, along with the full scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by the disclosure.
It should be noted that the present invention includes the following aspects.
[Aspect 1]
a first conveyor;
a second conveyor separated from the first conveyor by a gap;
a transparent plate positioned in the gap and coupled to at least one of the first conveyor and the second conveyor;
a support ring positioned at least partially in the gap and coupled to at least one of the first conveyor and the second conveyor;
at least one imaging device coupled to the support ring;
a first light source coupled to the support ring;
a controller in electronic communication with the support ring and the at least one imaging device;
A system comprising
During operation, the first light source emits light directed toward an object on the transparent plate, the controller receives imaging data from the at least one imaging device, the controller comprises: The system for constructing a 3D model of the second portion of the object contained within the first portion of the object.
[Aspect 2]
2. The system of aspect 1, wherein the at least one imaging device transmits imaging data to the controller, the imaging data including one of interactance imaging data and transmittance imaging data.
[Aspect 3]
Aspect 1. The system of aspect 1, wherein the object is a tuna fillet and the first light source emits light at a wavelength equal to one of approximately 1260 nanometers, approximately 805 nanometers, or approximately 770 nanometers.
[Aspect 4]
2. The system of aspect 1, wherein the processor processes the imaging data using machine learning in the form of convolutional neural networks.
[Aspect 5]
The convolutional neural network receives the image data and outputs a plurality of silhouettes based on the image data corresponding to the second portion of the object, and the processor projects the silhouettes onto a plurality of projections. and analyzing intersections between the plurality of projections to construct the 3D model.
[Aspect 6]
The support ring includes a plurality of cameras coupled to the support ring, each of the plurality of cameras capturing one of transmittance, interactance or reflectance imaging data from the first light source. A system according to claim 1.
[Aspect 7]
7. The system of aspect 6, wherein the support ring includes a second light source coupled to the support ring, the second light source emitting light directed toward the transparent plate during operation.
[Aspect 8]
a conveyor having a space between a first portion and a second portion of the conveyor;
a plate positioned in the space and coupled to the conveyor;
a support ring positioned at least partially in the space and coupled to the conveyor;
at least one light source coupled to the support ring;
an imaging device coupled to the support ring;
a processor in electronic communication with the imaging device;
A device comprising
During operation, the support ring rotates between at least a first position and a second position, the at least one light source emits light directed toward an object on the plate, and the imaging device receives light from the at least one light source after the light has passed through the object;
The processor outputs a first set of image data from the imaging device when the support ring is in the first position and a second set of image data from the imaging device when the support ring is in the second position. The device receiving a set of image data and outputting a 3D model of an interior portion of the object from the first set of image data and the second set of image data.
[Aspect 9]
The processor utilizes machine learning to process the first set of image data and the second set of image data into a plurality of silhouettes, project the plurality of silhouettes onto a plurality of projections, and 9. The device of aspect 8, wherein the dimensional model is based on intersections between each of the plurality of projections.
[Aspect 10]
Further comprising a second light source coupled to the support ring, the imaging device captures a third set of image data from the second light source when the support ring is in the first or second position. 9. The device of aspect 8, wherein capturing, the processor utilizes the third set of image data to clarify boundaries of the three-dimensional model.
[Aspect 11]
9. The device of aspect 8, wherein the imaging device is a spectrograph and the at least one light source emits light at a wavelength selected from about 1260 nanometers, about 805 nanometers, or about 770 nanometers.
[Aspect 12]
emitting light from a light source, said emitting step comprising directing said light through an object having a first portion and a second portion, said second portion within said first portion; a step surrounded by
Capturing light from the light source with an imaging device after the light has passed through the object, the captured light being an image of the first portion and the second portion received by the imaging device. a step corresponding to the data;
sending the image data to a processor;
analyzing the image data by the processor to detect a boundary between the first portion and the second portion, the analyzing step utilizing machine learning to detect the second portion; generating a three-dimensional representation of the portion;
How to prepare.
[Aspect 13]
13. The method of aspect 12, wherein emitting light from the light source comprises emitting the light at a wavelength selected from about 1260 nanometers, about 805 nanometers, or about 770 nanometers.
[Aspect 14]
13. The method of aspect 12, wherein utilizing machine learning to generate the three-dimensional representation of the second portion comprises the machine learning utilizing a deep convolutional neural network to process the image data. the method of.
[Aspect 15]
13. The method of aspect 12, wherein analyzing the image data by the processor includes utilizing machine learning to output a plurality of two-dimensional silhouettes corresponding to the image data of the second portion.
[Aspect 16]
16. Aspect 15, wherein analyzing the image data by the processor includes utilizing machine learning to create a plurality of projections, each projection corresponding to a respective one of the plurality of two-dimensional silhouettes. Method.
[Aspect 17]
The analyzing step includes generating a three-dimensional representation using machine learning, analyzing intersections between each of the plurality of projections to output a three-dimensional representation of the second portion of the object. 16. The method of aspect 15, further comprising:

Claims (15)

a first conveyor;
a second conveyor separated from the first conveyor by a gap;
a transparent plate positioned in the gap and coupled to at least one of the first conveyor and the second conveyor;
a support ring positioned at least partially in the gap and coupled to at least one of the first conveyor and the second conveyor;
a plurality of imaging devices coupled to the support ring;
a first light source coupled to the support ring and emitting light at a wavelength equal to one of about 1260 nanometers, about 805 nanometers, or about 770 nanometers;
A system comprising a controller in electronic communication with the support ring and the plurality of imaging devices, comprising:
During operation, the first light source emits light directed toward food on the transparent plate, the controller receives imaging data from the plurality of imaging devices, and the controller receives imaging data from the plurality of imaging devices. constructing a 3D model of a second portion of the food product contained within the first portion of the food product ;
The system, wherein the plurality of imaging devices transmit imaging data to the controller, the imaging data including one of interactance imaging data and transmittance imaging data. 2. The system of claim 1, wherein the food product is tuna fillet. 2. The system of claim 1, wherein the processor processes the imaging data using machine learning in the form of convolutional neural networks. The convolutional neural network receives image data from the plurality of imaging devices and outputs a plurality of silhouettes based on the image data corresponding to the second portion of the food product , and the processor generates a plurality of projections. 4. The system of claim 3, wherein the silhouette is projected onto the 3D model and the intersection between the plurality of projections is analyzed to construct the 3D model. The support ring includes a plurality of cameras coupled to the support ring, each of the plurality of cameras capturing one of transmittance, interactance or reflectance imaging data from the first light source. , the system of claim 1. 6. The system of claim 5 , wherein the support ring includes a second light source coupled to the support ring, the second light source emitting light directed toward the transparent plate during operation. a conveyor having a space between a first portion and a second portion of the conveyor;
a plate positioned in the space and coupled to the conveyor;
a support ring positioned at least partially in the space and coupled to the conveyor;
at least one light source coupled to the support ring and emitting light at a wavelength equal to one of about 1260 nanometers, about 805 nanometers, or about 770 nanometers;
an imaging device coupled to the support ring;
A device comprising a processor in electronic communication with the imaging device, comprising:
During operation, the support ring rotates between at least first and second positions, the at least one light source emits light directed toward the food product on the plate, and the imaging device receives light from the at least one light source after the light has passed through the food product ;
The processor outputs a first set of image data from the imaging device when the support ring is in the first position and a second set of image data from the imaging device when the support ring is in the second position. The device receiving a set of image data and outputting a 3D model of an inner portion of the food product from the first set of image data and the second set of image data. The processor utilizes machine learning to process the first set of image data and the second set of image data into a plurality of silhouettes, project the plurality of silhouettes onto a plurality of projections, and 8. The device of claim 7 , wherein a model is based on intersections between each of said plurality of projections. Further comprising a second light source coupled to the support ring, the imaging device captures a third set of image data from the second light source when the support ring is in the first or second position. 8. The device of claim 7, wherein capturing, the processor utilizes the third set of image data to clarify boundaries of the 3D model. 8. The device of claim 7, wherein said imaging device is a spectrograph. conveying the food product on a conveyor;
emitting light from a light source, said emitting step comprising directing said light through said food product having a first portion and a second portion, said second portion being said first portion; said light source emitting light at a wavelength equal to one of about 1260 nanometers, about 805 nanometers or about 770 nanometers ;
Capturing light from the light source with an imaging device after the light has passed through the food product , wherein the captured light is an image of the first portion and the second portion received by the imaging device. a step corresponding to the data;
sending the image data to a processor;
analyzing the image data by the processor to detect a boundary between the first portion and the second portion, the analyzing step utilizing machine learning to detect the second portion; and generating a three-dimensional representation of the portion. 12. The method of claim 11 , wherein utilizing machine learning to generate the three-dimensional representation of the second portion comprises the machine learning utilizing a deep convolutional neural network to process the image data. described method. 12. The method of claim 11 , wherein analyzing the image data by the processor includes utilizing machine learning to output a plurality of two-dimensional silhouettes corresponding to the image data of the second portion. 14. The step of analyzing the image data by the processor comprises utilizing machine learning to create a plurality of projections, each projection corresponding to a respective one of the plurality of two-dimensional silhouettes. the method of. The analyzing step includes generating a three-dimensional representation using machine learning, analyzing intersections between each of the plurality of projections to output a three-dimensional representation of the second portion of the food product . 15. The method of claim 14 , further comprising: