EP3491613A1

EP3491613A1 - Horticultural care tracking, validation and verification

Info

Publication number: EP3491613A1
Application number: EP17853934.2A
Authority: EP
Inventors: Adam Phillip Takla Greenberg; Matthew Charles King
Original assignee: Iunu Inc
Current assignee: Iunu Inc
Priority date: 2016-09-21
Filing date: 2017-09-21
Publication date: 2019-06-05
Also published as: CA3038966A1; WO2018057799A1; EP3491613A4

Abstract

A system to collect information about plants on a per plant or near per plant basis and to promulgate remedial courses of action is disclosed. A horticultural operation collects plant information, aggregates the information, and allows an administrative worker such as a master grower to review issues about plants in a grow operations and the associated potential remedial course of action. The administrative worker can then browse data about the affected plants to determine whether to select the potential remedial course of action. Browsing and review of issues may be done collaboratively by remote administrative workers. Remedial courses of action are then dispatched to line workers solely using an iconographic language, who in turn provide notifications when the courses of action are done. Administrative workers may then validate and verify that the courses of action have been properly executed and evaluate the courses of action's efficacy.

Description

HORTICULTURAL CARE

TRACKING, VALIDATION AND VERIFICATION

RELATED APPLICATIONS

[0001] This application is a continuation-in-part of application of U.S. Utility Patent Application Serial No. 15/271,569, filed on September 21, 2016, entitled, "Hi -Fidelity Computer Object Recognition Driven Horticultural Feedback Loop," which is hereby incorporated herein in its entirety by this reference."

BACKGROUND

[0002] A horticultural operation generally includes planting, cultivation and the harvesting of plants. Large scale horticultural operations may include multiple grow operations in multiple locations with staff with widely varying capabilities. A master grower, responsible for a large scale horticultural operation, is faced with many challenges to collect actionable information about plants and grow operations in his or her care, to identify issues, to identify remedial courses of action, and to dispatch those courses of action to workers. In other words, a master grower is responsible for implementing and executing a horticultural feedback loop - the monitoring and remediation of a horticultural operation.

[0003] One challenge is to collect and share information in a timely and effective fashion. Present operations are generally manual, inconsistent, and slow to collect let alone aggregate in a fashion conducive for developing a remedial course of action. Even if a course of action is developed, dispatching those course of actions to workers with widely differing capabilities present additional challenges.

[0004] Central to the concept of a feedback loop, including a horticultural feedback loop, is the ability to not only track remedial courses of action but also to verify that a course of action has been properly performed and to determine the efficacy of that course of action. Accordingly, there are many challenges to implement a true horticultural operation feedback loop.

BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The Detailed Description is set forth with reference to the accompanying figures.

[0006] Figure 1 is a top level context diagram for horticultural care tracking, validation and verification.

[0007] Figure 2 is an architectural diagram of an exemplary hardware, software and communications environment for horticultural care tracking, validation and verification.

[0008] Figure 3 is an exemplary image of interest for horticultural care tracking, validation and verification.

[0009] Figure 4 is a flow chart of an exemplary issue identification and dispatching process for horticultural care tracking, validation and verification.

[0010] Figure 5 is a flow chart of an exemplary collaborative process for horticultural care tracking.

[0011] Figure 6 is a flow chart of an exemplary display patterns of multiple images during horticultural care tracking.

[0012] Figure 7 is a flow chart of an exemplary client side task management process for horticultural care tracking

[0013] Figure 8 is a flow chart of an exemplary validation and verification process for horticultural care tracking. DETAILED DESCRIPTION

Context of Horticultural Care Tracking, Validation and Verification

Overview

[0014] Modern horticultural operations are potentially far flung and heterogeneous operations. Figure 1, is a context diagram 100 of an exemplary horticultural operation 102 instrumentation for automated informational and operational feedback. With this instrumentation and other disclosed infrastructure, horticultural care tracking, validation and verification is enabled.

[0015] A horticultural operation 102 may have several locations. For example, there may be several greenhouses illustrated as greenhouse A 104(a) and greenhouse B 104(b). Note that the greenhouses need not be physically proximate to each other. In fact, the greenhouses may be many miles away. They are related in that they are under the responsibility of the horticultural operation 102. For example, they may be owned by the same company. Also note that greenhouses 104 need not actually be greenhouses, but may also be a location where plants are grown, for example an open field or a hydroponic operation.

[0016] A greenhouse 104 may have multiple grow operations 106(a), 106(b) and 106(c). Although grow operations 106 are generally in the same location, a grow operation 106 may be a logical construct. For example, in the Figure 1, grow operation A 106(a) is sited entirely within greenhouse A 104(a) and grow operation C 106(c) is sited entirely within greenhouse B 104(b). However, grow operation B 106(b) is partially sited within greenhouse A 104(a) and partially sited within greenhouse B 104(b). For example, grow operation B 106(b) may be a special set of plants grown under the same purchase order, and accordingly a company may seek to track those plants together under the same grow operation 106. [0017] Each grow operation 106, may have one or more plants 108 under care. As shown in Figure 1, three types of plants 108 are illustrated, plant X, plant Y and plant Z. Notwithstanding Figure 1, A grow operation 106 comprise only one type of plant or alternatively multiple types of plants.

[0018] As part of instrumentation, each plant 108, or substantially each plant 108 may have its own image capture device 110 (abbreviated "ICD" in Figure 1). An image capture device may be a digital video camera or alternatively a digital still camera configured to capture images periodically. Each image capture device 1 10 has the capability of uploading captured images either directly to a server, for example in the internet/cloud 112, or via an intermediate server (not shown). The image capture device 110 may be part of a larger suite of sensors (not shown) networked to a data capture function (not shown) which upload plant, telemetry, media and other data to an image processing server 114 in the internet/cloud 112. A workflow server 116 handles the coordination of information to different servers and users. The image capture device 110, internet/cloud 112, image processing server 114, workflow server 116, data capture function, and suite of sensors are described in further detail with respect to Figure 2.

[0019] Once plant, telemetry, media and other data from the image capture devices 110 and other sensors are uploaded to the image processing server 114, the information may be analyzed. After analysis, the images, other images, and results of the analysis may be made available via the workflow server 116, to administrative personnel including a master grower 118, responsible for the entire horticultural operation 102, or a grower 120, responsible for a subset of the grow operations 106 and reporting to the master grower. Administrative personnel 118 and 120 generally are responsible for reviewing the state of plants within the horticultural operation 102, identifying issues and directing remedial courses of action to address those identified issues. [0020] To aid administrative personnel 118 and 120, the image processing server 114 may perform image analysis to automatically detect issues in plants. Alternatively, the workflow server 116 make plant images available to administrative personnel 118 and 120 on demand. Information is provided from the workflow server 116 to administrative personnel 118 and 120 via a dispatching device 122. The dispatching device 122 is generally a mobile device and is described in further detail with respect to Figure 2.

[0021] Additionally, line workers 124 may also have worker devices 126. The workflow server 116 provides telemetry, media, and other information such as workflow information to the line workers 124 via the worker devices 126. The worker device 126 is generally a mobile device and is described in further detail with respect to Figure 2.

[0022] In one embodiment of workflow, the workflow server 116 provides telemetry, media and other information to a dispatching device 122. Information may be a notification 128. Notifications 128 generally are small messages that indicate that there is an issue, or that an event has occurred. A notification 128 may include related telemetry and/or media. Provision of related telemetry and/or media is described in further detail with respect to Figure 4.

[0023] An administrative person 118 and 120 may then review the telemetry, media and other information on the dispatching device 122. In some cases notification 128 is an indication of an issue in a plant, as determined by analysis performed by the image processing server 114. In other cases, an issue is identified by an administrative person 118 and 120 reviewing telemetry, media and other information on the dispatching device 122. Upon the administrative person 118 and 120 selecting or specifying a remedial course of action to address the identified issue, the dispatching device 122 sends a dispatch 130 to the workflow server 116 to instruct a line worker 124 as selected by the workflow server 116, to perform the remedial course of action. [0024] The notifications 128 may be made to a line worker 124 via an iconographic language. In some cases, line workers 124 may have a different primary language than the administrative personnel 118 and 120 or may have different educational levels. By using an iconographic language for dispatches, the iconographic language provides lingua franca to effectively bridge any language and/or educational divides. The iconographic language is described in further detail with respect to Figure 7.

[0025] The dispatch 130 is then forwarded to the line worker 124 via the line worker's worker device 126. The line worker 124 then performs the course of action in the dispatch 130 and may send a notification 128 indicating completion. In some cases, the line worker 124 may also provide some annotations 132 to associate with the notification 128 indicating completion, which the workflow server 116 may then make available to the administrative personnel 118 and 120 to review on a dispatching device 122.

[0026] In some cases, administrative personnel 118 and 120 may identify issues by reviewing telemetry, media and other information on the dispatching device 122. Before committing to a remedial course of action, an administrative person 118 and 120 may seek to collaborate with another administrative person 118 and 120. Accordingly, the dispatching devices 122 enable collaborative viewing, communication and commenting by remote administrative personnel 118 and 120. Collaborative viewing is described in further detail with respect to Figure 4.

[0027] In this way, Figure 1 illustrates an exemplary complete horticultural feedback loop, with integrated automated diagnoses and analysis, as well as independent analysis by administrative workers 118 and 120, and with communication with line workers 124 via an iconographic language. Exemplary Hardware, Software and Communications Environment

[0028] Prior to disclosing horticultural care tracking, validation and verification and related techniques, an exemplary hardware, software and communications environment is disclosed. Figure 2 illustrates several possible embodiments of a hardware, software and communications environment 200 for horticultural care tracking, validation and verification related techniques.

Telemetry Capture

[0029] Images are captured via an image capture device 110. As described above, the image capture device 110 may be a digital video camera or a digital still camera configured to capture images periodically and/or on demand. Because of the automated nature of digital video and/or digital still cameras, it is possible to economically deploy image capture devices 110 on a per plant or substantially per plant basis.

[0030] Generally an image capture device 110 may take visible light spectra pictures, but may also extend to non-visible spectra such as infrared and ultraviolet. The image capture device 110 may have an on board application programming interface (API) enabling programmatic control. Alternatively the image capture device 110 may be networked thereby enabling remote control.

[0031] Not all telemetry collected is image based. In addition to an image capture device 110, sensors 202 may collect telemetry on a per plant or substantially per plant basis. Any sensor 202 that may be connected to a standard computer input/output interface (described below) may be added. Without limitation, sensors 202 may include light meters, water meters, potential of hydrogen (pH) acid/alkali meters, and the like.

[0032] Some telemetry from sensors 202 will be the sensor readings themselves. Some telemetry may be calculated from sensor 202 measurements. For example, a light meter may measure light intensity for that moment of time. However, an extrapolation calculation may estimate the daily light integral, which is the total light applied to a plant over a given time period (in this case a day).

[0033] Other telemetry from different sensors 202 may be combined. For example, a light meter may provide a measurement of the amount of light over time and an oxygen sensor may measure amount of 0₂ generated by a plant over time. From these two measurements, the photosynthetic efficiency measurements, such as the relative photosynthesis index may be calculated.

[0034] Telemetry from sensors 202 may be combined with outside information. For example, a sensor 202 providing telemetry for the amount of vapor in the air, may be combined with the water saturation point, to calculate the vapor pressure deficit. The vapor pressure deficit is the difference between the amount of water in the air and the amount of water the air can hold if saturated.

[0035] Control functions for telemetry capture may be in a separate telemetry capture function 204. The telemetry capture function 204 may incorporate the image capture device 110 and may be part of a larger integrated device, such as a lumiere feedback device. Indeed, the image capture function 204 may be part of a lumiere feedback device itself.

Hardware for Telemetry Capture Control Function

[0036] The telemetry capture control function 204 is generally hosted on a computing device. Exemplary computing devices include without limitation personal computers, laptops, embedded devices, tablet computers, smart phones, and virtual machines. In many cases, computing devices are to be networked.

[0037] The computing device for the telemetry capture control function 204 may have a processor 206 and a memory 208. The processor 206 may be a central processing unit, a repurposed graphical processing unit, and/or a dedicated controller such as a microcontroller. The computing device for the telemetry capture control function 204 may further include an input/output (I/O) interface 210, and/or a network interface 212. The I/O interface 210 may be any controller card, such as an universal asynchronous receiver/transmitter (UART) used in conjunction with a standard I/O interface protocol such as RS-232 and/or Universal Serial Bus (USB). The network interface 212, may potentially work in concert with the I/O interface 210 and may be a network interface card supporting Ethernet and/or Wi-Fi and/or any number of other physical and/or datalink protocols. Accordingly, sensors 202 may interface with the telemetry capture function 204 via a connected port, serial or parallel, and/or via networking.

[0038] Memory 208 is any computer-readable media which may store several software components including an operating system 214 and software components such as an data flow controller 216 and/or other applications 218. In general, a software component is a set of computer executable instructions stored together as a discrete whole. Examples of software components include binary executables such as static libraries, dynamically linked libraries, and executable programs. Other examples of software components include interpreted executables that are executed on a run time such as servlets, applets, p- Code binaries, and Java binaries. Software components may run in kernel mode and/or user mode.

[0039] Computer-readable media includes, at least, two types of computer-readable media, namely computer storage media and communications media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. In contrast, communication media may embody computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transmission mechanism. As defined herein, computer storage media does not include communication media.

Data Flow

[0040] The data flow controller 216 is a software component responsible for managing the capture of telemetry, receiving images from the image capture device 110 (if not integrated with the telemetry capture function 204) and sensor data streams from sensors 202, the local management of received images and data, and potentially the transmission of received images and data off the telemetry capture function 204 over a network.

[0041] Other applications 218 may be utilities to perform image and/or data processing, such as compression and/or encryption. Other utilities 218 may include functions to create data streams based on calculations from telemetry one or more sensors.

[0042] The data flow controller 216 may also manage the transmission of received images and data. Specifically, it may transmit an image or data to a known network location via the network interface 212. The known network locations may include a server 220 or an internet/cloud location 112.

[0043] Upon transmission, the telemetry flow controller 216 may enlist in notifications to determine that the transmission was successful. The telemetry flow controller 216 may also transmit notifications to other device subscribing to its notifications indicating status of a transmission. Servers

[0044] The telemetry capture function 204 may communicate to a server 220. Example servers include the image processing server 114 and the workflow server 116. The server 220 is any computing device that may participate in a network. The network may be, without limitation, a local area network ("LAN"), a virtual private network ("VPN"), a cellular network, or the Internet. The server 220 is analogous to the host computer for the telemetry capture function 220 described above. Specifically, it will include a processor 222, a memory 224, and a network interface 224. In the memory will be an operating system 228 and software components 230 comprising applications. The server 220 may communicate with a data store 232 via a direct input/output connection via an input/output interface (not shown), or via a network connection. The data store 232 may be, without limitation, a file server itself, a network aware storage (NAS), or a database.

[0045] Two applications 230 on the server may be an image processing server 114, and a workflow server 116. An image processing server 114 is described in co-pending application serial number 15/271,569, entitled "Hi Fidelity Computer Object Recognition Based Horticultural Feedback Loop," and is herein incorporated by reference. Workflow in the workflow server 116 is described in greater detail with respect to Figure 4.

Dispatching Device

[0046] A dispatching device 122 is any computing device that can participate on a network. Generally a dispatching device 122 is a mobile device such as a laptop, a tablet computer and/or a cell phone. As with any computing device, a dispatching device 122 will have a processor 240 and a memory 242 with analogous characteristics to the host computer for the telemetry capture function 220 as described above. The dispatching device 122 may include a display 244 to display output and an input 246 to receive user input. In some embodiments, the display 244 and the input 246 may be integrated together, for example like a touch screen in a tablet computer or smart phone. Note also that a laptop may also have a touchscreen as well as a keyboard for input thereby providing multiple inputs 246. To participate on a network, the dispatching device 122 will have a network interface 248, analogous to the network interface 212 described above.

[0047] In the dispatching device 122 memory 242 is an operating system 250, and a number of application components to support dispatch device functions. Those components may include a dispatcher software component 252, a status receiver software component 254, an analyzer software component 256 and a reporter software component 258.

[0048] The dispatching device 122 provides the software for an administrative person 118 and 120 to initiate a feedback loop. Specifically, the dispatcher software component 252 is used to dispatch remedial courses of action, and the status receiver software component 254 is used to receive status updates on the dispatched remedial courses of action. The operation of the dispatcher software component 252 and the status receiver software component 254 is described in further detail with respect to Figure 4.

[0049] The dispatching device 122 also provides the software for an administrative person 118 and 120 to analyze and diagnose potential issues in plants and horticultural operations in general. Accordingly, the analyzer software component 256 may facilitate analysis on images of interest, including performing static analysis and sequential analysis. The reporter software component 258 may report issues to bring to the attention of an administrative person 118 and 120. Such reporting may be in the form of providing queries on notifications and premade reports. Static analysis and sequential analysis is described in co-pending application serial number 15/271,569, entitled "Hi Fidelity Computer Object Recognition Based Horticultural Feedback Loop," and is herein incorporated by reference. Analysis may also include review of images of interest described below with respect to Figure 6. Example reporting is described in further detail below with respect to Figure 8.

Line Worker Device

[0050] A line worker device 126 is any computing device that can participate on a network. Generally a dispatching device 126 is a mobile device such as a laptop, a tablet computer and/or a cell phone. As with any computing device, a dispatching device 126 will have a processor 260 and a memory 262 with analogous characteristics to the host computer for the telemetry capture function 220 as described above. The line worker device 126 may include a display 264 to display output and an input 266 to receive user input. In some embodiments, the display 264 and the input 266 may be integrated together, for example like a touch screen in a tablet computer or smart phone. Note also that a laptop may also have a touchscreen as well as a keyboard for input thereby providing multiple inputs 266. To participate on a network, the line worker device 126 will have a network interface 268, analogous to the network interface 212 described above.

[0051] In the line worker device 126 memory 262 is an operating system 272, and a number of application components to support line worker functions. Those components may include a task receiver software component 272, a notifier software component 274 and an annotation software component 274.

[0052] The line worker device 126 is to include software to enable line workers 242 to receive remedial course of action, and to provide feedback. The feedback is then sent via the line worker device 126 to the workflow server 116 for future distribution to administrative workers 118 and 120. [0053] The task receiver software component 272 receives dispatches 130 from administrative personnel 118 and 120 and notifications 128 from the workflow server. Notifications 128 and dispatches 130 are then queued by the task receiver software component 272. The line worker indicates on the line worker device 126 when a task is complete, or otherwise provides status. The notifier software component 274 manages the transmission of task status to the workflow server 116. In some cases, the line worker may provide textual, image, and other information about the task. The line worker may annotated the task or alternatively an image of interest associated with a task with the annotation software component 276. Workflow for the line worker 124 and the line worker device 126 is described in further detail with respect to Figure 7.

Image of Interest

[0054] Prior to discussing the operational flow of horticultural care tracking validation and verification, it is useful to describe the notion of an image of interest. Figure 3 is a diagram 300 of an image of interest 302. Specifically, an image of interest 302 is not merely an image, but is an image with an identified target 304 usually a plant, an identified artifact 306 of the target, and potentially associated with an identified issue record 308.

[0055] A target 304 in the image of interest 302 is the subject of the image. When an image 302 is captured by an image capture device 110, the image capture device 110 is targeting a real world object. The image 302 will have portions (e.g. pixels) capturing the subject of the image, and the image 302 will have portions capturing the background and/environment surrounding the subject of the image and other items that are not the subject of the image. The target 304 in the image of the interest 302 are the portions of the image 302 that capture the subject of the image. [0056] When separating the target 304 in the image of interest 302 from the rest of the image, typically image segmentation techniques are applied by the workflow server 116 or other servers. Generally the portion of the image 302 relating to the target 304 will be subdivided into different image segments. Accordingly, in a horticultural operation, the plant/target 304 in an image 302 will itself be segmented. Generally each segment will correspond to a portion of the plant 304. For example, individual leaves, flower petals, stems and branches may be discerned from individual image segments. Each image segment corresponding to a feature in a target is an artifact. Accordingly, an image of interest 302 not only contains a target 304, but also can subdivide out artifacts 306, i.e. image segments corresponding to target features.

[0057] Images processed by the telemetry capture function 204 and the workflow server 1 16 in a horticultural information will almost always contain a target 304, generally a plant. What makes an image an image of interest 302 is that the image is a candidate for the workflow server 116 to bring to the interest of an administrative person 118 and 120 via a dispatching device 122. This is accomplished by making an image of interest 302 capable of association with an issue 308. Note that an image may be an image of interest 302, but will not yet have been associated with an issue. Rather an image of interest 302 will have the potential to be associated with an issue record 308 in a persistent fashion. This may be accomplished in several ways. Images 302 may be stored in a relational database in an image table and issue records 308 may be stored in an issue table. A cross- reference table relating images to issues would then store associations of images to issues. Alternatively, an image 302 may store a pointer to an issue record 308 as part of the image's container.

[0058] An issue record 308 is any representation, usually in text, that describes an anomaly in the target/plant 302. An issue record 308 may also include information about the likely severity of the issue, thereby aiding in ranking issues. An issue record may also include annotation information, provided by administrative personnel 118 and 120 and/or line workers 124. Alternatively, annotations may be stored separately, but associated with the image 302 and/or issue record 308. As with the issue record 308 itself, an association of an annotation may be via a cross-reference table or a pointer as described above.

[0059] The workflow server 116 may create an issue record 308 automatically by performing image processing on the image 302. Alternatively, an administrative person 118 and 120 may browse images 302 and create an issue record 308 via the dispatching device 122. When an issue record 308 is created, it is associated by the workflow server 116 with the image 302.

[0060] In addition to identifying issues, the workflow server 116 may also store a table of remediation courses of action associated with issues. In this way, where an issue record 308 is associated with an image 302, the issue record 308 may be used to query the workflow server for at least one remediation course of action.

[0061] Upon the association of an issue record 308 with an image 302, the workflow server 116 may send a notification 128 to the dispatching device 122, or multiple administrative persons 118 and 120. In this way, an image of interest 302 contains image segments comprising the target/plant 304 and at least one artifact 306, and may be associated with an issue record 308 enabling the workflow server 116 to trigger notifications about the issue 308.

Issue Identification and Remediation Course of Action Dispatching

[0062] In a horticultural feedback loop, issues may be identified from many sources including line workers 124, the workflow server 116 and its automation, and administrative personnel 118 and 120. Suggested remediation courses of action may also come from many sources. However, the decision whether to execute on a remediation course of action generally is done by an administrative person 118 and 120. This process may be done from a dispatching device 122. Figure 4 is a flow chart 400 of an exemplary process to identify issues and dispatch remediation courses of action.

[0063] In block 402, a dispatching device 122 receives a plurality of images of interest. Per the discussion with respect to Figure 3, an image of interest 302 in a horticultural feedback loop includes a plant 304 and an artifact of the plant 306. An image of interest 302 may be associated with an issue record 308 as well.

[0064] In block 404, the dispatching device 122 may display at least one of the received images. The dispatching device 122 may select an image to display based on a ranking of issues associated with the received images. Images 302 that are associated with issue records 308 may be ranked by severity of issue. Ranking issue records by priority and/or severity is described with respect to Figure 3. In some cases, a user may browse through multiple images prior to selection, or may look at images in sequence. Various user interface patterns in reviewing multiple images is described in further detail with respect to Figure 6.

[0065] Alternatively, the dispatching device 122 may select an image to display based on an ordering by time or other grouping or an image to be displayed may be selected from a user query.

[0066] A user may desire to add information to an image. Such information is called an annotation. Annotations may simply be markers such as "pin drop" where a marker is placed on an image to enable finding the image again quickly. Other annotations may be graphical marks on the image, such as circling artifacts or adding notes. Yet other annotations may be text commentary. Annotation data may be stored as associated with an image or with an issue. This association is described with respect to Figure 3. [0067] Annotations are optional, but in block 406, the user may indicate a desire to add an annotation by making a user interface gesture to select an image. User interface gestures are acts that trigger events in software to indicate input. For a tablet computer, a tap on the image constitutes one possible user interface gesture. For a laptop with a graphical user interface, a mouse click on an image may constitute a possible user interface gesture.

[0068] Once an image is selected with a user interface gesture, a user may enter the annotation information 408. Generally, as per block 410, the annotation information is displayed either while the annotation information is entered, or soon thereafter. At this point the annotation information is associated and stored with the relevant issue and/or image.

[0069] Displayed images may also display issues associated with the image. In block 412, an issue that has been identified, and is associated with the image may be displayed. The workflow server 116 stores remediation courses of action for issues. A remediation course of action may comprise one or more tasks. For a particular issue record 308 associated with an image 302, the workflow server 116 may be queried for the associated remediation courses of action. The association of remediation courses of action to issues is described with respect to Figure 3. One, some, or all remediation courses of action, or their constituent tasks, may then be retrieved and displayed on the dispatching device 122.

[0070] At this point, an image with a plant has been selected, where the image is displayed along with a present issue under consideration. In block 416, the user may then make a user interface gesture to either select a remediation course of action associated with the present issue or in block 418 may specify an alternative course of action. Where an alternative course of action is specified, the alternative may be stored with the working server, and associated with the present issue. In this way, the alternative may be provided for images with the same issue. Over time, an administrator may update the list of remediation courses of action as stored in the workflow server 116.

[0071] Now by virtue of selecting an image, a plant and a present issue have been selected and a remediation course of action has been specified. The dispatching device 122 will use the dispatching software component 252 to dispatch the specified remediation course of action. To do so, the user may specify a deadline. Specifically, in block 420, by specifying a time threshold to the dispatching software component 252 for when the remediation course of action is to be performed. One a deadline is specified, the user may make a user interface gesture for the dispatching software component 252 to dispatch the remediation course of action to the workflow server 116. The workflow server 116 will then identify candidate line workers 124 associated with the selected plant in the selected image and with matching qualifications to perform the dispatched remediation course of action. Where there are multiple candidate line workers 124 to perform the remediation course of action, the workflow server will schedule the remediation course of action to the first available worker. In block 422, once the workflow server 116 dispatches the remediation course of action, the dispatching device 122 may graphically indicate a successful dispatch.

[0072] As time passes, line workers 124 will perform tasks and send notifications as to whether the work has been performed. The workflow server 116 will forward those notifications to the status receiver software component 254 in the dispatching device 122. The status receiver software component is a notifications subscriber. Whenever a remediation course of action is dispatched, by the dispatching software component 252, the status receiver software component 254 notifies the workflow server 116 that it is to subscribe to all notifications associated with that dispatch. [0073] As time passes, the workflow server 116 then sends notifications back to the status receiver software component 254. Some notifications are that the task was completed. Other notifications is that the task in progress but there was a complication. If, per block 424, such a notification arrives, per block 426 the dispatching device 122 will graphically indicate that the dispatched course of action was performed. This may be by removing the dispatched course of action from a list, adding a checked mark icon to the course of action, or some other graphical indicia.

[0074] Dispatched courses of action are associated with deadlines. As time passes, when the specified time arrives, per block 428, any courses of action not performed may be reported to a user. In this way, the user may follow-up if needed.

[0075] Accordingly, the workflow server 116 provide images to a dispatching device 122, and display at least one image. The user may select an image, and with it a plant and an associated issue. The associated issue may be used to retrieve remedial courses of action from the workflow server 116. The user may then select a remedial course of action or specify an alternative for dispatch. The course of action is dispatched via a dispatcher software component 252, and notifications about the dispatch are received with a status receiver software component 254, thereby completing the horticultural feedback loop.

Collaboration

[0076] Large scale horticultural operations may be spread over geographically distant greenhouses 104. Growers 120 and master growers 118 may be geographically separated. The dispatching devices 122 are configured to facilitate remote collaboration.

[0077] Collaboration may have a communications mode, which is the type of communication. Common communications modes are voice, video and text. [0078] Collaboration may be in different degrees. A full screen may be shared or only part of a screen may be shared. Collaboration may be assymetric where only one user has control or symmetric where both users have equal control. This generally is implemented by sharing privileges on the two collaborating dispatching devices 122 as described below.

[0079] Figure 5, is a flow chart 500 of an example initiation of a collaboration session.

[0080] In block 502, a dispatching device 122 receives a notification for a request to communicate. The notification contains the identity of the requestor and potentially the network address of the requestor's dispatching device 122. The notification will also indicate the mode of communications such as video conference, voice, screen sharing as enumerated above.

[0081] In block 504, the receiving user may make a user interface gesture on his or her dispatching device 122 to accept the communications request. Alternatively the user may cancel or dismiss the request. If the communications request is communicated, the mode of communications is established. In either case, in block 506, the communications request reply is sent.

[0082] If the communications request is accepted, the dispatching device 122 performs a communications channel handshake with a communications server, either in the workflow server 116 or connected to the workflow server 116. In this way, per block 508, a communications channel is established. The communications server matches the mode of communication e.g. voice, video, or text.

[0083] Once the communications channel and a communications mode is established, collaboration privileges are exchanged in block 510. Privileges for a display may be set to not be shared, to be partially shared, or to be completely shared. Privileges for input may be to allow the other user to control the screen, and to enter annotations. Where privileges for both users are the same, the privileges exchanged are symmetric. Otherwise, the privileges are asymmetric.

[0084] For horticultural feedback, privileges may be assigned to a shared image of interest. The privileges set may be to annotate the image, to associate an image with an issue, and to dispatch a remedial course of action. Privileges specific to an image of interest are exchanged in block 512.

[0085] After the communications channel for the selected communications mode is established, and privileges are exchanged, the collaboration communications may proceed.

User Interface Patterns for Displaying Multiple Images

[0086] When the workflow server 116 sends images to a dispatching device 122 a user may desire to review multiple images in sequence. There are various user interface patterns involving multiple images. Figure 6 is a flow chart 600 of some exemplary patterns.

[0087] In block 602, the dispatching device 122 receives several images of interest from the workflow server 116. The images are associated with a date time stamp, usually embedded in the image's file format or metadata. The dispatching device 122 may take at least two of the images, a first image with a first date time stamp and a second image with a second date time stamp. One of the images will usually, but not always be an image captured later in time than the other.

[0088] The dispatching device 122 may support multiple patterns to display the two images. One may be to show the images side by side. Another is to overlay the images and highlight differences. In block 604, the user selects, via a user interface gesture, which user interface pattern for the dispatching device to use.

[0089] By way of example, one pattern is to display one image and use a toggle to display the second image. In block 606, the user interface gesture is interpreted by the display device 122 as a request to toggle images. Accordingly, in block 608, the display device 122 will display the second image responsive to the toggle request.

[0090] By way of another example, in block 610, the user interface gesture is interpreted to display multiple images in the context of a chart over time. Specifically, a chart shows telemetry over time. Telemetry may include vapor pressure deficit, daily light integral and relative photosynthesis index. As the user selects a spot on the chart, per block 612, an image that most closely corresponds to the time of that spot is selected, and per block 608 is displayed. Telemetry values for that time may also be displayed. In this way, the image of the plant being measured by the telemetry is displayed along with the telemetry.

[0091] In general, different user interface patterns are supported by the dispatching device 122 to enable users to review multiple images.

Line Worker Device Horticultural Care Tracking

[0092] Thus far the description of the horticultural care feedback loop has been about the dispatching of remedial courses of action. Courses of action are dispatched by the workflow server 116 to line workers 124 via their line worker devices 126. Figure 7 is a flow chart 700 of the operations from the perspective of the line work 124 and the line worker device 126.

[0093] As described with respect to Figure 4, remedial courses of action may comprise multiple tasks. When a course of action is dispatched from a dispatching device 122, the course of action is sent to the workflow server 116. The workflow server 116 retrieves the tasks comprising the course of action. Those tasks are to be scheduled to at least one line worker 124. The workflow server 116 identifies line workers 124 are working on the plant associated with the course of action, and have the capabilities to perform the task. Where there are multiple candidate line workers 124, the workflow server 116 load balances work by giving the task to the first available line worker 124, or to the line worker 124 with the smallest workload. Where there are no qualified line workers, the workflow server 116 will send a notification to the dispatching device 122 that initiated the course of action.

[0094] When a task is dispatched to a line worker 124, in block 702 the line worker's device 126 receives the task with the task receiver software component 272. The task receiver software component 272 then sorts the tasks by priority and per block 704 displays the queued tasks.

[0095] Often, line workers 124 have widely varying primary languages and educational skills. Accordingly, in block 704, the displayed queued tasks are shown via icons. An icon is a graphical representation or symbol. Here the icons displayed are symbols of known tasks, such as watering, pruning, increasing light, and harvesting. The set of icons, or symbols to indicate the various tasks is called an iconographic language. The iconographic language may include numbers and/or values. For example, an icon to water may be accompanied by an indication of the amount of water to use. However, because the tasks do not rely on a primary language, the iconographic language may provide a lingua franca between the administrative personnel 118 and 120 and the line workers 124.

[0096] The line workers 124 will then perform the queued tasks. As those tasks are completed, the line workers 124 may make a user interface gesture per block 706 to indicate that the task was performed. Upon receiving the gesture, the line worker device 126 will send a notification via the notifier software component 274 to the workflow server 116 that the task was performed. Generally the notification will include a date time stamp indicating when the task was reported as complete. This information will later be used during verification and validation as described with respect to Figure 8. [0097] Since the dispatching device 122 that dispatched the task enlisted in notifications relating to the course of action at time of dispatch, the dispatching device 122 may receive the notification that the task was complete.

[0098] In the meantime, per block 710, the line worker device 126 will graphically indicate that the task was performed. This may be by removing the task, changing the color, or adding a completeness icon such as a check box.

[0099] In some cases, the line worker 124 may be prevented from completing the task. In this event, the line worker may select an icon that corresponds to common impediments to completing a task. Examples may include insufficient fertilizer, power outage and the like. As with receiving tasks, the status indicators use solely an iconographic language. Upon receiving a user gesture selecting the icon, the line worker device 126 dispatches a notification corresponding to the selected icon via the notifier software component 274.

[00100] Some line workers 124 may desire to send a more complete explanation beyond the limitations of an iconographic language. In these circumstances, the line worker 124 may supplement the notification with text and/or media images via the annotation software component 274. The annotation software component provides a user interface to enter text messages and/or attach multimedia files, such as images. Annotations may be sent as part of notifications or may be sent separately. Where an annotation is sent without a notification, a generic notification will be generated by the workflow server 116 so that the dispatching device 122 may be notified of the annotation, and may receive the annotation.

Horticultural Reporting, Validation and Verification

[00101] Returning to the perspective of the administrative personnel 118 and 120, the dispatching device 122 has subscribed to notifications relating to any dispatched remedial courses of action. Specifically the status receiver software component 254 subscribes to notifications. So when a line worker device 126 indicates that a task is complete, the workflow server 116 routes the notification to the dispatching device 122. However, having a task reported as performed is not the same as the task having been performed properly or completely. Accordingly, the dispatch device 122 supports validation and verification of tasks. Figure 8 is a flow chart 800 of an exemplary process to do so.

[00102] In block 802, a dispatching device 122 receives a plurality of tasks to be performed for a particular plant. Generally this occurs when a user has the dispatching device 122 query the workflow server for all pending tasks for a specified plant. However, tasks may be queried from other criteria, such as by remedial course of action, by line worker, by grow operation and the like. In block 804, the workflow server 116 also provides the most recent notifications relating to those tasks.

[00103] To verify that a task was done or done properly, a user may have the dispatching device 122 request an image of the subject plant just before the task was reported as done and a second image of the subject plant just after the task was reported as done. In block 806, the images may be reviewed, for example via the patterns as described with respect to Figure 6, and a user may determine whether the task was done properly, or if the task and/or remedial course of action was affected. With this information, a user may modify remedial courses of action, and may corrections if needed.

[00104] Block 806 need not be manual. For large horticultural operations, it is not practical for administrative personnel 118 and 120 to manually review all tasks. Accordingly, in block 806, a workflow server 116 may work with an image processing server 114 to compare the image prior to the task and the image after the task. If there are no noticeable differences, then either the task was not performed correctly, or was not performed at all. Alternatively, if the plants continue to display artifacts with issues, then the task was not performed or was not effective. In general trend analysis and sequential analysis may be used to determine the efficacy of dispatched tasks and/or courses of action. Sequential analysis is described in co-pending application serial number 15/271,569, entitled "Hi Fidelity Computer Object Recognition Based Horticultural Feedback Loop," and is herein incorporated by reference.

[00105] In some cases, the user may detect fraudulent reporting. In extreme cases, there may be indications of theft. To detect these scenarios, in block 808, the workflow server 116 in conjunction work with the image processing server 114 may determine that the images do not agree with what was reported by the line workers 125. For example, the image processing server 114 may determine that the plant is no longer in the image, even though the plant is reported as healthy and in place. The workflow server 116 may then search other images for the plant, and if not found, in block 810 interpret the images as a stolen plant. In this circumstance, a notification of a stolen plant may be sent by the workflow server to the dispatching device 122.

[00106] Accordingly, the workflow server 116 has collected notifications about tasks and remedial courses of action. Not only are notifications provided to administrative personnel 118 and 120, the workflow server 116 in conjunction with the image processing server 114 may compare images proximate in time to tasks being performed, and may interpret the images for efficacy, proper performance of tasks, or in extreme cases theft. In this way, administrative personnel 118 and 120 may focus on the most impactful notifications in the horticultural feedback loop.

Conclusion

[00107] Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

CLAIMS WHAT IS CLAIMED IS:

1. A method to evaluate detected anomalies of a plant, comprising:

receiving a plurality of images corresponding to a plurality of plants respectively, wherein at least one of the plurality of images is an image of interest containing an identified plant, an identified artifact, and is associated with an identified issue of the identified plant corresponding to the identified artifact; and

displaying the image of interest along with the identified issue, wherein the identified issue is based on an analysis of the identified artifact.

2. The method of claim 1, comprising:

identifying at least one course of action to address the identified issue;

displaying the at least one course of action;

receiving a selection of the at least one course of action; and

graphically indicating that the at least one course of action has been dispatched to a line worker.

3. The method of claim 1, comprising:

displaying at least one identified course of action to address the identified issue; receiving an indication that at least one identified course of action is not accepted; specifying an alternate course of action;

storing the alternate course of action; and

graphically indicating that the alternate course of action has been dispatched to a line worker.

4. The method of claim 2, compri

receiving a notification that the at least one course of action dispatched to the line worker has been performed; and

graphically indicating that the at least one course of action has been performed.

5. The method of claim 2, compri

specifying a time threshold as to when the at least one course of action dispatched to the line worker is to be performed; and

upon determining that a notification that the at least one course of action has been performed has not been received within the time threshold, triggering an alert; and

graphically indicating that the at least one course of action has not been performed.

6. The method of claim 3, comprising:

receiving a user interface gesture selecting an image of interest;

displaying, responsive to the user interface gesture, at least one additional image related to the image of interest.

7. The method of claim 6, wherein displaying is performed on both a local display and a remote display, and is initiated from a communicated request to perform a collaborative review.

8. The method of claim 1, wherein the identified artifact is displayed with a graphical indicia distinguishing the identified artifact from other portions of the image of interest.

9. The method of claim 1, wherein the image of interest is a first image of interest, and wherein at least one of the plurality of images is a second image of interest containing the identified plant and displaying the second image of interest.

10. The method of claim 9, comprising:

receiving a user interface gesture to toggle the displaying of the image of interest; and

displaying the second image of interest responsive to the receiving of the user interface gesture.

11. The method of claim 10, comprising displaying a chart that indicates changes in at least one parameter of the identified plant over a period of time;

wherein the user interface gesture is a selection on the chart, and wherein the second image of interest corresponds to the period of time represented by the selection on the chart.

12. The method of claim 9, wherein the first image of interest or the second image of interest represents a baseline image of the identified plant.

13. The method of claim 1, comprising:

receiving a user interface gesture to annotate an image of interest;

receiving annotation information; and

graphically displaying the annotation information and a graphical link to the image of interest.

14. A method to queue tasks to a horticultural line worker, comprising:

receiving at least one task to perform on a plant; and

displaying the task to perform graphically using solely an iconographic language.

15. The method of claim 14, comprising:

receiving a user interface gesture indicating that the at least one task to be performed is complete;

sending a notification that the at least one task to be performed is complete; and graphically indicating on a display that the at least one task has been performed.