JP2021114645A - Processing device, imaging apparatus, processing method, and program - Google Patents

Abstract

To provide a processing device capable of gripping an article state with a simple structure in a low consumption power, an imaging apparatus, a processing method, and a program.SOLUTION: A processing device includes: an acquisition part that acquires information related to a height of an article in a first image acquired by photographing at a first date, information related to a height of an article in a second image acquired by photographing at a second date, and information related to a distance from a reference surface of the height of the article at the second date to an optical system article side main point used for photographing; and a calculation part that calculates at least one of information related to an analysis region in the first image by using the information acquired by the acquisition part and the information related to an optical system focal distance at the first date.SELECTED DRAWING: Figure 1

Description

The present invention relates to a processing device, an imaging device, a processing method, and a program.

Conventionally, a system for grasping a change in a plant's growth state from a fixed-point image of a plant has been known. For example, in paddy rice cultivation, this system can diagnose growth indicators such as leaf color, number of stems, and plant height from images of paddy rice, and can grasp the growth state of paddy rice at each growth period from the transition of the growth indicators.

However, in fixed-point photography, the shooting range of paddy rice changes when the plant height of paddy rice changes. When the growth index is diagnosed from an image in which the imaging range of paddy rice has changed, the diagnostic value of the growth index is affected not only by the change in the growth state of the paddy rice but also by the change in the imaging range of the paddy rice, so the growth condition of the paddy rice is appropriate. I can't figure it out.

Patent Document 1 describes a height measuring device that measures the height of a plant at the time of imaging, an elevating mechanism that raises and lowers at least one of the imaging device or a table on which a plant is placed, and a control device that controls the raising and lowering operation of the elevating mechanism. An image generator including the above is disclosed. In the image generator of Patent Document 1, in order to keep a predetermined range of the plant within the field of view, the elevating operation of the elevating mechanism is controlled so that the distance between the image pickup device and the plant at the time of initial setting is kept constant. NS.

Japanese Patent No. 6172657

Ryoji Sasaki et al., "Estimation of plant height of transplanted paddy rice based on maximum and minimum temperatures", Journal of the Crop Science Society of Japan, Vol. 69 (Appendix No. 1), p. 166-167, 2000,

Since the image generator of Patent Document 1 consumes a large amount of power to drive the elevating mechanism, it is difficult to install it in a field such as a paddy field or a field where there is little power supply equipment. In addition, an elevating mechanism and an external power source such as a solar cell or a battery for driving the elevating mechanism are required, which complicates the configuration.

An object of the present invention is to provide a processing device, an imaging device, a processing method, and a program capable of grasping the state of an object with low power consumption and a simple configuration.

The processing apparatus as one aspect of the present invention includes information on the height of an object in the first image acquired by photographing at the first date and time, and the second image acquired by photographing at the second date and time. An acquisition unit that acquires information on the height of the object and information on the distance from the reference plane of the height of the object at the second date and time to the principal point on the object side of the optical system used for photographing, and the acquisition unit acquires the information. It is characterized by having a calculation unit for calculating at least one of the information regarding the analysis region in the first image and the information regarding the focal length of the optical system at the first date and time using the obtained information.

Further, the processing method as another aspect of the present invention includes information on the height of the object in the first image acquired by photographing at the first date and time, and the second image acquired by photographing at the second date and time. The step of acquiring the information on the height of the object in the image of the above, and the information on the distance from the reference plane of the height of the object at the second date and time to the principal point on the object side of the optical system used for photographing, and the first step. Using information about the height of the object at the date and time, information about the height of the object at the second date and time, and information about the distance from the reference plane to the principal point on the object side of the optical system at the second date and time, It is characterized by having a step of calculating at least one of the information about the analysis area in the first image and the information about the focal length of the optical system at the first date and time.

According to the present invention, it is possible to provide a processing device, an imaging device, a processing method, and a program capable of grasping the state of an object with low power consumption and a simple configuration.

It is a figure explaining the difference in the photographing range due to the difference in the height of a plant. It is a block diagram of the image processing apparatus of Example 1. FIG. It is a flowchart which shows the processing flow of the image processing apparatus of Example 1. FIG. It is a hardware block diagram of the image processing apparatus of Example 1. FIG. It is a block diagram of the image pickup apparatus of Example 2. It is a figure explaining the difference in the photographing range due to the difference in the height of a plant. It is a flowchart which shows the processing flow of the image pickup apparatus of Example 2.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same member is given the same reference number, and duplicate description is omitted.

First, an outline of the present invention will be described before giving a specific description of each embodiment.

FIG. 1 is a diagram for explaining the difference in the photographing range due to the difference in the height of the plant, and shows how the fixed point image of the plant taken from above is acquired by using the imaging unit installed directly above the plant. ing. The image pickup unit has an image pickup optical system and an image pickup element. Focal length of the imaging optical system is f, the distance from the ground (reference plane of the object height) to the object side principal point of the imaging optical system (hereinafter, the imaging section height) is fixed at H _c.

1 (a) is a low height of the plants (plant height is H ₁₎ shows a state. At this time, the object distance is D ₁ . The fixed point image I ₁ represents an image acquired by the image sensor. The image analysis region ROI ₁ represents a corresponding region in the fixed point image I ₁ in a range surrounded by a square having a side 2R about the optical axis of the imaging optical system indicated by the alternate long and short dash line.

1 (b) is higher plant height (the height of the plant is H ₂₎ shows a state. At this time, the object distance is D ₂ . The fixed point image I ₂ represents an image acquired by the image sensor. The image analysis region ROI ₂ represents a corresponding region in the fixed point image I ₂ in a range surrounded by a square having a side 2R about the optical axis of the imaging optical system indicated by the alternate long and short dash line.

As shown in FIGS. 1 (a) and 1 (b), when the height of the plant changes, the image analysis area in the fixed point image differs even if the range of the same area is set as the photographing range. Therefore, if the information from the fixed image analysis area is used, the growth state of the plant cannot be properly grasped. For example, when the growth state of a plant is _{grasped from the fixed point images I 1} and I ₂ , if the image analysis area in both fixed point images is _{set to ROI 1} , the image analysis is performed for the image areas in different shooting ranges.

When grasping the growth state of a plant from a fixed-point image, the statistical value (average value, median value, etc.) of the growth state of the plant in the image analysis area is used as a representative value of the field in order to reflect the growth state of the entire field. It is calculated. Therefore, when image analysis is performed on image areas in different shooting ranges, the representative value of the field is affected not only by the change in the growing state of the plant but also by the change in the shooting range. It becomes difficult.

For example, when the plant is rice or wheat, the rice strain or wheat strain whose growth state is grasped is determined. Therefore, when image analysis is performed on the image areas of different shooting ranges, the rice strain or wheat strain whose growth state is grasped is determined. Changes, and it is not possible to properly grasp the growth state of rice and wheat. In addition, the number of stems, which is one of the growth indexes when diagnosing the growth of rice and wheat, is often evaluated by the number of stems per unit area. The number of stems cannot be diagnosed properly.

Hereinafter, details of each embodiment of the present invention will be described.

In this embodiment, the image processing apparatus 100 that sets the image analysis region in the fixed point image of the plant according to the height of the plant will be described.

FIG. 2A is a block diagram of the image processing device 100. The image processing device 100 includes an input unit 101, a storage unit 102, a calculation unit 103, and a processing unit 104. As shown in FIG. 2B, the calculation unit 103 includes an acquisition unit 103a, a calculation unit 103b, and a setting unit 103c. The arithmetic unit 103 may include a processor and a memory programmed to execute the function. Further, although the calculation unit 103 is provided in the image processing device 100 in this embodiment, it may be configured as a processing device different from the image processing device 100.

Storage unit 102, a specific date and time, information on the distance from the ground to the object side principal point of the imaging optical system (imaging unit height) H _ics, image in the information H _i1, and the fixed-point image I ₁ relating to plant height _{The information ROI i1} regarding the analysis area is stored. Information about the height of the plant _Hi1 is information in the second image acquired by taking a picture at a specific date and time, and is, for example, distance information measured by using a stereo camera or a ToF (Time of Flight) camera. Obtained using. _{Further, as information Hi1} regarding the height of the plant, as shown in Non-Patent Document 1, the height of the plant at an arbitrary time is used as the initial value, and the temperature data from the measurement date of the initial value to the m-day is used. Then, information on the height of the plant on the m-day may be obtained.

Information about the image analysis area ROI _i1 is, for example, a label image in which pixels corresponding to the image analysis area are labeled with 1 and pixels not corresponding to the image analysis area are labeled with 0. When the image analysis area is a polygon, the coordinate information of each vertex of the polygon is also included in the _{information ROI i0 regarding the image analysis area.}

FIG. 3 shows the present implementation when the height of the plant included in the imaging range corresponding to the image analysis area ROI _{1 in the fixed point image I 1} at a specific date and time changes from H _{1 to H 2 as shown in FIG.} It is a flowchart which shows the processing flow of the image processing apparatus 100 of an example.

In step S110, obtaining unit 103a, the input unit 101, and acquires an arbitrary time (first time) information _{H i2} to high _{H 2} of the plant in the first image obtained by shooting at.

In step S120, obtaining unit 103a from the storage unit 102, at a particular date and time (second time), the information _{H ics} related to the imaging unit height, information on plant height _{H i1,} and the image of fixed-point _{I 1} The information ROI _i1 regarding the image analysis area ROI ₁ is acquired.

The order of steps S110 and S120 may be reversed.

In step S130, the calculation unit 103b calculates the information ROI _i2 regarding the image analysis region ROI _{2 in the fixed point image I 2 at} a specific date and time by using the information acquired in step S110 and step S120.

Specifically, when the boundary coordinates _{of the image analysis area ROI 1} when the center of the fixed point image is the origin is _{(x 1} , y ₁ ), the calculation unit 103b uses the following equations (1) to (4). calculating the arbitrary date and time of the image analysis area ROI ₂ boundary coordinates _(x 2, _{y 2)} with. Here, the boundary coordinates are coordinates representing the boundary between the image analysis area and the area other than the image analysis area (non-image analysis area).

Information about _{the image analysis region ROI 2 in FIG. 1 (b) ROI i2} by calculating the boundary coordinates (x ₂ , y ₂ ) for all the boundary coordinates (x ₁ , y ₁ ) in FIG. 1 (a). Can be calculated.

After the flow of FIG. 3 has been completed, setting unit 103c sets the image analysis area in fixed point image _{I 2} in the image analysis area ROI ₂ based on information _{ROI i2} related to the image analysis region ROI ₂ calculated. The processing unit 104 grasps the change in the growing state of the plant by using the information from the _{image analysis area ROI 2.}

As described above, in the present embodiment, the image analysis area is set so that the same imaging range is obtained when the height of the plant changes without providing the drive unit or the power source for moving the drive unit. As a result, it is possible to appropriately grasp the growing state of the plant with low power consumption and easily.

The image processing device 100 of this embodiment can be realized by, for example, the hardware configuration shown in FIG. FIG. 4 is a hardware configuration diagram of the image processing device 100. The image processing device 100 includes a system bus 301, a CPU 302, a RAM 303, a ROM 304, an HDD 305, an input unit 306, and a display unit 307 that transfer data between the configurations. The storage unit 102 can be mounted on the ROM 111 and the HDD 112, and the calculation unit 103 can be mounted on the CPU 302.

In this embodiment, the image pickup apparatus 200 that sets the focal length of the optical system according to the height of the plant will be described. The image pickup apparatus 200 is installed at a fixed point position.

FIG. 5 is a block diagram of the image pickup apparatus 200. The image pickup device 200 includes an image pickup unit 201, a height acquisition unit 202, a calculation unit 203, a storage unit 204, and a control unit 205. As shown in FIG. 5B, the calculation unit 103 has an acquisition unit 203a and a calculation unit 203b. The arithmetic unit 203 and the control unit 205 may each include a processor and a memory programmed to execute each function. Further, although the calculation unit 203 and the control unit 205 are provided in the image pickup apparatus 200 in this embodiment, they may be configured as processing apparatus different from the image pickup apparatus 200.

FIG. 6 is a diagram for explaining the difference in the photographing range due to the difference in the height of the plant, and shows how the image pickup unit 201 installed directly above the plant is used to acquire a fixed point image of the plant photographed from above. Shown. In FIG. 6, those having the same symbols as those in FIG. 1 have the same meaning, and thus the description thereof will be omitted here.

The image pickup unit 201 has an image pickup optical system and an image pickup element. In this embodiment, it is possible to change the focal length of the imaging optical system. As a means for changing the focal length, a variable focal length optical system (zoom lens) in which the focal length changes continuously may be used as the imaging optical system, or a configuration in which a plurality of optical systems having different focal lengths are switched is used. You may. Distance from the ground to the object side principal point of the imaging optical system (hereinafter, the imaging section height) is fixed at H _c.

6 (a) is a low height of the plants (plant height is H ₁₎ shows a state. At this time, the object distance is D ₁ . Storage unit 204, FIG. 6 state in (a specific date and time) of (a), information _{f i1} relates to the focal length _{f 1} of the imaging optical system, the information _{H i1} about the height _{H 1} of the plant, and the imaging optical system from the ground the distance to the object side principal point (imaging unit height) stores information about the H _c. Information H _i1 about the height H ₁ of the plant is the information in the second image obtained by shooting at a specific date and time.

In such a precondition, plant height, as shown in Fig. 6 (b) describes the processing of the image pickup apparatus 200 in the case of changes to H _2. FIG. 7 is a flowchart showing a processing flow of the image pickup apparatus of this embodiment.

In step S210, obtaining unit 203a, the height obtaining unit 202 obtains any time (first time) information _{H i2} to high _{H 2} of the plant in the first image obtained by photographing in .. The height acquisition unit 202 is composed of, for example, a stereo camera or a ToF (Time of Flight) camera. In this case, in step S210, a region corresponding to the plant is extracted from the distance image acquired by the stereo camera or the ToF camera, and information _Hi1 regarding the height of the plant is acquired based on the distance information of the extracted region. NS. Further, the height acquisition unit 202 is not limited to a stereo camera or a ToF camera, and may be, for example, a calculation processing device equipped with a program for calculating the height of a plant by the method described in Non-Patent Document 1. .. In the method described in Non-Patent Document 1, since the height of the plant is estimated using the temperature data, the calculation processing device may further include a temperature sensor for acquiring the temperature data. Further, the calculation processing device may have a communication means for acquiring the meteorological data at the point where the image pickup device 200 stored on the network is installed via the network.

In step S220, the acquisition unit 203a, from the storage unit 204, receives information _{Hic regarding the height of the imaging unit, information Hi1} regarding the height of the plant, and the focal length of the imaging optical system at a specific date and time (second date and time). Acquire _{information f 1c} regarding the distance.

The order of steps S210 and S220 may be reversed.

In step S230, the calculation unit 203b uses the information acquired in steps S210 and S220 to provide information f about the focal length f _{2 of the imaging optical system (when the height of the plant is H 2) at a specific date and time.} Calculate _i2.

Calculator 203b is specifically calculates the focal length f ₂ of the imaging optical system of a particular date and time using the following equation (5).

f ₂ = f ₁ (H _{c −} H ₂ ) / (H _{c −} H ₁ ) (5)
In the case where the object side principal point of the imaging optical system is changed according to the focal length of the imaging optical system, it is preferable to calculate the focal length f ₂ using the following equation (5a).

f ₂ = f ₁ (H _c1- H ₂ ) / (H _c2- H ₁ ) (5a)
Here, H _c1 is the distance _{from the ground corresponding to the focal length f 1} to the principal point on the object side of the imaging optical system, and H _{c 2} is the distance from the ground corresponding to the focal length f 2 calculated by the equation (5) to the imaging optical system. Is the distance to the principal point on the object side of.

Further, the calculation of the equation (5a) is performed until the focal length f ₂ converges _{, where H c2 is the distance from the ground corresponding to the focal length f 2} calculated by the equation (5a) to the principal point on the object side of the imaging optical system. It is even more preferable to repeat.

Further, based on the change amount data of the object side principal point relative to change in the focal length of the imaging optical system may be corrected in the focal length f ₂ calculated by the equation (5).

In step S240, the control unit 205 sets the focal length of the imaging optical system to f ₂ .

In step S250, the imaging unit 201 acquires an image of the plant.

FIG. 6 (b) shows a state after the flow of FIG. 7 is performed when the plant height is changed to H _2. At this time, the object distance is D ₂ . In FIG. 6 (b), the focal length of the imaging optical system is changed to f _2. As shown in FIGS. 6A and 6B, by setting the focal length of the imaging optical system according to the height of the plant, an image having the same shooting range can be obtained even if the height of the plant changes. Can be obtained.

As described above, in the present embodiment, the focal length of the imaging optical system is changed without requiring a large amount of electric power when the height of the plant changes without providing a large-scale drive unit. As a result, it is possible to appropriately grasp the growing state of the plant with low power consumption and easily.

In this specification, the object to be photographed is a plant, but it may be a growing object such as a fungus or a crystal. Moreover, although rice was described as an example of a plant, the plant of the present invention is not limited to rice. The present invention can be applied to any plant such as wheat whose height changes.
[Other Examples]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit that realizes one or more functions (for example, an ASIC (Application-Specific Integrated Circuit)).

Although the preferred embodiments and examples of the present invention have been described above, the present invention is not limited to these embodiments and examples, and various combinations, modifications and modifications can be made within the scope of the gist thereof.

103, 203 Calculation unit (processing device)
103a, 203a Acquisition unit 103b, 203b Calculation unit

Claims (14)

Information about the height of the object in the first image acquired by shooting at the first date and time, information about the height of the object in the second image acquired by shooting at the second date and time, and the first. An acquisition unit that acquires information on the distance from the reference plane of the height of the object to the principal point on the object side of the optical system used for the photographing at the date and time of 2.
It has a calculation unit that calculates at least one of information about an analysis region in the first image and information about a focal length of the optical system at the first date and time using the information acquired by the acquisition unit. A characteristic processing device. The acquisition unit acquires information regarding the analysis region in the second image, and obtains information.
The processing apparatus according to claim 1, wherein the calculation unit calculates information regarding an analysis region in the first image using the information acquired by the acquisition unit. The calculation unit sets the height of the object at the first date and time as H ₁ , the height of the object at the second date and time as H ₂ , and the optical from the reference plane at the second date and time. When the distance to the principal point on the object side of the system is H _c and the boundary coordinates of the analysis region in the second image are (x ₁ , y ₁ ),

Comprising processing apparatus according to claim 2, characterized in that to calculate the boundary coordinates of the analysis region in the first image using the equation (x _{2, y 2).} The processing apparatus according to claim 2 or 3, wherein the number of the objects included in the analysis region in the first image is equal to the number of the objects included in the analysis region in the second image. The acquisition unit acquires information regarding the focal length of the optical system at the second date and time, and obtains information about the focal length of the optical system.
The processing apparatus according to claim 1, wherein the calculation unit calculates information regarding the focal length of the optical system at the first date and time using the information acquired by the acquisition unit. The calculation unit sets the height of the object at the first date and time as H ₁ , the height of the object at the second date and time as H ₂ , and the optical from the reference plane at the second date and time. When the distance to the principal point of the system on the object side is H _c and the focal length of the optical system at the second date and time is f ₁ .
f ₂ = f ₁ (H _{c −} H ₂ ) / (H _{c −} H ₁ )
The processing apparatus according to claim 5, _{wherein the focal length f 2} of the optical system at the first date and time is calculated by using the above equation. The processing apparatus according to claim 5 or 6, wherein the number of the objects included in the first image is equal to the number of the objects included in the second image. It is further characterized by further having information on the height of the object on the second date and time, and a storage unit for storing the distance from the reference plane to the principal point on the object side of the optical system on the second date and time. The processing apparatus according to any one of claims 1 to 7. The processing apparatus according to any one of claims 1 to 8, wherein the object is a plant. The processing apparatus according to claim 9, wherein the plant contains at least one of rice and wheat. An imaging unit that acquires an image of an object,
An imaging device comprising the processing device according to any one of claims 1 to 10. The imaging apparatus according to claim 11, further comprising a height acquisition unit that acquires information regarding the height of the object at the first date and time. Information about the height of the object in the first image acquired by shooting at the first date and time, information about the height of the object in the second image acquired by shooting at the second date and time, and the first. A step of acquiring information on the distance from the reference plane of the height of the object to the principal point on the object side of the optical system used for the photographing at the date and time of 2;
Information on the height of the object on the first date and time, information on the height of the object on the second date and time, and the object-side principal point of the optical system from the reference plane on the second date and time. It is characterized by having a step of calculating at least one of the information about the analysis region in the first image and the information about the focal length of the optical system at the first date and time using the information about the distance to. Processing method to do. A program comprising executing the processing method according to claim 13.

Images