CN115953344B

CN115953344B - Image processing method, device, electronic equipment and storage medium

Info

Publication number: CN115953344B
Application number: CN202310213553.4A
Authority: CN
Inventors: 尚跃; 丁琪
Original assignee: Shanghai Ju Yue Electronics Co ltd
Current assignee: Shanghai Ju Yue Electronics Co ltd
Priority date: 2023-03-08
Filing date: 2023-03-08
Publication date: 2023-05-30
Anticipated expiration: 2043-03-08
Also published as: CN115953344A

Abstract

The invention discloses an image processing method, an image processing device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a profile to be used comprising a target object; determining positions to be used of at least two points to be triggered in the profile to be used under a first coordinate system; determining a position to be acquired of at least one image complement information under a first coordinate system based on each position to be used, the image attribute of the profile to be used and the at least one image complement information corresponding to the target object; controlling the target electron microscope to move to each position to be acquired so as to obtain a complementary image to be used corresponding to the corresponding position to be acquired; and determining a target complement image based on each complement image to be used and each complement image to be complemented. The problems of low image acquisition efficiency and poor quality caused by repeated image acquisition of the chip in the prior art are solved, automatic acquisition of complementary image is realized, convenience of image acquisition is improved, and image quality is improved.

Description

Image processing method, device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to an image processing method, an image processing device, an electronic device, and a storage medium.

Background

At present, due to the complexity of the internal circuit of the chip, when the internal circuit of the chip is analyzed, a high-precision chip image is required to be acquired based on devices such as an electron microscope and the like for analysis, but in the process of image acquisition, many external factors (such as environmental factors) may exist, so that noise exists in the acquired image, and the requirement of the internal circuit analysis of the chip is not satisfied.

In order to solve the above problem, in the prior art, a method of re-shooting a chip is adopted to collect a noise-free high-precision image for analyzing an internal circuit of the chip, but the shooting of the whole chip again may have poor image shooting quality, and further, the problem of poor analysis effect of the internal circuit may also result.

Disclosure of Invention

The invention provides an image processing method, an image processing device, electronic equipment and a storage medium, which are used for realizing automatic acquisition of complementary images, improving the convenience of image acquisition, enabling the finally obtained target object image to be of high quality, and further achieving the technical effect of improving the analysis accuracy of an internal circuit of the target object.

According to an aspect of the present invention, there is provided an image processing method including:

Acquiring a to-be-used profile including a target object, which is shot in advance; the to-be-used profile comprises at least two to-be-triggered points;

determining positions to be used of the at least two points to be triggered under a first coordinate system; the first coordinate system corresponds to the electronic microscope platform, and the target object is placed on the electronic microscope platform;

determining a to-be-acquired position of at least one image complement information under the first coordinate system based on each to-be-used position, the image attribute of the to-be-used profile and the at least one image complement information corresponding to the target object, which is determined in advance;

controlling the target electron microscope to move to each to-be-acquired position so as to obtain a to-be-used complement image corresponding to the corresponding to-be-acquired position;

determining a target complement image based on each of the complement images to be used and the complement images to be complemented; wherein the image to be complemented corresponds to the target object or a sampled object consistent with the target object.

According to another aspect of the present invention, there is provided an image processing apparatus including:

the to-be-used profile acquisition module is used for acquiring a to-be-used profile including a target object, which is shot in advance; the to-be-used profile comprises at least two to-be-triggered points;

The to-be-used position determining module is used for determining to-be-used positions of the at least two to-be-triggered points under a first coordinate system; the first coordinate system corresponds to the electronic microscope platform, and the target object is placed on the electronic microscope platform;

the to-be-acquired position determining module is used for determining the to-be-acquired position of at least one piece of image complement information under the first coordinate system based on each to-be-used position, the image attribute of the to-be-used profile and the at least one piece of image complement information which is predetermined and corresponds to the target object;

the to-be-used complement image determining module is used for controlling the target electron microscope to move to each to-be-acquired position so as to obtain a corresponding to-be-used complement image under the corresponding to-be-acquired position;

the target complement image determining module is used for determining a target complement image based on each to-be-used complement image and each to-be-complement image; wherein the image to be complemented corresponds to the target object or a sampled object consistent with the target object.

According to another aspect of the present invention, there is provided an electronic apparatus including:

at least one processor; and

A memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,

the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the image processing method according to any one of the embodiments of the present invention.

According to another aspect of the present invention, there is provided a computer readable storage medium storing computer instructions for causing a processor to execute the image processing method according to any one of the embodiments of the present invention.

According to the technical scheme, the to-be-used profile including the target object is obtained; determining positions to be used of at least two points to be triggered in the profile to be used under a first coordinate system; determining a position to be acquired of at least one image complement information under a first coordinate system based on each position to be used, the image attribute of the profile to be used and at least one image complement information corresponding to the target object, which is determined in advance; controlling the target electron microscope to move to each position to be acquired so as to obtain a complementary image to be used corresponding to the corresponding position to be acquired; the method comprises the steps of determining a target complement image based on each to-be-used complement image and each to-be-complement image, solving the problems of low image acquisition efficiency and poor quality caused by repeated image acquisition of a chip in the prior art, determining the to-be-used position of a trigger point under a first coordinate system by marking the point position in a to-be-used overview image of a target object, further determining the corresponding to-be-acquired position of each image complement information under the first coordinate system based on a plurality of to-be-used positions, the image attribute of the overview image and at least one predetermined image complement information corresponding to the target object, automatically controlling an electron microscope to acquire the image at the to-be-acquired position, realizing the automation of image acquisition of the complement image, obtaining the to-be-used complement image, and supplementing the to-be-complement image to the target object, and improving the convenience and the effectiveness of image acquisition, so that the finally obtained target object image is high-quality, and further achieving the technical effect of improving the internal circuit analysis accuracy of the target object.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a flowchart of an image processing method according to a first embodiment of the present invention;

FIG. 2 is an exemplary diagram of an operation page provided according to a second embodiment of the present invention;

fig. 3 is a schematic structural view of an image processing apparatus according to a third embodiment of the present invention;

fig. 4 is a schematic structural diagram of an electronic device implementing an image processing method according to an embodiment of the present invention.

Detailed Description

In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example 1

Fig. 1 is a flowchart of an image processing method according to a first embodiment of the present invention, where the method may be performed by an image processing apparatus, which may be implemented in hardware and/or software, and the image processing apparatus may be configured in a computing device. As shown in fig. 1, the method includes:

S110, acquiring and pre-shooting a to-be-used profile including the target object.

It should be noted that, in practical applications, after a high-precision image of an object is acquired, there may be a case where some regions in the high-precision image are unclear, and at this time, an image acquisition may be performed by locating a position where the unclear regions in the high-precision image need to be complemented on an object (such as the same product) that is consistent with the object, so as to complement the acquired image to the unclear regions. The target object may be an object that corresponds to an object in the image that needs to be completed, such as a chip, motherboard, integrated circuit, etc. The profile to be used may be used to reflect the general state of the target object, such as a front view or a full view. The to-be-used profile may be an image including the target object obtained by photographing the target object based on the photographing device, and the photographed to-be-used profile may be stored in a certain buffer location to be called. The profile to be used comprises at least two points to be triggered. The point to be triggered may be a point to be triggered, such as an upper left corner, a lower right corner, an upper right corner, a middle position, and the like.

Specifically, the profile to be used including the target object may be retrieved from a cache location storing the profile to be used through the interface.

In order to improve automation and convenience of the complement, the implementation manner of acquiring and pre-shooting the to-be-used profile including the target object may be as follows: when the trigger loading image control is detected, displaying a path editing box; when receiving the path information input in the path editing box, displaying a profile list corresponding to the path information; the selected history profile is triggered in the profile list as a profile to be used and displayed.

Wherein the load image control may be a button for loading an image, the path edit box may be a rectangular window for keyboard entry and editing of text, through which a user may edit various text, numbers, or passwords. The overview chart list comprises at least one history overview chart corresponding to the target object, and each history overview chart can be regularly distinguished and displayed, so that the page content is clearer.

In this embodiment, when the user clicks the loading image control on the system page, the loading image control is considered to be triggered; or when the user inputs a shortcut key (such as a shortcut key I) corresponding to the "loading image" by using the keyboard, the system is considered to detect that the loading image control is triggered. At this time, the profile list may be displayed directly, the path edit box may be displayed, the user may search the profile in the disk, when the user clicks a certain file in the disk, the path information corresponding to the file may be displayed in the path edit box, the profile list corresponding to the path information may be displayed, or after the path edit box is displayed, the user may input the path information corresponding to the profile list in the path edit box, and when the path confirmation control is triggered, the profile list corresponding to the path information may be displayed. The user can click on the history profile in the profile list to take the selected history profile as the profile to be used and load the profile into the system, and then the profile to be used can be displayed in a system page, so that the point to be triggered is determined on the displayed profile to be used.

S120, determining positions to be used of at least two points to be triggered in a first coordinate system.

The first coordinate system corresponds to the electronic microscope platform, and the target object is placed on the electronic microscope platform. The electron microscope platform may be a scanning electron microscope, such as an electron microscope, for acquiring high-precision images of the target object.

In this embodiment, certain positions (such as the upper left corner, the lower right corner, etc.) may be identified by identifying the outline of the target object in the to-be-used profile, and these positions may be used as the to-be-triggered points, further, it may be determined which position of the to-be-used profile on the target object placed in the electronic microscope platform the to-be-triggered point is located, and the position may be used as the to-be-used position of the to-be-triggered point in the first coordinate system, so that the size of the target object may be determined based on the to-be-used position subsequently, and further, the complement position may be determined.

In order to improve the accuracy of determining the size of the target object and ensure the accuracy of determining the position of the complement map later, the point location can be selected by a user based on the displayed profile map to be used. Specifically, before determining the to-be-used position of the at least two to-be-triggered points under the first coordinate system, the method further comprises: when the trigger position mark control is detected, determining at least two points to be triggered in the profile to be used based on the trigger operation of the profile to be used by a user, and marking and displaying the points to be triggered.

Wherein the position-marking control may be a button for actuating a function for marking a position.

In this embodiment, the user may click the position mark control of the system page, or may input a shortcut key by using a keyboard, and trigger the position mark control, so that the user is allowed to mark a position point on the profile to be used, and the user may select the point to be triggered by clicking the position on the profile to be used, and correspondingly select at least two points to be triggered from the profile to be used. The selected point to be triggered can also be displayed in a distinguishing manner in the profile to be used. For example, the spot may be highlighted, color displayed altered, or icon displayed, etc.

In this embodiment, in order to better select the point to be triggered from the profile to be used, the image can be scaled by using a mouse wheel according to the actual situation, several scaling factors of 2 times, 4 times, 0.5 times, 0.25 times and 0.125 times are supported, a "moving image" button can be clicked, the profile to be used is clicked or dragged by using a mouse, the position of the movable image can be moved, a "jump to origin" button can be clicked, and the moved image can be automatically moved back to the initial loading state. The point to be triggered may be selected in the displayed profile to be used by scaling the profile to be used to a certain size or moving to a certain position.

It should be noted that, after the profile to be used is imported into the system, the positions of the parts of the target object in the profile to be used are under the system coordinate system, and the position of the point to be triggered under the first coordinate system may be obtained by performing position conversion on the position information of the point to be triggered under the system coordinate system. Optionally, an implementation manner of determining the positions to be used of the at least two points to be triggered in the first coordinate system may be: and processing the position information of the current point to be triggered in the to-be-used profile based on a predetermined coordinate conversion matrix for at least two point to be triggered to obtain the to-be-used position of the point to be triggered in the first coordinate system.

The coordinate transformation matrix may be a rotational translation matrix between the system coordinate system and the first coordinate system. The method for determining the position to be used of each point to be triggered in the first coordinate system is the same, and any point to be triggered can be used as the current point to be triggered for description.

In this embodiment, the position to be used of the point to be triggered under the first coordinate system may be obtained by performing translational rotation processing on the position information of the current point to be triggered in the to-be-used profile by using the coordinate transformation matrix. For example, the points to be triggered are the positions of the upper left corner and the lower right corner of the profile to be used, and the coordinates of the two points to be triggered on the electronic microscope platform are respectively A (xa, ya) and B (xb, yb) after being processed by the coordinate transformation matrix.

S130, determining a to-be-acquired position of at least one image complement information under a first coordinate system based on each to-be-used position, the image attribute of the to-be-used profile and the at least one image complement information corresponding to the target object.

Wherein the image attribute may be the resolution of the image. The image complement information may be position information to be complemented in the target object, such as a center point of the target object, an upper left position, a lower right position, and the like.

In this embodiment, the resolution of the profile to be used may be obtained as the image attribute by parsing the profile to be used. At least one image complement information corresponding to the target object can be called, for example, coordinates of a position needing complement, which are edited in advance, can be loaded by clicking a complement position loading button of the system page to serve as the image complement information, and the loaded image complement information can be displayed on the system page. If the number of the image complement information is a plurality of, selecting certain image complement information, and determining the position to be acquired of the image complement information under a first coordinate system by combining each position to be used and the image attribute so as to acquire the complement image based on the position to be acquired; after the plurality of image complement information is loaded, the position to be acquired of each image complement information under the first coordinate system is sequentially determined based on the loading sequence, so that the electronic microscope platform can sequentially acquire high-precision images of the target object corresponding to each position to be acquired, and the convenience of complement image acquisition is improved.

In this embodiment, based on each position to be used, the image attribute of the profile to be used, and at least one piece of image complement information predetermined to correspond to the target object, the implementation manner of determining the position to be acquired of the at least one piece of image complement information in the first coordinate system may be: determining the pixel size of the profile to be used based on each position to be used and the image attribute; and determining the position to be acquired of the at least one image complement information under the first coordinate system based on each position to be used, the pixel size and the at least one image complement information.

The pixel size refers to a pixel size, and includes a lateral dimension (width) and a longitudinal dimension (height).

In practical applications, the pixel size of the profile to be used may be calculated by taking each position to be used and the image attribute as parameters of a function for calculating the pixel size. Alternatively, the image attribute may be resolution, where resolution refers to the number of pixels displayed, including the number of horizontal pixels and the number of vertical pixels, and for example, resolution 160×128 refers to the number of horizontal pixels being 160 and the number of vertical pixels being 128. Based on each position to be used and the image attribute, the implementation manner of determining the pixel size of the target object may be: determining size information of a target object based on each position to be used; based on the resolution and size information, a pixel size is determined. Specifically, the positions to be used may be positions of an upper left corner and a lower right corner of the image, the abscissa of the two positions to be used may be subtracted to obtain a lateral distance, the ordinate of the two positions to be used may be subtracted to obtain a longitudinal distance, the lateral distance may be used as a width of the target object, the longitudinal distance may be used as a height of the target object, and the size of the target object may be obtained based on the height and the width of the target object. Further, the vertical pixel number in the high and resolution of the target object can be used as the quotient as the pixel size of the longitudinal direction of the profile to be used, and The width of the target object is multiplied by the number of horizontal pixels in the resolution to obtain a quotient as the horizontal pixel size of the profile to be used, and the pixel size of the profile to be used can be determined based on the horizontal pixel size and the vertical pixel size of the profile to be used. For example, assuming that the target object is a chip a, coordinates of each position to be used are a (xa, ya) and B (xb, yb), respectively, the size of the chip a may be calculated as: transverse dimension

Longitudinal dimension->

The resolution ix x iy of the profile to be used for the chip a (or the chip B of the same product as the chip a) can be calculated according to the lateral dimension H, the longitudinal dimension V, and the resolution ix x iy, and the lateral and longitudinal pixel dimensions of the profile to be used are respectively: lateral pixel size +.>

Vertical pixel size +.>

。

Further, after determining the pixel size of the profile to be used, the position to be acquired of each image complement information under the first coordinate system may be calculated by the pixel size, the position to be used, and at least one image complement information. Specifically, based on each position to be used, the pixel size, and at least one image complement information, an implementation manner of determining the position to be acquired of the at least one image complement information in the first coordinate system may be: for at least one image complement information, determining an intermediate value corresponding to the current image complement information based on the pixel size and the current image complement information; and determining a to-be-processed position from the to-be-used positions, and determining a to-be-acquired position of the current image complement information under a first coordinate system based on the to-be-processed position and the intermediate value.

The image complement information includes a horizontal complement coordinate and a vertical complement coordinate. The method for determining the position to be acquired of each piece of image complement information under the first coordinate system is the same, and any piece of image complement information can be used as current image complement information for description.

In this embodiment, the product of the lateral complement coordinates and the lateral pixel size of the pixel size may be processed, and the obtained product value may be used as an intermediate value, and the product of the longitudinal complement coordinates and the longitudinal pixel size of the pixel size may be processed, and the obtained product value may be used as an intermediate value. The position information of the upper left corner position in each position to be used may be taken as the position to be processed. Further, the intermediate value determined based on the transverse complement coordinate and the transverse pixel size may be added to the transverse complement coordinate in the position to be processed, the obtained sum is used as the transverse coordinate of the current image complement information under the first coordinate system, the intermediate value determined based on the longitudinal complement coordinate and the longitudinal pixel size is added to the longitudinal complement coordinate in the position to be processed, the obtained sum is used as the longitudinal coordinate of the current image complement information under the first coordinate system, the position to be acquired may be determined based on the transverse coordinate and the longitudinal coordinate of the current image complement information under the first coordinate system, and at this time, the position to be acquired is relative to the position information under the electronic microscope platform, so that the electronic microscope controlling the electronic microscope platform moves to the position to be acquired to perform image acquisition.

For example, assuming that the pixel size is (px, py), the position coordinate (i.e., image complement information) of the target object requiring the complement is (cx, cy), the position to be used is a (xa, ya), and the chip position coordinate (i.e., the position to be collected) on the electronic microscope platform can be calculated according to the position coordinate of the complement, the position to be used and the pixel size, and the chip position coordinate is (xa+cx, ya+cy).

And S140, controlling the target electron microscope to move to each position to be acquired so as to obtain the corresponding complementary image to be used at the corresponding position to be acquired.

Wherein the target electron microscope may be an electron microscope.

Specifically, after determining each position to be acquired, the target electron microscope can be controlled to move to each position to be acquired in sequence, shooting is performed when the target electron microscope moves to the current position to be acquired, and a to-be-used supplementary map image corresponding to the corresponding position to be acquired is obtained, and at the moment, part information in the to-be-used supplementary map image is matched with image supplementary information. The electronic microscope platform can be operated to be positioned to a pre-image supplementing position according to the position to be acquired, and the electronic microscope platform is controlled to be at a specified position and automatically grabs images, so that full automation of image supplementing is realized.

And S150, determining a target complement image based on each complement image to be used and each complement image to be complemented.

Wherein the image to be complemented corresponds to the target object or a sampled object consistent with the target object. It should be noted that, in the actual image complement scenario, after a high-precision image is acquired for a certain product, the product (i.e. a sampled object) may be applied to a subsequent process, but some positions of the product in the high-precision image need to be subjected to image complement, where a product consistent with the sampled object may be used as a target object to perform part sampling on the product, so as to complement the product to the high-precision image (i.e. the image to be complemented) of the sampled object.

In this embodiment, each image to be complemented may be padded into the image to be complemented, so as to obtain the complemented image as the target complement image. For example, according to the image complement information corresponding to the image to be used, the image to be used can be accurately complemented to the region matched with the image complement information in the image to be complemented.

According to the technical scheme, a to-be-used profile including a target object is obtained; determining positions to be used of at least two points to be triggered in the profile to be used under a first coordinate system; determining a position to be acquired of at least one image complement information under a first coordinate system based on each position to be used, the image attribute of the profile to be used and at least one image complement information corresponding to the target object, which is determined in advance; controlling the target electron microscope to move to each position to be acquired so as to obtain a complementary image to be used corresponding to the corresponding position to be acquired; the method comprises the steps of determining a target complement image based on each to-be-used complement image and each to-be-complement image, solving the problems of low image acquisition efficiency and poor quality caused by repeated image acquisition of a chip in the prior art, determining the to-be-used position of a trigger point under a first coordinate system by marking the point position in a to-be-used overview image of a target object, further determining the corresponding to-be-acquired position of each image complement information under the first coordinate system based on a plurality of to-be-used positions, the image attribute of the overview image and at least one predetermined image complement information corresponding to the target object, automatically controlling an electron microscope to acquire the image at the to-be-acquired position, realizing the automation of image acquisition of the complement image, obtaining the to-be-used complement image, and supplementing the to-be-complement image to the target object, and improving the convenience and the effectiveness of image acquisition, so that the finally obtained target object image is high-quality, and further achieving the technical effect of improving the internal circuit analysis accuracy of the target object.

Example two

As an alternative embodiment of the foregoing embodiment, a specific application scenario example is given to make the technical solution of the embodiment of the present invention further clear to those skilled in the art. In particular, reference may be made to the following details.

For example, a profile of the chip (i.e. the target object) may be photographed (i.e. the profile is to be used), and the resolution ix x iy of the profile is obtained. And then placing the chip needing to be complemented into a scanning electron microscope, finding out positions (namely positions to be used) on the chip in an electron microscope platform corresponding to points to be triggered at the left upper corner and the right lower corner of the profile diagram, and respectively recording the positions to be used of the chip on the electron microscope platform, namely A (xa, ya) and B (xb, yb). From the A, B coordinate values, the chip size can be calculated as: transverse dimension

Longitudinal dimension->

. The pixel sizes in the horizontal and vertical directions of the profile to be used can be calculated from the horizontal dimension H, the vertical dimension V, and the resolution ix x iy, respectively: lateral pixel size +.>

Vertical pixel size +.>

. The corresponding on-platform chip position coordinates may be calculated according to the complement image position coordinates (i.e., image complement information) recorded by the operator, such as (cx, cy), the position a (xa, ya) and the pixel size, where the chip position coordinates are (xa+cx×px, ya+cy×py). The platform can be operated to be positioned to the position of the pre-compensation graph according to the coordinates. Correspondingly, the electronic microscope platform can be controlled to a designated position according to a plurality of pre-recorded map-supplementing coordinates and automatically grasp images, so that full automation of map supplementing is realized.

In order to further improve automation and convenience of the complement, multiple functions under the menu 'high-level' control can be displayed by clicking the menu 'high-level' control on the page in the system page, such as complement functions and regional photographing. The operation page of the map complement function can be opened by clicking the control corresponding to the map complement function or inputting a shortcut key corresponding to the map complement function, and the operation page is shown in fig. 2. The overview chart can be selected by triggering an image loading control in an operation page, the selected overview chart is displayed in an overview chart display column to be used, a zooming function can be started by clicking a zooming image control according to actual conditions, and the overview chart in the display column is zoomed by using a mouse wheel, so that multiple zooming magnifications of 2 times, 4 times, 0.5 times, 0.25 times, 0.125 times and the like are supported. The "moving image" control can also be clicked to start the moving function, and the position of the profile in the display column can be dragged by clicking with a mouse. The "jump to origin" control may also be clicked to automatically re-move the moved profile back to the initial position in the upper left corner of the window. After the chip to be subjected to the supplementary shooting is placed in the electron microscope to adjust the position, a position marking control (comprising a marking starting point and a marking ending point) can be clicked, the point in the profile diagram is clicked, the point is marked, and the position of the marked point under the imaging of the electron microscope is found. Different edit points may be displayed in the profile with different colored marks, respectively. Further, the coordinate of the position to be complemented (namely, the image complement information) marked in advance can be loaded by clicking the loading position control, a plurality of image complement information is displayed in an image complement information display column, the coordinate position can be displayed on the overview chart and marked for display (for example, a red circle cross is used as a prompt), the picture can be automatically centrally displayed at the position, and meanwhile, the electron microscope platform is moved to the position to automatically capture the image.

Example III

Fig. 3 is a schematic structural diagram of an image processing apparatus according to a third embodiment of the present invention. As shown in fig. 3, the apparatus includes: the profile to be used acquisition module 310, the position to be used determination module 320, the position to be acquired determination module 330, the complement to be used image determination module 340, and the target complement image determination module 350.

Wherein, the to-be-used profile acquisition module 310 is configured to acquire a to-be-used profile including a target object captured in advance; the to-be-used profile comprises at least two to-be-triggered points; a to-be-used position determining module 320, configured to determine to-be-used positions of the at least two to-be-triggered points in a first coordinate system; the first coordinate system corresponds to the electronic microscope platform, and the target object is placed on the electronic microscope platform; a to-be-acquired position determining module 330, configured to determine, based on each to-be-used position, an image attribute of the to-be-used profile, and at least one predetermined image complement information corresponding to the target object, a to-be-acquired position of the at least one image complement information in the first coordinate system; the to-be-used complement image determining module 340 is configured to control the target electron microscope to move to each to-be-acquired position, so as to obtain a to-be-used complement image corresponding to the to-be-used complement image at the corresponding to-be-acquired position; a target complement image determining module 350, configured to determine a target complement image based on each of the to-be-used complement image and the to-be-complement image; wherein the image to be complemented corresponds to the target object or a sampled object consistent with the target object.

On the basis of the above device, optionally, the to-be-used profile acquisition module 310 includes a path editing box display unit, a profile list display unit, and a to-be-used profile determination unit.

The path editing frame display unit is used for displaying a path editing frame when the trigger loading image control is detected;

a profile list display unit configured to display a profile list corresponding to route information input in the route editing box when the route information is received; wherein the profile list comprises at least one history profile corresponding to the target object;

and the to-be-used profile determining unit is used for taking the history profile triggered to be selected in the profile list as the to-be-used profile and displaying the history profile.

On the basis of the device, the device optionally further comprises a to-be-triggered point position determining module.

And the to-be-triggered point position determining module is used for determining at least two to-be-triggered point positions in the to-be-used profile based on the triggering operation of the user on the to-be-used profile when the triggering position marking control is detected, and marking and displaying the to-be-triggered point positions.

On the basis of the above device, optionally, the to-be-used position determining module 320 is specifically configured to process, for the at least two to-be-triggered points, position information of a current to-be-triggered point in the to-be-used profile based on a predetermined coordinate transformation matrix, so as to obtain a to-be-used position of the to-be-triggered point in a first coordinate system.

On the basis of the above device, optionally, the to-be-acquired position determining module 330 includes: and the pixel size determining unit and the position to be acquired determining unit.

A pixel size determining unit configured to determine a pixel size of the profile to be used based on each of the positions to be used and the image attributes;

and the to-be-acquired position determining unit is used for determining the to-be-acquired position of the at least one image complement information under the first coordinate system based on each to-be-used position, the pixel size and the at least one image complement information.

On the basis of the above apparatus, optionally, the image attribute includes resolution, and the pixel size determining unit includes: the size information determining subunit and the pixel size determining subunit.

A size information determining subunit, configured to determine size information of the target object based on each of the positions to be used;

And a pixel size determining subunit configured to determine the pixel size based on the resolution and the size information.

On the basis of the above apparatus, optionally, the pixel size determining unit includes: an intermediate value determining subunit and a position to be acquired determining subunit.

An intermediate value determining subunit configured to determine, for the at least one image complement information, an intermediate value corresponding to the current image complement information based on the pixel size and the current image complement information;

and the to-be-acquired position determining subunit is used for determining a to-be-processed position from the to-be-used positions and determining the to-be-acquired position of the current image complement information under the first coordinate system based on the to-be-processed position and the intermediate value.

The image processing device provided by the embodiment of the invention can execute the image processing method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

Example IV

Fig. 4 is a schematic structural diagram of an electronic device implementing an image processing method according to an embodiment of the present invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 4, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.

Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the respective methods and processes described above, for example, an image processing method.

In some embodiments, the image processing method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the image processing method in any other suitable way (e.g. by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.

The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.

The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.

Claims

1. An image processing method, comprising:

acquiring a pre-shot profile to be used, which comprises a target object; the to-be-used profile comprises at least two to-be-triggered points;

determining positions to be used of the at least two points to be triggered on the target object under a first coordinate system; the first coordinate system corresponds to the electronic microscope platform, and the target object is placed on the electronic microscope platform;

determining a target complement image based on each of the complement images to be used and the complement images to be complemented; wherein the image to be complemented corresponds to the target object or a sampled object consistent with the target object;

wherein the determining, based on each of the to-be-used positions, the image attribute of the to-be-used profile, and at least one piece of predetermined image complement information corresponding to the target object, the to-be-acquired position of the at least one piece of image complement information in the first coordinate system includes:

determining a pixel size of the to-be-used profile based on each of the to-be-used locations and the image attributes;

and determining a position to be acquired of the at least one image complement information under the first coordinate system based on each position to be used, the pixel size and the at least one image complement information.

2. The method according to claim 1, wherein the acquiring a pre-photographed profile to be used including a target object includes:

when the trigger loading image control is detected, displaying a path editing box;

when receiving the path information input in the path editing box, displaying a profile list corresponding to the path information; wherein the profile list comprises at least one history profile corresponding to the target object;

and taking the history profile triggered to be selected in the profile list as the profile to be used and displaying the history profile.

3. The method of claim 1, further comprising, prior to said determining the location to be used on the target object in the first coordinate system, prior to said determining the at least two points to be triggered:

when the trigger position mark control is detected, determining at least two to-be-triggered points in the to-be-used profile based on the trigger operation of the user on the to-be-used profile, and marking and displaying the to-be-triggered points.

4. The method of claim 1, wherein the determining the location to be used on the target object for the at least two points to be triggered in a first coordinate system comprises:

And processing the position information of the current point to be triggered in the to-be-used profile based on a predetermined coordinate transformation matrix for the at least two point to be triggered to obtain the to-be-used position of the point to be triggered on the target object under a first coordinate system.

5. The method of claim 1, wherein the image attributes comprise a resolution, and wherein determining a pixel size of the overview to be used based on each of the locations to be used and the image attributes comprises:

determining size information of the target object based on each to-be-used position;

the pixel size is determined based on the resolution and the size information.

6. The method of claim 1, wherein the determining the location of the at least one image complement information to be acquired in the first coordinate system based on each of the location to be used, the pixel size, and the at least one image complement information comprises:

for the at least one image complement information, determining an intermediate value corresponding to the current image complement information based on the pixel size and the current image complement information; wherein the intermediate value is determined based on a product process of the pixel size and the current image complement information;

And determining a to-be-processed position from the to-be-used positions, and determining a to-be-acquired position of the current image complement information under the first coordinate system based on the to-be-processed position and the intermediate value.

7. An image processing apparatus, comprising:

the to-be-used profile acquisition module is used for acquiring a pre-shot to-be-used profile comprising a target object; the to-be-used profile comprises at least two to-be-triggered points;

the to-be-used position determining module is used for determining to-be-used positions of the at least two to-be-triggered points on the target object under a first coordinate system; the first coordinate system corresponds to the electronic microscope platform, and the target object is placed on the electronic microscope platform;

The target complement image determining module is used for determining a target complement image based on each to-be-used complement image and each to-be-complement image; wherein the image to be complemented corresponds to the target object or a sampled object consistent with the target object;

the to-be-acquired position determining module comprises a pixel size determining unit and a to-be-acquired position determining unit;

the pixel size determining unit is used for determining the pixel size of the to-be-used profile image based on each to-be-used position and the image attribute;

the to-be-acquired position determining unit is configured to determine, based on each to-be-used position, the pixel size, and the at least one image complement information, a to-be-acquired position of the at least one image complement information in the first coordinate system.

8. An electronic device, the electronic device comprising:

at least one processor; and

the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the image processing method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions for causing a processor to implement the image processing method of any one of claims 1-6 when executed.