WO2017063596A1

WO2017063596A1 - Method, apparatus and device for processing sitemap

Info

Publication number: WO2017063596A1
Application number: PCT/CN2016/102215
Authority: WO
Inventors: 梁捷; 梁卡喆
Original assignee: 广州神马移动信息科技有限公司
Priority date: 2015-10-16
Filing date: 2016-10-14
Publication date: 2017-04-20
Also published as: CN105260469A; CN105260469B

Abstract

A method for processing a sitemap, an apparatus and a device. The method comprises: acquiring the sitemap of a website according to pre-set information (101); acquiring the links of pages in the sitemap and accessing same (102); according to the access result, deleting links recorded in the sitemap which affect search thereof (103); and generating a new sitemap (104). According to the method, the quality of sitemaps can be improved, and the possibility of recording by a search engine can also be increased, thus satisfying the respective requirements of the website and of the search engine.

Description

Method, device and device for processing website map

The present application claims the priority of the Chinese Patent Application filed on October 16, 2015, the Chinese Patent Application No. CN201510676894.0, entitled "A Method, Apparatus and Apparatus for Processing Website Maps", the entire contents of which are hereby incorporated by reference. The present application is incorporated by reference.

Technical field

The present invention relates to the field of mobile internet technologies, and in particular, to a method, device and device for processing a website map.

Background technique

Currently, search engines often find pages through links within websites (also known as sites) and other sites. Sitemaps can be used to make it easier for sites to tell search engines what pages are available on the site. The simplest form of sitemap is the XML (Extensible Markup Language) file, which lists the URLs in the site and other metadata about each URL (the time of the last update, the frequency of the changes, and the relative The importance of other URLs on the site, etc., so that search engines can crawl the content of the site more intelligently. Simply put, a sitemap can be understood as a list of links on a website. Generating a sitemap and submitting it to a search engine can make the content of the site easy to include, including those that are deeper, which is a good way for a site to talk to a search engine.

However, the quality of the website links contained in the sitemaps provided by many websites is low, and there may be many problems, such as link errors, inconsistent or untimely updates of the links, etc., which will waste resources that the search engine crawls. This leads to the fact that although the site provides a sitemap, the search engine is based on The result of the crawl does not necessarily include the sitemap's website link, and may also trigger the search engine's demotion rules, reduce the number of links to the site, and reduce the search ranking of the site.

Summary of the invention

In order to solve the above technical problem, the present invention provides a method, device and device for processing a website map to improve the quality of website links in the website map.

According to an aspect of the present application, a method for processing a website map includes: obtaining a website map of the website according to the preset information; obtaining a link of the page in the website map and performing the access; and deleting the influence of the search in the website map according to the result of the access Link; generate a new site map.

Preferably, after the obtaining the link of the page in the website map and performing the access, the method further includes: extracting a keyword and a text feature value from the accessed page; and extracting the keyword and the text feature value from the pre-stored keyword and the text feature value according to the extracted keyword Compare the results and delete the links in the sitemap that affect the search.

Preferably, the deleting the link affecting the search inclusion in the website map according to the access result comprises: deleting the corresponding link when the access result is an HTTP 404 error that cannot be accessed; or, when the access result is a page response time greater than or equal to the set When the threshold is set, the corresponding link is deleted; or, when the result of the visit is that the title, keyword and description of the page are incomplete, the corresponding link is deleted; or, the result of the visit is the body content of the page and the title, keyword and When the description does not match, delete the corresponding link.

Preferably, the deleting the link affecting the search in the website map according to the comparison result of the extracted keyword and the text feature value and the pre-stored keyword and the text feature value comprises: according to the extracted keyword and the text feature value and the pre-stored Keywords and body The comparison result of the feature values is consistent, and it is determined that the content is repeatedly submitted, and the corresponding link is deleted.

Preferably, the method further comprises: providing the generated new website map to the search engine for access.

Preferably, the method further includes: recording the collected data that is searched and included by the search engine after accessing the new website map.

According to another aspect of the present application, an apparatus for processing a website map includes: an obtaining module, configured to acquire a website map of the website according to the preset information; and an access module, configured to obtain a website map obtained according to the obtaining module, Obtaining a link of the page in the website map and accessing; the first processing module is configured to delete a link in the website map that affects the search and the inclusion according to the access result of the access module; and generate a module, configured to process in the first processing module After generating a new site map.

Preferably, the device further includes: a second processing module, configured to extract a keyword and a text feature value from the accessed page, and compare the extracted keyword and the text feature value with the pre-stored keyword and the text feature value, Deleting a link in the website map that affects the search and inclusion; the generating module is configured to generate a new website map after the first processing module and the second processing module perform processing.

Preferably, the device further comprises: an output module, configured to provide the new website map generated by the generating module to the search engine for access.

Preferably, the device further includes: a monitoring module, configured to record the collected data that is searched and included by the search engine after accessing the new website map.

Preferably, the first processing module includes: a first deleting unit, configured to delete a corresponding link when an access result is an HTTP 404 error that cannot be accessed; or a second deleting unit, where the access result is a page When the response time is greater than or equal to the set threshold, the corresponding link is deleted; or, the third deletion unit is used for accessing The result is that when the title, keyword and description of the page are incomplete, the corresponding link is deleted; or, the fourth deleting unit is used to delete when the result of the access is that the content of the page does not match the title, keyword and description of the page. Corresponding link.

According to another aspect of the present application, there is provided a processing apparatus comprising: a memory for storing a program, a processor, for executing a method comprising the following steps by calling a program stored in the memory: according to preset information Obtain a website map of the website; obtain a link to the page in the website map and access; delete the link affecting the search and inclusion in the website map according to the result of the visit; generate a new website map.

In accordance with another aspect of the present application, a computer readable medium having non-volatile program code executable by a processor, the program code causing the processor to perform the method described above.

It can be found that the technical solution of the embodiment of the present invention accesses the link of the page in the website map, and after the link result is found to have a link affecting the search and inclusion, the link affecting the search and inclusion in the website map is deleted, and then generated. New website map, so that you can optimize the original website map of the website, try to avoid the various links in the website map with bad or error-prone links, which can improve the quality of the website map, and can also be added by the search engine. The possibility to meet the needs of websites and search engines.

DRAWINGS

The above and other objects, features, and advantages of the present invention will become more apparent from the aspects of the embodiments of the present invention. The reference numbers generally represent the same components.

1 is a schematic flow chart of a method of processing a website map according to an embodiment of the present application;

2 is a schematic flow chart of a method of processing a website map according to another embodiment of the present application;

3 is a schematic flow chart of a method of processing a website map according to still another embodiment of the present application;

4 is a schematic block diagram of an apparatus for processing a website map according to an embodiment of the present application;

FIG. 5 is a schematic block diagram of an apparatus for processing a website map according to another embodiment of the present application; FIG.

Figure 6 is a schematic block diagram of a processing device in accordance with still another embodiment of the present application.

detailed description

Preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While the invention has been described in terms of the preferred embodiments of the present invention, it should be understood that Rather, these embodiments are provided so that this disclosure will be thorough and complete.

Also, in the description of the present application, the terms "first", "second", and the like are used merely to distinguish a description, and are not to be construed as indicating or implying a relative importance.

The application of the present invention provides a method for processing a website map, which can improve the quality of website links in the website map and meet the respective needs of the website and the search engine.

1 is a schematic flow chart of a method of processing a website map in accordance with one embodiment of the present application.

As shown in Figure 1, it includes:

Step S101: Obtain a website map of the website according to the preset information.

In this step, the website map of the website may be obtained according to the setting information provided by the website after being negotiated with the website.

Step S102: Obtain a link of a page in the website map and perform access.

In this step, a link is obtained for each page in the website map, and the links of the respective pages are links of respective URLs (Uniform Resource Locators), and the URL links are separately accessed for verification.

Step S103: Delete a link in the website map that affects the search and inclusion according to the access result.

If the link in the site map has an inaccessible HTTP 404 error, too long access time, or other issues, it will affect the search engine's inclusion of the site link.

In this step, the links affecting the search inclusion in the website map are deleted according to the result of the visit, including:

Delete the corresponding link when the result of the access is an unreachable HTTP 404 error; or,

When the result of the visit is that the page response time is greater than or equal to the set threshold, the corresponding link is deleted; or,

When the result of the visit is that the title, keyword, and description of the page are incomplete, delete the corresponding link; or,

When the result of the visit is that the body content of the page does not match the title, keyword, and description of the page, the corresponding link is deleted.

Of course, the case and manner of deleting the link affecting the search and inclusion in the website map according to the result of the access are not limited in this embodiment, and may include other.

Step S104: Generate a new website map.

In this step, after deleting the links affecting the search and inclusion in the website map, rearrange and generate a new website map.

Further, in this embodiment, after step S104, the method may further include: providing the generated new website map to the search engine for access.

Moreover, after the generated new website map is provided to the search engine for accessing, the recorded data may be recorded and searched after the search engine accesses the new website map.

It can be found that the technical solution of the embodiment of the present invention firstly accesses the link of the page in the website map, and after the link result is found to affect the search and the link included in the search, the link affecting the search and inclusion in the website map is deleted, and a new link is generated. Site map, so that you can optimize the original site map of the site, try to avoid the various links in the site map are not good or error-prone links, which can improve the quality of the site map, and can also be added by the search engine. Possibility to meet the needs of websites and search engines.

The technical solution of the present application is further described in more detail below.

2 is a schematic flow chart of a method of processing a website map in accordance with another embodiment of the present application.

As shown in Figure 2, it includes:

Step S201: Obtain a website map of the website according to the preset information.

This step is described in the above description of step S101.

Step S202: Obtain a link of a page in the website map and perform an access.

This step is described in the above description of step S102.

Step S203: Delete a link in the website map that affects the search and inclusion according to the access result.

This step is described in the above description of step S103.

Step S204, extracting keywords and text feature values from the accessed pages.

In this step, the content of the page may be extracted by using different existing algorithms, and the feature value of the text is extracted from the content of the text. The application of the present invention is not limited.

Step S205: Delete a link affecting the search and inclusion in the website map according to the comparison result between the extracted keyword and the text feature value and the pre-stored keyword and the text feature value.

In this step, the extracted keyword and the text feature value are compared with the pre-stored keyword and the text feature value, respectively, to determine whether the comparison result is consistent.

Specifically, the extracted keywords are compared with the pre-stored keywords, and the extracted text feature values are compared with the pre-stored text feature values. When the comparison results are all consistent, the extracted keywords and the body feature values may be considered. It is consistent with the comparison result of the pre-stored keyword and the text feature value, and it is judged that the content is repeatedly submitted, and the corresponding link is deleted.

Step S206, generating a new website map.

Further, in this embodiment, after the new website map is generated in step 206, step S207 may be further included, and the generated new website map is provided to the search engine for access.

In this step, the generated new website map may be replaced with the original website map of the website for the search engine to visit the website to access the new website map, or may be set by the website, and the search engine directly accesses the service platform that generates the new website map. The new website map, the application of the present invention is not limited, as long as the search engine can access the new website map.

It should be noted that the processing of the above steps S202 and S203 has no necessary sequential relationship with the processing of steps S204 and S205, and the above-mentioned step arrangement is only for convenience of description.

It should be noted that, after the step S207, the method further includes: recording the collected data that is searched and included by the search engine after accessing the new website map.

It can be found that, in the technical solution of the embodiment of the present application, the link affecting the search and inclusion in the website map can be deleted according to the result of the access, and the comparison result between the extracted keyword and the text feature value and the pre-stored keyword and the text feature value can be deleted. The links in the site map that affect search inclusion provide optimization. In addition, the search engine may also record the collected data after the search engine accesses the new website map, thereby providing reference for the subsequent website map modification or for analyzing the website.

FIG. 3 is a schematic flowchart of a method for processing a website map according to still another embodiment of the present application. In this embodiment, a website map is processed by a sitemap service platform.

Specifically, as shown in FIG. 3, including:

Step S301: The sitemap service platform performs data extraction on the sitemap of the website according to the setting information of the website.

In this step, the website and the sitemap service platform (hereinafter referred to as the service platform) are pre-negotiated, and the mapping relationship between the sitemap and the service platform is set by the website, and the service platform is allowed to process the sitemap according to the setting information provided by the website, for example, the address information. Website setting mapping relationships can be implemented in XML. The service platform can extract data from the sitemap according to the setting information provided by the website, and obtain the URL information of each link therein.

Step S302: The service platform checks the URLs in the extracted sitemap to determine whether the access URL has an HTTP404 error that cannot be accessed. If yes, step S311 is performed to delete the URL from the sitemap and record the reason. If not, step S303 is performed. .

The HTTP404 error means that the webpage pointed to by the link does not exist, that is, the URL of the original webpage is invalid. This often happens, for example, the URL URL generation rule is changed, the webpage file is renamed or moved, and the import link is misspelled. The original URL address cannot be accessed; when the web server receives a similar request, it will return a 404 status code, telling the browser that the requested resource does not exist. Therefore, when an HTTP404 error that the URL cannot access is displayed, it indicates that the URL has expired. At this time, the URL is deleted from the sitemap and the reason is recorded.

Step S303: The service platform determines whether the page response speed of the access URL is abnormal. If yes, step S311 is performed to delete the URL from the sitemap and record the reason. If not, step S304 is performed.

When the URL can be accessed normally, the response speed of the page is detected, and the response speed can be measured by the response time. If the response time is greater than or equal to the set threshold, the response speed is considered abnormal, and if it is less than the set threshold, the response speed is considered normal. The threshold value can be set according to experience, for example, set to 500 milliseconds or 1 second, which is not limited by the application of the present invention.

When it is necessary to explain, the page history access response speed can be compared with the current access response speed to determine whether the response speed is abnormal. If the current response time is much larger than the historical response time and the difference exceeds a certain threshold, the response speed is considered abnormal.

Therefore, when the page response speed is abnormal, it indicates that the page corresponding to the URL may have a problem or the network connection corresponding to the URL may have a problem, which may affect the browsing experience of the user. At this time, the URL is deleted from the sitemap and the reason is recorded.

Step S304: The service platform determines whether the TKD of the page is incomplete. If yes, step S311 is performed to delete the URL from the sitemap and record the reason. If not, step S305 is performed.

TKD is an abbreviation for title title, keyword keywords, description description. The format of TKD can be as follows:

<title>This is the title content</title>

The keyword keyword is a vocabulary that a website administrator sets for a certain page of the website so that the user can search the webpage through the search engine, and the keyword represents the market positioning of the website. Description description, also known as "content label", "description label" or "content summary", reflecting the main content of the web page.

Generally, the complete TKD is in line with the search engine search rules. If the TKD is incomplete and does not conform to the search engine's search rules, the search engine may not search the page or include the link content. Therefore, it is found that the URL is deleted from the sitemap when the TKD is incomplete and the reason is recorded.

Step S305: The service platform determines whether the content of the page body does not match the TKD. If yes, step S311 is performed to delete the URL from the sitemap and record the reason. If not, step S306 is performed.

In this step, according to the body content in the page, it is judged whether the keyword in the TKD appears in the body text, and whether the content of the body text corresponds to the title and description of the TKD. If the keyword in the TKD appears, the content of the body text is the title of the TKD. Corresponding to the description, indicating that it is a match, otherwise it does not match. If it doesn't match, it may be that the body is set incorrectly, or the TKD setting is wrong. These will affect the search engine's search quality and affect the user's browsing experience. Therefore, if the content of the page body is found to be mismatched with TKD, the URL is deleted from the sitemap and the reason is recorded.

Step S306: The service platform performs keyword extraction on the content of the page, and extracts a text feature value for the body content.

In this step, the service platform may perform keyword extraction on the content of the page by using different existing algorithms, and extract the feature value of the body text, which is not limited by the application.

For example, the keyword extraction can use the existing TFIDF (term frequency-inverse document frequency) algorithm, which mainly uses a dictionary to save all word information, and then sorts the dictionary by value value. Finally, the top words with the weights are ranked as keywords. For example, the text feature extraction method based on the context framework or the text feature extraction method based on the ontology may be adopted for extracting the text feature value from the body content.

Step S307: The service platform compares the extracted keyword and the text feature value with the keyword and the text feature value stored by the service platform, and checks whether there is a case of repeatedly submitting the content. If yes, step S311 is performed to delete the URL from the sitemap. And the reason is recorded, and if no, step S308 is performed.

In this step, the extracted keyword and the text feature value are compared with the keyword and the text feature value stored by the service platform, and the body relevance is matched. If the same keyword and body feature value are found on the service platform, the judgment is performed. Repeat for content. By this matching detection, it is possible to check whether there is a case where the content is repeatedly submitted. On the service platform, the detected keyword and body feature values of each page are stored in advance.

In step S308, the service platform saves the extracted keywords, text feature values, and corresponding links for use in subsequent checksums.

Step S309: The service platform generates the processed new sitemap data for the search engine to acquire.

In this step, the website may be set to instruct the search engine to directly obtain the sitemap from the service platform, or the service platform may directly replace the new sitemap with the original sitemap of the website.

Step S310: The service platform monitors the latest sitemap data.

If the URL of the sitemap is included in the search engine, the tag information will be returned, and the service platform monitoring URL is included in the search engine, which can provide reference for subsequent adjustment of the sitemap.

Step S311: The service platform deletes the link from the sitemap, and records the reason for the website to analyze.

In this step, the reason why the link is deleted can be recorded in detail for analysis by the website.

It can be found that the technical solution of the embodiment of the present invention analyzes and filters the sitemap data of the obtained website, and performs access verification on the link provided by the sitemap, and further extracts the keyword content and extracts the feature value of the body content, and Pre-stored keywords and body feature values are matched to avoid submitting duplicate content or poor quality content. Finally, the search engine can monitor the inclusion of the sitemap. Through the above processing, the application of the present invention can optimize the quality of the sitemap, improve the content of the website content collected by the search engine, enable the search engine to better include the website page, and improve the search caused by the duplicate content and the spam submission to the search engine. The issue of power reduction can also better monitor the content of the website.

The method for processing a website map according to the embodiment of the present invention is described in detail. Correspondingly, the embodiment of the present invention further provides an apparatus for processing a website map.

4 is a schematic block diagram of an apparatus for processing a website map according to an embodiment of the present application.

As shown in FIG. 4, an apparatus for processing a website map includes: an obtaining module 401, an access module 402, a first processing module 403, and a generating module 404. The device for processing a website map according to an embodiment of the present application may be a service platform or the like.

The obtaining module 401 is configured to obtain a website map of the website according to the preset information.

After the device negotiates with the website, the obtaining module 401 obtains the website map of the website according to the setting information provided by the website.

The access module 402 is configured to obtain a link of the page in the website map and access according to the website map acquired by the obtaining module 401.

The access module 402 obtains each URL link in the website map and separately accesses the URL link for verification.

The first processing module 403 is configured to delete a link in the website map that affects the search and inclusion according to the access result of the access module 402.

The first processing module 403 deletes the links in the website map that affect the search listing according to various different access results.

The generating module 404 is configured to generate a new website map after the processing by the first processing module 403.

FIG. 5 is another schematic block diagram of an apparatus for processing a website map according to an embodiment of the present application.

As shown in FIG. 5, an apparatus for processing a website map includes: an obtaining module 401, an access module 402, a first processing module 403, and a generating module 404. The functions of each module are described in FIG.

In addition, the device further includes: a second processing module 405.

The second processing module 405 is configured to extract a keyword and a text feature value from the accessed page, and delete a search result included in the website map according to the comparison result between the extracted keyword and the text feature value and the pre-stored keyword and the text feature value. The generating module 404 is configured to generate a new website map after the first processing module 403 and the second processing module 405 perform processing.

The second processing module 405 determines that the content is repeatedly submitted and deletes the corresponding link according to the comparison result between the extracted keyword and the text feature value and the pre-stored keyword and the text feature value.

The device also includes an output module 406.

The output module 406 is configured to provide a new website map generated by the generating module to the search engine for access.

The embodiment of the present invention can replace the generated new website map with the original website map of the website, and the search engine can access the new website map by the website, or can be set by the website, and the search engine directly accesses the service platform to access the new website map. The embodiment of the invention is not limited as long as the search engine can access the new website map.

The device also includes a monitoring module 407.

The monitoring module 407 is configured to record the collected data that is searched and included by the search engine after accessing the new website map.

The first processing module 403 includes: a first deleting unit 4031, a second deleting unit 4032, a third deleting unit 4033, or a fourth deleting unit 4034.

The first deleting unit 4031 is configured to delete the corresponding link when the access result is an HTTP 404 error that cannot be accessed.

The second deleting unit 4032 is configured to delete the corresponding link when the access result is that the page response time is greater than or equal to the set threshold.

The third deleting unit 4033 is configured to delete the corresponding link when the access result is that the title, the keyword, and the description of the page are incomplete.

The fourth deleting unit 4034 is configured to delete the corresponding link when the access result is that the body content of the page does not match the title, keyword, and description of the page.

Further, the embodiment of the present application further provides a processing device.

Figure 6 is a schematic block diagram of a processing device 600 in accordance with one embodiment of the present application.

As shown in FIG. 6, the processing device 600 provided by the embodiment of the present application includes: a memory 601 and a processor 602.

The memory 601 is used to store programs.

The processor 602 is configured to execute a method including the following steps by calling a program stored in the memory 601:

Obtain a website map of the website according to preset information;

Get a link to the page in the site map and access it;

According to the result of the visit, delete the link in the website map that affects the search and inclusion;

Generate a new sitemap.

It should be noted that, for further description of the foregoing method, refer to the detailed description in the foregoing method flow, and details are not described herein again.

In this embodiment, the processor 602 executes various functions and data processing by executing the above-mentioned program stored in the memory 601, that is, the method for processing the website map in the embodiment of the present application. The memory 601 can include, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read only memory (PROM), and an erasable read only. Erasable Programmable Read-Only Memory (EPROM), Electric Erasable Programmable Read-Only Memory (EEPROM), and the like. The processor 602 can execute the foregoing method stored in the memory 601 after receiving the execution instruction, and implement the method defined in the flow disclosed in any embodiment of the present application.

Processor 602 can be an integrated circuit chip with signal processing capabilities. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP processor, etc.), or a digital signal processor (DSP) or an application specific integrated circuit (ASIC). ), off-the-shelf programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general purpose processor may be a microprocessor or the processor or any conventional processor or the like.

It can be understood that the structure shown in FIG. 6 is merely illustrative, and the processing device 600 may further include more or less components than those shown in FIG. 6, or have a different configuration from that shown in FIG. 6. The components shown in Figure 6 can be implemented in hardware, software, or a combination thereof.

The modules and units of the apparatus for processing a website map in the foregoing embodiments may be implemented by software code. In this case, the modules and units described above may be stored in the memory 601 of the processing device 600. The above modules and units can also be implemented by hardware such as an integrated circuit chip.

In summary, the technical solution of the embodiment of the present invention analyzes and filters the sitemap data of the obtained website, performs access verification on the link provided by the sitemap, and performs keyword extraction and text feature value extraction on the body content, and Matches pre-stored keyword and body feature values to avoid submitting duplicate content or poor quality content. Finally, the search engine can monitor the inclusion of the sitemap. Through the above processing, the application of the present invention can optimize the quality of the sitemap, improve the content of the website content collected by the search engine, and enable the search engine to better include the website page, and also solve the search caused by the duplicate content and the spam submission to the search engine. The issue of power reduction can also better monitor the content of the website.

The technical solution according to the application of the present invention has been described in detail above with reference to the accompanying drawings.

Furthermore, the above method according to an embodiment of the present application may also be implemented as a computer program comprising computer program code for causing the processor to execute the method provided by the above embodiment of the present application when executed by the processor instruction. Alternatively, the method according to the present application may also be embodied as a computer program product comprising a computer readable medium having processor-executable non-volatile program code, the program code being processed as described The apparatus is executed to perform the above method of the present application. The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems and methods in accordance with various embodiments of the present application. In this regard, each block of the flowchart or block diagram can represent a module, a program segment, or a portion of code that includes one or more of the Executable instructions. It should also be noted that in some alternative implementations, the functions noted in the blocks may also occur in a different order than the ones in the drawings. For example, two consecutive blocks may be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, can be implemented in a dedicated hardware-based system that performs the specified function or operation. Or it can be implemented by a combination of dedicated hardware and computer instructions.

The embodiments of the present invention have been described above, and the foregoing description is illustrative, not limiting, and not limited to the disclosed embodiments. Without deviation Numerous modifications and changes will be apparent to those of ordinary skill in the art. The choice of terms used herein is intended to best explain the principles, practical applications, or improvements of the techniques in the various embodiments of the embodiments, or to enable those of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

A method for processing a website map, comprising:

Obtain a website map of the website according to preset information;

Get a link to the page in the site map and access it;

According to the result of the visit, delete the link in the website map that affects the search and inclusion;

Generate a new sitemap.
The method according to claim 1, wherein the obtaining a link of the page in the website map and performing the access further comprises:

Extracting keyword and body feature values for the accessed page;

According to the comparison result of the extracted keyword and the text feature value and the pre-stored keyword and the text feature value, the link affecting the search index in the website map is deleted.
The method according to claim 2, wherein the deleting the link affecting the search in the website map according to the comparison result of the extracted keyword and the text feature value and the pre-stored keyword and the text feature value comprises:

According to the comparison result of the extracted keyword and the text feature value and the pre-stored keyword and the text feature value, it is determined that the content is repeatedly submitted, and the corresponding link is deleted.
The method according to claim 1, wherein the deleting the link affecting the search inclusion in the website map according to the result of the access comprises:

Delete the corresponding link when the result of the access is an unreachable HTTP 404 error; or,

When the result of the visit is that the page response time is greater than or equal to the set threshold, the corresponding link is deleted; or,

When the result of the visit is that the title, keyword, and description of the page are incomplete, delete the corresponding link; or,

When the result of the visit is that the body content of the page does not match the title, keyword, and description of the page, the corresponding link is deleted.
The method of claim 1 further comprising: providing the generated new website map to a search engine for access.
The method of claim 5, wherein the method further comprises:

Recording the search data that the search engine searches for and records after accessing the new website map.
An apparatus for processing a website map, comprising:

An obtaining module, configured to obtain a website map of the website according to the preset information;

An access module, configured to obtain a link of a page in the website map according to the website map acquired by the obtaining module, and access the website;

a first processing module, configured to delete, according to the access result of the access module, a link in the website map that affects the search and inclusion;

And a generating module, configured to generate a new website map after the processing by the first processing module.
The device according to claim 7, wherein the device further comprises:

a second processing module, configured to extract a keyword and a text feature value from the accessed page, and delete a link affecting the search index in the website map according to the comparison result between the extracted keyword and the text feature value and the pre-stored keyword and the text feature value ;

The generating module is configured to generate a new website map after the first processing module and the second processing module perform processing.
The device according to claim 7, wherein the device further comprises:

And an output module, configured to provide the new website map generated by the generating module to the search engine for access.
The device according to claim 9, wherein the device further comprises:

The monitoring module is configured to record the collected data that is searched and included by the search engine after accessing the new website map.
The apparatus according to claim 7, wherein the first processing module comprises:

The first deleting unit is configured to delete the corresponding link when the access result is an HTTP404 error that cannot be accessed; or

a second deleting unit, configured to delete the corresponding link when the access result is that the page response time is greater than or equal to the set threshold; or

a third deleting unit, configured to delete the corresponding link when the access result is that the title, keyword, and description of the page are incomplete; or

And a fourth deleting unit, configured to delete the corresponding link when the access result is that the body content of the page does not match the title, keyword, and description of the page.
A processing device, comprising:

Memory for storing programs,

And a processor, configured to execute a method including the following steps by calling a program stored in the memory:

Obtain a website map of the website according to preset information;

Get a link to the page in the site map and access it;

According to the result of the visit, delete the link in the website map that affects the search and inclusion;

Generate a new sitemap.