Feature #6506

Include robots.txt file with specific exclusions in release tarballs

Added by Dan Gillean over 7 years ago. Updated over 6 years ago.

Status:NewStart date:03/25/2014
Priority:MediumDue date:
Assignee:-% Done:

0%

Category:-
Target version:-
Sponsored:No Tested version:

Description

Previous experience has shown that when webcrawlers hit AtoM resources, they can timeout pages by requests to resources such as the EAD links. Similarly, there is some data that users would not want exposed, such as user account data, etc.

Since we have had to deal with this on a case-by-case basis for many hosted installations, it might be a good idea to include a default robots.txt file that excludes specific resources known to cause performance or security issues when a bot crawls an AtoM page.

If users wish to further tailor the document, there will already be a file in the right place for them to work with.

History

#1 Updated by Jesús García Crespo over 7 years ago

  • Target version changed from Release 2.0.2 to Release 2.1.0

#2 Updated by Jesús García Crespo over 7 years ago

  • Target version changed from Release 2.1.0 to Release 2.2.0

#3 Updated by Sarah Romkey over 6 years ago

  • Target version deleted (Release 2.2.0)

#4 Updated by Dan Gillean over 6 years ago

  • Project changed from Access to Memory (AtoM) to AtoM Wishlist
  • Category deleted (Installation)

Moved to AtoM wishlist until sponsored for inclusion.

#5 Updated by Jesús García Crespo over 6 years ago

  • Assignee deleted (Jesús García Crespo)

Also available in: Atom PDF