Erowid Archives & Backups
Archives, Downloads, Snapshots, Mirrors, and Sperowidered Versions
We believe that it is important for people to be able to make good quality local copies of Erowid (or any website). Erowid is dedicated to making the site as useful to as many people as possible and we believe that an important function of library sites is that they be friendly to archivers and offline readers. However, the issue is complicated by practical performance issues, bandwidth constraints, and technical hurdles that make this somewhat more difficult than it may first appear.
A huge load is placed on the Erowid server by spiders and robots that attempt to download the entire site. We have been working to produce an easily downloaded version of the site as well as mirror servers that can be more easily spidered and archived. Many archiver robots do a very poor job with large, complicated websites especially those that use javascript navigation.
In 2004, with the help of JL, we were able to launch our first mirror of the site on a server in Germany (de1.erowid.org). In June 2005, with the help of Stu, we now have a system in place for making regular mirrors of the site and downloadable tar files that can be used to put Erowid on CD or on a local harddrive. This is based on the work we've been doing on Sperowider, an archiver robot designed to work on the Erowid site.
Happily, there are several ways you can now archive Erowid in a community-friendly way:
A huge load is placed on the Erowid server by spiders and robots that attempt to download the entire site. We have been working to produce an easily downloaded version of the site as well as mirror servers that can be more easily spidered and archived. Many archiver robots do a very poor job with large, complicated websites especially those that use javascript navigation.
In 2004, with the help of JL, we were able to launch our first mirror of the site on a server in Germany (de1.erowid.org). In June 2005, with the help of Stu, we now have a system in place for making regular mirrors of the site and downloadable tar files that can be used to put Erowid on CD or on a local harddrive. This is based on the work we've been doing on Sperowider, an archiver robot designed to work on the Erowid site.
How to Spider Erowid #
With the world-wide reach of Erowid.org, it is not uncommon for dozens of spiders to be running concurrently and one poorly behaved spider can slow down access for every visitor to the site. If you use spidering software or an archiver robot on Erowid, you will be expected to keep very close watch on it as it runs. If it hits the server for more than 10 hits per second or downloads more than 1,000 pages in a day without following our guidelines, your access to the site may be blocked.Happily, there are several ways you can now archive Erowid in a community-friendly way:
List of Downloadable Archive Locations #
- de1.erowid.org/erowid_archives/ : Hosted by JL
List of Mirror Locations #
- de1.erowid.org : Hosted by JL