Here are some steps I’ve taken towards delivering a more reliable RSSmeme:
- All static files are being served by S3. My server is set to redirect any requests to static files (like the widget) to the file on S3. This means that if RSSmeme slows down and you have the widget installed; your site will not be affected. Occasionaly S3 has a breakdown; if you are concerned with that affecting your site then please install the FeedBurner FeedFlare instead. It is just as easy and the only way it will slow your site is if FeedBurner’s servers slow down (which I find highly unlikely).
- The real culprit here wasn’t entirely static files. It was FriendFeed Stats. The global statistics page was tying up an apache process for a long time while it calculated the results. My server doesn’t have much memory so stealing an apache process is a very big deal. Imagine 10 of you hitting the same global FriendFeed Stats page with an unprimed cache? All of my apache processes get stuck, other users get angry, and chaos ensues. So I’ve disabled the global statistics pages. All user specific pages will continue to work.
- Over 250 different IPs (all requesting bogus URLs clearly trying to get access to my server) have been dropped using iptables. I still need to figure out a better way to automate this process. I think if an IP frequently requests a URL that results in a 404 then we can safely drop them instead of wasting time processing their requests.
Again; I’m sorry! But things are much better now.