Legality of content scraping & republishing

hi, Matt, how's life?

I remember asking a similar question several years ago in a law lecture - there was no definite answer then. A couple of years later there was a test case where search engine caching was ruled 'fair use'.

Anyway, on to your problem...

If I was you, I would:

1. Publish snippets from scraped pages, with links through to the full version.
2. Include full content only from feeds, or content where you've done some kind of licensing deal.
3. Put a mechanism in place so that you could switch off particular sites from content scraping if they requested it.

In terms of robots.txt, I would:

1. Avoid crawling any restricted sites
2. Avoid publishing any content where any kind of 'no-cache'/'no archive' meta tags have been set (that's probably what's happening with the sky content you mention?)

I hope that helps you - would be interested to hear more about what you're  doing!

daniel