 |
 |
Hello Joe, Thanks for your thoughful response. > The problem is, a few years ago, that spammers started sending bogus > requests to servers, to try to get them to show up in your stats pages. > In ORA's case, they're only showing the top 20, and they presumably get > lots of requests, so someone would have to hit them pretty hard to get > something to show up. I'm guessing they're using the referer from the server log, parsing the search attributes, and generating the tags. So, tags represent hits in Google (and other engines) that actually generated a link to O'Reilly's sites. Presumably, a search on "Brittany Spears" won't generate such a link. However, to ensure that spammers can't circumvent this, they seem to be matching the potential tags back against their index to make sure they match items on the site. Since "Brittany Spears" is not in their index, it shouldn't match as a valid tag. I guess it depends on what constitutes an index item on their site. Probably not the full-text of all their blogs and articles where "Brittany Spears" may very well be mentioned. > If you're thinking about exposing your server logs, I'd recommend the > following: > > 1. Don't give out IP addresses of the requestors > (privacy reasons) Yep. Bibliominers often recommend anonymizing IP addresses if that information is going to be part of the analysis. This mostly pertains to usage data of the website or of licensed e-resources, where you chuck the identifying IP address but keep certain demographic information (if you have it) such as what department or status (faculty, student, staff, etc) they represent. Even then it is tricky since usage of the "Journal of an extremely niche topic" might, in itself, identify a unique user on campus. > 2. Don't put on a public page any data that's generated by the > user-agent, to include HTTP_USER_AGENT, HTTP_REFERER and > QUERY_STRING. All have been used by spammers to insert URLs to > try to get links back to their sites. Yep. Privacy would be a concern, for sure. I _think_ O'Reilly's approach should work so that searches on, say, personal names tied to some controversial idea won't get broadcast as a tag. > 3. Filter out all entries with 'error' results (people trying to > probe your system for vulnerabilities, etc.) If we just pull the referers from specific search engines, this concern ought to be addressed, but a second layer of filtering would definitely be in order. > 4. Filter out all 'intranet' pages or other pages that the general > public shouldn't be going to. Yep. > 5. Avoid giving information that provides signatures of the CMS > you're using, or other signatures of potential vulnerabilities. Makes sense. > 6. Use robot.txt to request search engines to not serve whatever > pages you generate. O'Reilly's approach is to regenerate an internal search on their site using the tag(s) as the query. Since the tag cloud is isolated to a single page, is this limit really necessary. I suppose it does make sense to not have the tag indexed by a search engine since the tags will change over time. > For the particular case of generating tag clouds from search results, the > problem lies in that you typically need to use QUERY_STRING if it's a > local search script, and HTTP_REFERER if it's a remote search engine that > linked to you. Both values can't be trusted. > > In this particular case, I probably wouldn't try a fully automated > approach -- I'd generate the page, but require someone to manually verify > it before it got posted. Yeah. Probably a good idea. Part of what got me thinking about this was listening to John Blyberg talking about Ann Arbor's catalog tagging (similar to Penn Tags) on a Talking with Talis podcast from last fall. It is a different deal than server logs, of course, since tagging is a voluntary activity. Different privacy issues although appropriateness and spam remain as concerns. http://www.aadl.org/catalog I was also thinking about other analysis tools such as Cloudalicious which graphs trends in tagging distribution on del.icio.us. It would be fun to track trends in tag popularity. http://cloudalicio.us/tagcloud.php -- Tom