News Sniffer Blog http://www.newssniffer.co.uk/blog Monitoring the corporate media Fri, 14 Nov 2008 19:05:56 +0000 http://wordpress.org/?v=2.6.2 en The Register references News Sniffer http://www.newssniffer.co.uk/blog/2008/11/14/the-register-references-news-sniffer/ http://www.newssniffer.co.uk/blog/2008/11/14/the-register-references-news-sniffer/#comments Fri, 14 Nov 2008 19:05:56 +0000 johnl http://www.newssniffer.co.uk/blog/?p=27 The Register used News Sniffer’s Revisionista to track the changes of a BBC article on the ID cards.

The report’s revision history, documented by News Sniffer, takes us on an impressive Odyssey from “Smith to unveil airport ID scheme”, through “Shops may take ID Card biometrics” to the final sales-pitch version, cunningly deniabled with quote marks.

http://www.theregister.co.uk/2008/11/14/bbc_stories_cant_wait/

]]> http://www.newssniffer.co.uk/blog/2008/11/14/the-register-references-news-sniffer/feed/ Updates not working for the last week http://www.newssniffer.co.uk/blog/2008/06/14/updates-not-working-for-the-last-week/ http://www.newssniffer.co.uk/blog/2008/06/14/updates-not-working-for-the-last-week/#comments Sat, 14 Jun 2008 01:51:01 +0000 johnl http://www.newssniffer.co.uk/blog/2008/06/14/updates-not-working-for-the-last-week/ Both Revisionista and Watch Your Mouth have not been updating correctly for about a week due. Working again now. Sorry for the outage.

]]> http://www.newssniffer.co.uk/blog/2008/06/14/updates-not-working-for-the-last-week/feed/ Watch Your Mouth down for a week http://www.newssniffer.co.uk/blog/2007/10/10/watch-your-mouth-down-for-a-week/ http://www.newssniffer.co.uk/blog/2007/10/10/watch-your-mouth-down-for-a-week/#comments Wed, 10 Oct 2007 18:06:59 +0000 johnl http://www.newssniffer.co.uk/blog/2007/10/10/watch-your-mouth-down-for-a-week/ The BBC upgraded their forum software on the Have Your Say site and it broke News Sniffer. If you’re interested, they’ve changed from using the “Thread ID” in their urls to using a “Forum ID”. I’ve done a quick patch up to fix things, but any comments censored in the last week or so are lost to News Sniffer. It’sl going to take a while whilst things get back up to speed, but it should be done by tomorrow morning.

Thanks to the News Sniffer reader who spotted the lack of updates and got in touch with me. It was a relatively quick fix once I knew about it.

]]> http://www.newssniffer.co.uk/blog/2007/10/10/watch-your-mouth-down-for-a-week/feed/ Now including BBC health articles http://www.newssniffer.co.uk/blog/2007/04/22/now-including-bbc-health-articles/ http://www.newssniffer.co.uk/blog/2007/04/22/now-including-bbc-health-articles/#comments Sun, 22 Apr 2007 15:27:12 +0000 johnl http://www.newssniffer.co.uk/blog/2007/04/22/now-including-bbc-health-articles/ I’ve just added the BBC health feed to Revisionista so we’ll start monitoring more health related articles. So far, the only health articles included were from the World or UK feeds.

]]> http://www.newssniffer.co.uk/blog/2007/04/22/now-including-bbc-health-articles/feed/ Now excluding BBC sport articles http://www.newssniffer.co.uk/blog/2007/04/22/now-excluding-bbc-sport-articles/ http://www.newssniffer.co.uk/blog/2007/04/22/now-excluding-bbc-sport-articles/#comments Sun, 22 Apr 2007 15:20:01 +0000 johnl http://www.newssniffer.co.uk/blog/2007/04/22/now-excluding-bbc-sport-articles/ Revisionista now explicitly excludes BBC sport articles from it’s monitoring and I’ve deleted any from the last couple of months. This is because these articles generate a vast number of revisions mostly due to score updates during games. This is wasting News Sniffer resources that could be better spent monitoring other articles.

]]> http://www.newssniffer.co.uk/blog/2007/04/22/now-excluding-bbc-sport-articles/feed/ Upgrade: wym comment cleanup, downtime, and improved search http://www.newssniffer.co.uk/blog/2007/04/20/upgrade-wym-comment-cleanup-downtime-and-improved-search/ http://www.newssniffer.co.uk/blog/2007/04/20/upgrade-wym-comment-cleanup-downtime-and-improved-search/#comments Fri, 20 Apr 2007 12:13:53 +0000 johnl http://www.newssniffer.co.uk/blog/2007/04/20/upgrade-wym-comment-cleanup-downtime-and-improved-search/ News Sniffer was down for a couple of hours yesterday afternoon whilst I upgraded the software that runs it. This fixes a few display bugs and more importantly patches up another problem where bbc comments were being misclassed as censored. The misclassification was due to a mistake when handling british summer time, so it’s only been prevalent since the end of March. We double checked all censored comment from the last few months and removed any misclassified ones.

Due to the downtime whilst we cleaned and reindexed the database, some news articles and bbc comments were not added to the database so will not be tracked. So, if you’re looking for a particular news article published yesterday afternoon, it’s likely that it is missing. Apologies for the inconvenience.

The biggest new feature is vastly improved search system.

It works like most search engines, so just type keywords in and you get results. You can be a bit more advanced too though. Say you want to find all censored bbc comments by the author ‘John Smith’. Do a search for:

author:"john smith"


You want all the comments mentioning the NHS but to exclude any mentioning Patricia Hewitt:

nhs -hewitt

Maybe you want to find all censored bbc comments with the words ‘iran’ and ‘bomb’ made in April 2007:

iran bomb created_at:200704*

To find a particular bbc comment, you can search on the bbc’s own comment id number:

bbcid:bbcid:2481215

To get all Revisionista news articles with the work ‘iraq’ in the title published today (19th April 2007):

title:iraq created_at:20070419

The BBC Have Your Say search is here, and the Revisionista news article search is here. Happy searching!

]]> http://www.newssniffer.co.uk/blog/2007/04/20/upgrade-wym-comment-cleanup-downtime-and-improved-search/feed/ New domain name http://www.newssniffer.co.uk/blog/2007/04/09/new-domain-name/ http://www.newssniffer.co.uk/blog/2007/04/09/new-domain-name/#comments Mon, 09 Apr 2007 15:36:01 +0000 johnl http://www.newssniffer.co.uk/blog/2007/04/09/new-domain-name/ News Sniffer now has its own domain name: www.newssniffer.co.uk
The old address (newssniffer.newworldodour.co.uk) will automatically redirect you, but update your bookmarks!

]]> http://www.newssniffer.co.uk/blog/2007/04/09/new-domain-name/feed/ No updates for the last 2 days http://www.newssniffer.co.uk/blog/2007/03/01/no-updates-for-the-last-2-days/ http://www.newssniffer.co.uk/blog/2007/03/01/no-updates-for-the-last-2-days/#comments Wed, 28 Feb 2007 23:23:01 +0000 johnl http://newssniffer.newworldodour.co.uk/blog/2007/03/01/no-updates-for-the-last-2-days/ Apologies but due to a VPN problem, News Sniffer has not been monitoring news articles or Have Your Say forums for the last 2 days. All working again now though. I’ll set up some alerting system to prevent this happening again.

]]> http://www.newssniffer.co.uk/blog/2007/03/01/no-updates-for-the-last-2-days/feed/ BBC UK Politics feed added to Revisionista http://www.newssniffer.co.uk/blog/2007/01/18/bbc-uk-politics-feed-added-to-revisionista/ http://www.newssniffer.co.uk/blog/2007/01/18/bbc-uk-politics-feed-added-to-revisionista/#comments Thu, 18 Jan 2007 13:56:51 +0000 johnl http://newssniffer.newworldodour.co.uk/blog/2007/01/18/bbc-uk-politics-feed-added-to-revisionista/ I’ve just added the BBC UK Politics section to Revisionista, so articles from it will be monitored for changes from now on.

]]> http://www.newssniffer.co.uk/blog/2007/01/18/bbc-uk-politics-feed-added-to-revisionista/feed/ BBC fixes RSS feeds - breaks Watch Your Mouth http://www.newssniffer.co.uk/blog/2006/12/08/bbc-fixes-rss-feeds-breaks-watch-your-mouth/ http://www.newssniffer.co.uk/blog/2006/12/08/bbc-fixes-rss-feeds-breaks-watch-your-mouth/#comments Fri, 08 Dec 2006 17:01:58 +0000 johnl http://newssniffer.newworldodour.co.uk/blog/2006/12/08/bbc-fixes-rss-feeds-breaks-watch-your-mouth/ When the BBC discovered News Sniffer, I was invited to discuss it on their techie lists. I mentioned a few of the problems I’d had with the feeds such as duplicate entries, a lack of useful caching HTTP headers and the huge size of the feeds. In response to this they looked into it and fixed the duplicate entries within a couple of weeks.

Yesterday they changed the default size of the feeds but also rejigged the RSS format. This broke Watch Your Mouth in a number of ways (mostly affecting only new threads since yesterday):

  • The timestamps of new comments weren’t reported correctly
  • The author details of new comments were missing
  • New thread description details missing

Due to a combination of some of the above, some comments were marked censored when they were in fact published. I’ve adjusted Watch Your Mouth in response to these changes and it’s working ok again now. I’ve also run the clean-up scripts so any published comments marked censored have been restored - you might notice a bunch of “censored” comments disappear from the indexes.

These kinds of problems are expected when you’re monitoring data that’s in the control of someone else (especially in ways they might not have ever intended). I just need to keep an eye on the situation and make alterations accordingly when problems arise.

To compare it to an “arms race” isn’t quite right because there is no evidence that the BBC are purposefully making life difficult for us. In fact, these changes are actually helpful.

UPDATE: Due to the combination of malformed BBC RSS timestamps (hours going from 0000 to 2400?!) and a bug in Watch Your Mouth, we’ve been missing a lot of potentially censored comments on some threads for the last 3 or 4 days. I’ve now written a workaround to this quirk so things should be back to normal.

]]> http://www.newssniffer.co.uk/blog/2006/12/08/bbc-fixes-rss-feeds-breaks-watch-your-mouth/feed/