I’ve tweaked the way BBC news articles are parsed for Revisionista. Unfortunately this means you’ll see a flurry of new revisions, with no actual changes (though the whole article will be marked as changed).
I also fixed the Guardian news article title parsing - they changed things around a bit. This shouldn’t result in any flurries.
The BBC Editors blog has mentioned News Sniffer today.
It’s largely just marginalised us, but it’s rather ambiguous. When they suggest that Revisionista will not find examples of bias, I can’t decide if they mean that the BBC is not biased, or that they are just very good at it.
Some of the recommended revisions are interesting, but maybe we need a comments feature so people can explain and discuss their recommendations.
And they also describe their censoring of ‘Have Your Say’ comments as “censoring” in quotation marks. I’m not sure what this is supposed to mean either. Is it not censorship when they remove comments? Or do they not remove comments?
See the top recommended censored comments for some interesting examples of “censorship”.
I rolled out a new version of News Sniffer last night. A lot of work went into rejigging the censored comment detection to make sure we don’t make mistakes. I also wrote a system to confirm censored comments by html scraping, which gives us a way to double check censorship.
This also allowed me to check the entire backlog of censored comments. There were a number of comments that we thought were censored but were not and those are now fixed. We also found a number of comments that we didn’t know had been censored.
Apologies for this. My understanding of the BBC HYS RSS feeds was flawed (to be frank, largely due to some brain-deadness on the part of the BBC forum software).
So you might notice that some existing censored comments disappeared but other ones appeared for the first time. I’ll be running the confirmation script regularly to monitor the new checking system (though not as regularly as the others as it’s a bit intensive).
The “Recommended Comments” page now lists the latest recommended comments, not the highest recommended as before. Comments in order of highest recommendations can now be found on the “Top Recommended Comments” page.
The thread listing pages now displays the number of published comments along with the number of censored. This helps give an idea of how busy a thread is (you might expect busier threads to be more censored).
And lastly, the thread display page now includes the BBC HYS description, which gives a bit of background to the thread and links to any news articles that might have prompted the it.
Thanks to datamining.typepad.com and currybet.net for the feedback that led to some of these improvements.
News Sniffer blipped on the BBC radar this week. Someone mailed their public “backstage” mailing list with a link and it eventually turned up on the backstage.bbc.co.uk blog.
Richard Sambrook, Director of BBC Global News, has commented too.
I’m not sure why he’s so Interested in what my name is, but I’ve added it clearly to the about page now. It’s not like it was a big secret.
An ex-BBC employee, Martin Belam, commented extensively on it too.
I also got invited to discuss News Sniffer on the backstage mailing list by Ian Forrester of the BBC. An archive of the mailing list is available. With a notable threads starting here. I posted some details of the inner workings of News Sniffer too.
The ‘Watch Your Mouth’ system is currently marking some comments as censored when they are not. This seems to be due to the BBC’s servers being out of sync with each other and I get out of date RSS feeds. I have a solution for this and am working on it.
This was brought to my attention (very gracefully) by a BBC employee.
UPDATE:: Problem fixed. No more comments should be mis-classified. I’m working on verifying the backlog, but it should only have been a small number of comments. It was a bit of a corner case causing the problem.
In drumming up some more attention for News Sniffer, I e-mailed the Biased-BBC blog which then linked us in a posting. Some comments on their post brought up the issue of News Sniffer Bias. Here is a quick response I put together:
I rolled out the new version of News Sniffer last night. It’s mostly changes to Revisionista:
The search system is a little crummy at the moment. I have plans to add a more advanced searching system in future.
Anyway, get recommending revisions!
Since the Lenin’s Tomb blog linked us, traffic has snowballed. News Sniffer has now appeared on Metafilter.com, Democratic Underground, Digg.com and a whole bunch more.
We even made it onto the BBC Radio 5 Live discussion forums. I imagine they’ll censor that though 
UPDATE: Medienbote, Germany’s daily media industry newsletter also featured News Sniffer.
Also, I noticed a link from what looks like an internal BBC development web site (they use MoinMoin as their wiki software ).
I was interviewed by Austria’s public service broadcaster, Österreichischer Rundfunk. The article is here in German. My actual e-mail reply to the journalist follows.
Read the rest of this entry »
I’ve made a few changes to fix the completely screwed layout that Internet Explorer users were seeing. Sorry about that.
The latest News Sniffer project went live on Sunday: Revisionista. It tracks changes in corporate news articles and marks the differences. So you can choose a BBC news article and see how it’s changed since it was first created. Most changes are on breaking new articles which get updated as more information becomes available, but some changes are rather telling of policy.
Currently only the BBC is monitored, but it’s pretty easy for me to add support for any site with an RSS feed (which I’ll be doing soon). Suggestions of what sites to monitor?
Some examples:
Read the rest of this entry »