When the BBC discovered News Sniffer, I was invited to discuss it on their techie lists. I mentioned a few of the problems I’d had with the feeds such as duplicate entries, a lack of useful caching HTTP headers and the huge size of the feeds. In response to this they looked into it and fixed the duplicate entries within a couple of weeks.

Yesterday they changed the default size of the feeds but also rejigged the RSS format. This broke Watch Your Mouth in a number of ways (mostly affecting only new threads since yesterday):

• The timestamps of new comments weren’t reported correctly
• The author details of new comments were missing
• New thread description details missing

Due to a combination of some of the above, some comments were marked censored when they were in fact published. I’ve adjusted Watch Your Mouth in response to these changes and it’s working ok again now. I’ve also run the clean-up scripts so any published comments marked censored have been restored – you might notice a bunch of “censored” comments disappear from the indexes.

These kinds of problems are expected when you’re monitoring data that’s in the control of someone else (especially in ways they might not have ever intended). I just need to keep an eye on the situation and make alterations accordingly when problems arise.

To compare it to an “arms race” isn’t quite right because there is no evidence that the BBC are purposefully making life difficult for us. In fact, these changes are actually helpful.

UPDATE: Due to the combination of malformed BBC RSS timestamps (hours going from 0000 to 2400?!) and a bug in Watch Your Mouth, we’ve been missing a lot of potentially censored comments on some threads for the last 3 or 4 days. I’ve now written a workaround to this quirk so things should be back to normal.