Cynic's Soapbox

We’ve been using SQL for storing various Spamassassin data for the last few years. It’s allowed us to provide our users with a lot of control and has been a good solution. However, a mixture of a inattention and a failure of some of the maintenance actions has led to a problem with the bayes tables growing to a size that is beyond sensible. We’ve started taking action to remedy the issue, but as we’re presently using MySQL the size of the database files isn’t shrinking The effort involved in correcting this is large enough to make me wonder whether it’s an opportunity to migrate to Postgres. Has anyone done this or have any opinions whether it’s a sensible move to make?

I’ve been developing a small app that should be a good fit for CouchDB. It’s a small app that will allow me to keep a series of notes, some of which are plain text and some are formatted in particular ways. The data needs to be shared between a number of devices with the ability to be edited on any of them – online and offline. It’s not a hugely complex app nor is it a unique problem, but it’s useful place to keep looking at CouchDB.

Why do I think CouchDB is a good fit? It’s NoSQL foundations means that data can be easily arranged and stored in ways that make sense for each type. The replication features will allow the various devices to sync with each other without any extra effort. Thanks to the Android Couchbase project it’s possible to use CouchDB on Android (albeit with a larger binary than I would like) allowing for offline storage.

As I’ll be using multiple devices, I’ve decided to start by letting the server allocate it’s default random _id which should avoid duplicates and not require me to devise some more complex scheme.

With a conventional SQL database the decision of how to arrange the data would be simple – a series of tables connected with joins. However, when contemplating how to arrange the data with NoSQL there doesn’t seem to be as simple a solution. This could be my inexperience with them but having read a lot of articles I think it’s just there are so many possible ways of arranging the data that the right solution will depend on the situation.

I’m keen to keep the complexity down but at the same time I don’t want to have a simpler database at the expense of making the interface more complex to work with. My original thought was to store all the related documents in an array embedded within a single document, for example

{
"_id": "12345",
"title": "Some Subject"
"notes": [
{
"type": "text",
"content": "Simple text note"
},
{
"type": "text",
"content": "Another simple piece of text"
}
]}


This is nice and simple and allows me to use a single request to get the entire document, but it adds a lot of complexity to the interface. When editing one of the notes, I would need to

1. request the document
2. edit the document
3. update the entire document

Far simpler from an editing point of view would be to have the notes as seperate records with a link back to their parent document. For example,

{
"_id": "12345",
"title": "Some Subject"
}
{
"_id": "23456",
"parent": "12345",
"type": "text",
"content": "Simple text note"
}
{
"_id": "34567",
"parent": "12345",
"type": "text",
"content": "Another simple piece of text"
}


This allows me to edit each document separately and so avoids the need to request the entire parent document and then manipulate the returned data prior to updating, but does raise the question of how I could efficiently request the document and all it’s related child documents. With some simple changes, a complex key in a view and careful use of parameters it turns out to be very possible.

First I alter the documents slightly by adding a type field to every document.

{
"_id": "12345",
"title": "Some Subject",
"type": "subject"
}
{
"_id": "23456",
"parent": "12345",
"type": "subject:note",
"content-type": "text",
"content": "Simple text note"
}
{
"_id": "34567",
"parent": "12345",
"type": "subject:note",
"content-type": "text",
"content": "Another simple piece of text"
}


Next I write a simple view function.


function(doc) {
if (doc.type) {
if (doc.type == "subject") {
emit([doc._id, 0], doc);
} else if (doc.type == "subject:note" && doc.topic) {
emit([doc.topic, 1], doc);
}
}
}

Running this query (without any parameters) produces the following result,


{"total_rows": 3, "offset": 0, "rows": [
{
"id": "12345",
"key": ["12345", 0],
"value": {...}
}, {
"id": "23456",
"key": ["12345", 1],
"value": {...}
}, {
"id": "34567",
"key": ["12345", 1],
"value": {...}
}
]}


The ‘…’ in the value fields will have the contents of the document, so this query returns me all the information I need. Filtering is also possible thanks to CouchDB and the usual key, startkey and endkey parameters. Additionally I can add fields to the complex key without changing how it functions. Nice.

Getting a single subject with all the related notes is as simple as requesting the view with the parameters ?startkey=["12345"]&endkey;=["12345", 2].

If I add additional document types then I can simply adjust my view and return them as well, using the number to differentiate between them. Parsing the data for an interface becomes simple and each document can be edited/updated in isolation.

This is another of the “this is to help me remember what I did to cure this problem” type posts.

Having recently updated the mail server an old issue resurfaced that caused one of my users to be unable to send mail correctly. We use SMTP AUTH to authenticate users and only allow relaying for authenticated users. It’s not an unusual configuration but as postfix is installed by Ubuntu it has this line in the main.cf config file

smtpd_tls_auth_only = yes

With this line only connections that are encrypted will offer the AUTH command. While this is great for security it means that users who ignore my advice to use encrypted connections are unable to relay mails.

I’ve commented out the line in main.cf allowing the default value of ‘no’ to be used which has allowed all users to send correctly. Of course, warnings have also been given to the affected users along with a reiteration of the advice to use an encrypted connection

As with many people I’m using the 64-bit version of Ubuntu 12.10. This means that when you install the various pieces you need for android development the tools found in the platform-tools directory won’t run. They give the simple error message

bash: android-sdk/platform-tools/adb: No such file or directory

The solution is simple enough, but it took me a few minutes to find, so maybe this post will save someone else that time.

The issue was that the apps are 32-bit and my install is 64-bit, so you need to install the 32-bit versions of the system libraries to make them happy.

sudo apt-get install ia32-libs

Et voila! The apps now run.

I’d suggest doing this before installing the eclipse ADT as it will check whether the apps can be run and if they can’t will produce some very strange and unhelpful errors when you try and look at an android project.

I did find a web page describing installing the apps via a ppa which claimed to avoid the need for the 32-bit libs, but when I tried it I couldn’t find any way of telling eclipse where to find the apps installed via the ppa and so was no further forward. If anyone knows the magic incantations to make this work then let me know as I’d prefer not to need the 32-bit libs.

Additionally, when trying to run the emulator I saw the following errors

[2013-01-06 17:58:48 - Emulator] Failed to load libGL.so
[2013-01-06 17:58:48 - Emulator] error libGL.so: cannot open shared object file: No such file or directory
[2013-01-06 17:58:48 - Emulator] Failed to load libGL.so
[2013-01-06 17:58:48 - Emulator] error libGL.so: cannot open shared object file: No such file or directory

The solution was provided by this blog post.

Making changes to a large project isn’t an easy decision and for a company like Google I’d imagine that such decisions are debated extensively, so when such changes are made I assume they bring genuine benefit. Having recently started using a Nexus 10 with Android 4.2.1 I was stunned when connecting it to the computer not to be able to simply transfer documents to it. A little googling around soon showed that the USB connectivity I’m used to in Cyanogen 9 is not available. It’s been replaced by the Media Transfer Protocol.

As MTP is a Microsoft developed protocol, connecting the table to Windows 7 works flawlessly and exactly as expected.

My primary desktop environment is Linux, and here the situation is less positive. Despite a lot of googling around I still can’t find a solution that works reliably. I have a script that will connect it, but that shouldn’t be needed. Simply connecting the device should enable me to view the contents and access the storage.

There are several projects and many websites that claim to show you how to do this, but having tried many of them I still can’t get it to work. Additionally, most of the guides require editing system configuration files! Last I checked it’s 2013 and yet to enable basic functionality I have to make changes like it’s 2008?

I can understand that Google made the change with the best of intentions, but I can’t help but feel disappointed that they didn’t expend some time and energy making sure that Linux (upon which Android depends) had full support.

I prefer to have the window buttons on the right, which I finally managed to do in gnome-shell using

gconftool-2 --set /apps/metacity/general/button_layout --type string ":minimize,maximize,close"

Happy days.

After less than 24 hours with Unity, I installed gnome-shell. After some 30 minutes I have a desktop environment than looks far closer to the way I want it to. Things are where I expect them to be and easily accessible. What’s more it still feels like Ubuntu

After installing the key to customising is to add various shell extensions, but thanks to https://extensions.gnome.org/ that is about as straightforward it can be

In case anyone is interested, the extensions I have added are

• Frippery Applications Menu by rmyorston
• Frippery Bottom Panel by rmyorston
• Frippery Panel Favorites by rmyorston
• Remove Accessibility by lomegor

Once installed the “dock” can still be accessed by moving the mouse to the top right corner and has all the usual functionality available (useful for adding items to the favourite menu).

Just under a year ago I swapped from Ubuntu to Mint. This was a reaction to the awful Unity desktop that they had decided to use as the default in 11.10. The reaction to my blog post about the change showed I wasn’t the only one who felt that way, but after living with Mint for a year I’ve swapped back to Ubuntu.

Throughout the year I’ve been using Mint it’s been a reasonable choice, but their update/upgrade mechanism requires a lot more care and feeding than Ubuntus and the system never seemed as up to date as I had experienced when using Ubuntu. Additionally I had ongoing niggles with some of the hardware on the machine (wifi and sound being the 2 main culprits) which seemed impervious to all fixes I tried. As I’ve left my days of kernel hacking behind I’d rather just use the machine than spend time debugging intermittent oddities. The lack of certain packages in apt on Mint finally pushed me to think about a switch, so as the effort involved in moving to Mint 14 was the same as moving to Ubuntu 12.10 I decided to make the switch.

The switch was also encouraged by several people who have commented that recent changes to Unity have made it less horrible and easier to deal with, so as I believe in second chances, I decided it was worth a try.

After backing up my entire partition, downloading and burning the 12.10 Desktop ISO I booted the machine using a USB DVD player and waited. Sure enough Ubuntu appeared and asked me to install. Simple. Adding the wifi password got me online and the installation was as painless as I remembered. After a reboot it was done.

The biggest question I had was whether I could cope with Unity or would have to change to gnome-shell. First impressions weren’t encouraging as I was confronted by a list of items the systems thought I should buy!? WTF! This was almost enough to make me ditch the whole experiment and grab a Mint 14 ISO, but in the spirit of giving it a try I continued on. After some google searches and some preference changes the nasty adverts went away, though with them they (apparently) took some of the other functionality. Removing most of the default icons from the “bar” on the left was easy enough – adding the ones I wanted was much harder, but eventually I managed. Finding the apps I wanted to run was a challenge and mostly involved typing the first few characters of their name into the dash search bar. Seriously? Not the best start.

As for the “Dash” and the various “lenses” that people seem to be excited about, I don’t really get it. It’s my machine and my desktop with apps on it that I have chosen. I should be able to arrange things in a way that makes sense to me. On an average day I only use a few apps and having them easily available to me is important. If I wanted to have how I should use my computer dictated I would be running OSX on a Mac.

Loading some of my media from the backup proved more encouraging as everything played straight away. The days of having to find odd packages to get the right codecs appear to have been banished.

I may be showing my age, but I prefer my window buttons to be on the right, so I had to run

gsettings set org.gnome.desktop.wm.preferences button-layout ":minimize,maximize,close

Simple enough, but couldn’t there be an easier option?

Overall Unity has improved, but still lacks in many areas. Too much of Unity seems directly cloned from OSX, right down to the little arrows next to the app icons in the bar thing. As I don’t really like how OSX works, this doesn’t feel like a positive direction to me. Nor does it feel as though they are innovating. For all it’s faults, Windows 8 does at least feel fresh and innovative.

Will I stick with Unity? I’m not sure yet, but I wouldn’t put your money on it.

When I started Variable Pitch my intent was to display the information in a clear, unbiased way as a counter to the frequently emotional debate around wind energy. It’s likely very naive but I believe that the information should speak for itself and people should frame their decisions based on that information rather than their emotional views.

It’s been heartening to have been contacted by several people about the site, often with positive suggestions and comments, but what’s even more heartening is that those contacting me have come from both sides of the issue. Could this mean my hopes for the site weren’t quite as naive as I thought?

Tomorrow will mark the start of 2013 and I’ll be watching to see that the site transitions into the new year in as smooth a way as it can. I’m hopeful it won’t cause any trouble, but time will tell.

Happy New Year to everyone.

The following is a list of the members of the Perth & Kinross Development Management Committee which will decide on the application for a monitoring mast on Wednesday this week.

• Tom Gray (SNP) tomgray@pkc.gov.uk
• Bob Band (SNP) bobband@pkc.gov.uk
• Henry Anderson (SNP) handerson@pkc.gov.uk
• Michael Barnacle (IND) mbarnacle@pkc.gov.uk
• Ian Campbell (CON) icampbell@pkc.gov.uk
• Ann Gaunt (LIB) agaunt@pkc.gov.uk
• Joe Giacopazzi (SNP) jgiacopazzi@pkc.gov.uk
• Callum Gillies (LAB) CGillies@pkc.gov.uk
• Alan Jack (IND) hajack@pkc.gov.uk
• John Kellas (SNP) JKellas@pkc.gov.uk
• Alan Livingstone (CON) alivingstone@pkc.gov.uk
• Murray Lyle (CON) MLyle@pkc.gov.uk
• Gordon Walker (SNP) gordonwalker@pkc.gov.uk

Update

For some reason the application has been withdrawn from the meeting tomorrow so will probably not be debated now until the new year. What this means is unclear at present. Time will tell.

The council have considered the application for a meteorological monitoring mast and as expected it has been referred to the Development Management Committee. This group of 13 councillors will now decide on whether to approve the application. Sadly the Development Quality Manager has recommended that it be approved, though this is based purely on the application and makes no allowance for the prospect of 4 120m turbines being built on the site. What a strange world we live in where such details are considered irrelevant

Interestingly the report for the committee does contains some comments that mirror those that the objectors raised.

Ministry of Defence
28  No objection subject to mast being fitted with aviation lighting.

and

33 The application site is located on a relatively exposed area of open farm land that is surrounded by a landscape that is characterised by undulating lowland rural countryside, small areas of woodland, scattered farmsteads and small settlements/hamlets. Whilst the countryside in this area is not subject to any specific landscape designation it is an attractive area of rural countryside.

34 The erection of a 90m high meteorological mast will be by far the tallest structure within the area and it will be visible for some distance across the surrounding countryside.

This paragraph then goes on to say…

Nevertheless it is considered that the slender, slim-line form of the mast with its narrow high tension wire stays will serve to minimise the visual impact of the structure.

I’m not sure if this is meant as humour or not, but as it will have a very bright red light on top of it I’m not convinced the impact will ever be minimised!

38 Whilst the application site is not within any identified protected wildlife sites, Methven Moss SSSI is located approximately 700m to the north of the site and Dupplin Loch SSSI/SPA is located 2.5km to the south east. The proposals will not directly affect either site but it is noted that Dupplin Loch is identified as an RSPB Important Bird Area.

Of course this then continues with

The RPSB have stated that they do not usually comment on met masts as there is no evidence that they pose a risk to birds. However, they advise that an exception to this is if they are located in a particularly sensitive location where a higher risk of collision with the guy wires may be expected (e.g. immediately near a large roost or breeding colony, particularly if this is a designated site i.e. an SSSI or SPA). In such circumstances they recommend that the Council consider marking the guy wires with brightly coloured reflectors a condition of
the planning consent. The Council’s own Bio Diversity Officer has also
recommended that bird deflector discs should be fitted to each guy wire.

So basically, the birds are on their own. Not only will we have a mast that will generate all sorts of odd noises as the wind whistles through the guy wires, but the presence of the bird deflectors will add additional odd noises. Hmm, that’s certainly not going to bother the birds is it? Given that the field immediately to the south of the site was home to several thousand migrating geese as recently as a week ago, this seems like a cavalier response.

As for the local bat populations? They’re not mentioned at all.

42 A number of concerns have been raised in regards to the potential light pollution from the aviation lighting required by the MOD. Whilst it is acknowledged that the aviation lighting will be visible during the night it is not considered that this will create any significant level of light pollution. It should also be noted that this type of light is common place on many large structures such as telecommunications and television masts that are found throughout
rural areas. In addition, if consent is granted then it would be for a temporary period and therefore the potential for light pollution would in any event be similarly limited to a temporary period.

What telecommunication or television towers? There aren’t any in this area which is why we have such a great view of the night sky. Apparently anything that makes an area desirable to live in can be ignored

I’m preparing a letter that I will send to the 13 members of the committee in the hope that they can see the sense in rejecting this proposal, though I feel it’s unlikely that they’ll be brave enough to go against the advice of their planning department. It’s interesting to note that despite the large number of local objections this proposal is still given preferential treatment in the current “rush for wind”.

As for the consideration of this proposal in isolation, this paragraph is interesting.

44 It is not considered that a meteorological mast falls within any of the accepted categories of development outlined within The Town and Country Planning (Environmental Impact Assessment) (Scotland) Regulations 2011. As such this development does not require the Council to adopt a screening opinion. However, it should be noted that a scoping request under paragraph 14 (1) of EIA Regulations has been submitted to the Council for a wind farm development comprising of 4 turbines at this location.

So there we have it. Consider this application in isolation despite the fact that there is a clear indication of the intent.

Given that Ecotricity are based hundreds of miles away their proposed wind farm will bring virtually no economic benefit to this area, yet the planners who are responsible for protecting this area seem to be bending over backwards to accommodate them. Surely that’s not how it’s supposed to work?

“In war, truth is the first casualty.”

Aeschylus Greek tragic dramatist (525 BC – 456 BC)

You don’t have to spend long looking at information about wind farm developments before the truth of the above statement becomes apparent. Whichever side of the argument people are on they make claims and counter claims that are difficult to verify. In such an atmosphere the smallest fact is repeated often enough that it attains religious status, resisting any attempt to validate it. Sadly this does nothing to further the debate.

When Ecotricity proposed building 4 120m turbines 700m from our house my interest in the subject grew. I found myself wanting to look at the facts and figures behind the attention grabbing claims and counter claims. I kept being told how energy efficient the turbines were and that they could produce a huge proportion of the electricity required. Both claims “felt” false and yet finding the hard figures to try and confirm or deny my feelings was hard.

Having finished some other projects I decided to spend some time trying to get the information in an effort to answer my questions. As I believe data should be free, I decided to write a small website that could be easily updated with the latest information on a regular basis so that my brief period of work would yield continuing results. So far so good, but where to get the data?

My first thought was to find a list of all the windfarms in Scotland. I was sure I’d easily be able to find such a list and that would give me a useful starting point. Sadly I was mistaken. While I was able to find lists of windfarms, none had enough information or was complete. I had already found the Ofgem reports of output for the stations but I failed to find a list that could be easily referenced to that data. I eventually decided to use the Ofegem data for a couple of reasons

• it’s the official data that official statistics should be based on
• it covers all the wind farms that are claiming subsidies
• it’s updated on a frequent basis

After cobbling together some scripts in Python I was able to get and parse the Ofgem reports for certificates issued and station information. These I then used to populate a database model I built in Django and suddenly I had the basics of the information and website I wanted.

I also wanted to put the stations on a map, so using the address information within the Ofgem data was a good starting point, but the data has a lot of inconsistencies and errors, so wasn’t perfect. The Ofgem data also doesn’t have information on the number of turbines installed, something which I thought would be useful to know. By searching the web the information is available and so I’ve started filling it in wherever I can find it, together with location information.

There are places where I would like to make the Ofgem data I have copied more consistent, but I haven’t. The data from the Ofgem reports is simply used as is – if there is an obvious mistake in their data it will be replicated on the website. (The one caveat is the address information which I have tidied up where needed.) The data is simply parsed and stored in a database model that allows it to be displayed on the site. I have added additional information, but have documented where that data comes from. All sources are free, available to anyone and documented on the website.

While my primary interest was the output from wind farms, that output is also their income so I have started trying to find sensible figures to allow the website to display representative figures for the income each station generates. Much of this money is paid for by electricity users and so should be easily available, but sadly the various companies make finding the required information awkward. Presently I have found a source for sensible average prices for Renewable Obligation Certificates and show that information on the site. I’d also like to show how much income the sales of electricity earned but I haven’t found a sensible average wholesale electricity price. Nor have I managed to get information about the issuance of LEC for stations as it is considered confidential.

The site i