Planet CentOS

I found an interesting trick in bash today that may help a few other folks as well. Occasionally I find that need to copy almost every file in a directory, except for one or two. Usually I'd copy everything and then delete the stragglers I didn't want from the destination directory. There had to be a better way, but as I said I'm lazy. Turns out I found the better way today. [jperrin@ferrata ~]$

I'm looking to put together a development cloud - a full featured one at that, on a budget. So here is what I'm thinking about compute nodes :

• AMD FX-6 6100 AM3+ cpu
• Motherboard to go with it
• 32 GB of ram
• 180 watt PSU
• 32GB SSD for local storage

I can get one of those 'sets' for just under £190.00 ; For Network, a HP ProCurve J9028A / their 1800-24G should do, and available cheaply off ebay. For Storage, I am thinking of repurposing my HP MicroServer with 4x500GB SATA's.

So four 'compute nodes' + switch + cables and disks for the MicroServer should clock in at £1,000.00 still need some sort of a case or rails ( intend to host this at home ).

What am I missing ? What might I be better off with ? £1k for 24 cores, 128gb of ram and 4 physical nodes seems like a good deal to me, but could I do better ?

- KB

I really don't want to turn this blog into an anti-gnome3 themed thing, but they seem to insist on terrible things. Being the type of person I am, if I find terrible things, I'm going to share terrible things. So in that spirit, here's your terrible thing: www.slideshare.net/juanjosanchezpenas/brightfuture-gnome This slide deck starts off like any other. A bit of backstory, a bit of

Off to another DC tomorrow morning, hoping to swap out some seriously old hardware ( ~ 7'ish years old ) with something slightly newer ( only 4 years old! )

While I've bounced around to various desktop environments, I have found that I always end up coming back to gnome. That is, until now. Gnome3 has already been widely regarded as a step in the wrong direction, however the developers appear to be largely ignorant of what the users want. The arrogance, and ignorance coming from the gnome community has finally pushed me to the breaking point. It's

There was a time when I spent lots of time in the Data Center. I didnt enjoy them very much then, too much noise, too many people telling me what to do and way too many unpaid overtime hours clocked up. The last of that was about 12 years ago. These days I goto the DC maybe three or four times in a year, and its mostly related to CentOS infrastructure or my own personal machines hosted around London. And unlike things 12 years ago, I quite enjoy my trips into the vault like colo rooms now.

Apart from a bunch of routine things that I need to sort out tomorrow, I'm hoping to bring online a hardware rng adapter, and a Tilera Server. If you havent seen this platform before, I recommend you do : specially if lots-of-cores is something you are keen on, or have a problem domain overlap with.

- KB

This guy has an absolutely fantastic set of cheat-sheets. If you're doing anything network related, these are definitely good to have on-hand.  packetlife.net/library/cheat-sheets/

We had an end user appear in the main #centos IRC channel the other day with a IPv6 problem.  That person had leased a VPS somewhere, and their provider had included and enabled IPv6, at least partially.  Something was wrong in the network fabric, so that while some IPv6 services worked, others did not; DNS results returned with AAAA record results; but then the VM hoster was not transiting port 80 TCP traffic.  Very curious, and frustrating to the end user who just wanted yum to work so they could install updates and packages on their instance

The culprit is the grafting in of IPv6 readiness in man 2 getaddrinfo.  This is the way of the future, so there is no fighting it on a long term basis, but tactically having a means to be IPv4 only is appealing for people just wanting to work in the older address space.  The TL; DR uptake of that man page is that in a properly functioning system, name resolution answers under IPv6 are preferred, and only if not available, does one fall back to the older IPv4.  But this places a premium on IPv6 actually working when present.  We've shipped a full native IPv6 setup for customers at PMMan for a couple of years ago, but I assure you that we had some head-scratching as we rolled it out, and found customers using tunnels from HE or SixXs were also leaking advertisements to other VM's.  We added rules to filter out the false traffic after a bit of tracing with TCPDUMP

I have blogged about it before when IPv6 link-local address resolution (the ^FE family) was confusing distcc under Debian a couple of years ago.  There are links in the CentOS wiki for approaches on disabling IPv6 traffic, which vary between C5 and C6

That last mentioned article has an outlink to a bugzilla ticket that offers food for thought.  It mentions in passing that one can direct a nameserver to NOT deliver IPv6 results with a fairly simple tweak

Another option is to add this to /etc/sysconfig/named:

OPTIONS="-4"

... so, ... it should be possible to set up a local cacheing nameserver on the localhost, configured to NOT return IVv6 content, and so workaround the issue.  This smells sort of 'hackish', but it would have the benefit of being a single method that should work in the general case and not be tied to any particular kernel version, or other variable

The CentOS.org infra is pretty well spread out, but we got caught out by the Sandy storm. Specially our mail and list services, which run from an Internap facility in New york.

Outage lasted from just after 03:00 hrs UTC October 30th 2012 to 07:12 hrs UTC October 30th 2012; and we dont seem to have lost any communcation. If you had bounce backs during that timeperiod, please do retry / resend the emails.

We dont have a machine that can be used as a hot-standby, but we do have fairly good backups. So should the machine go offline completely or suffer major damage, we would be able to bring services back on a different machine with no real loss of data.

- KB

You've probably read that Ansible uses by default paramiko for the SSH connections to the host(s) you want to manage. But since 0.5 (quite some ago now ...) Ansible can use plain openssh binary as a transport. Why ? simple reasons : you sometimes have complex scenario and you can for example declare a ProxyCommand in your ~/.ssh/config if you need to use a JumpHost to reach the real host you want to connect to. That's fine and I was using that for some of the hosts i have to managed (specifying -c ssh when calling ansible, but having switched to a bash alias containing that string and also -i /path/to/my/inventory for those hosts).

It's great but it can lead to strange results if you don't have a full look at what's happening in the background. Here is the situation I just had yesterday : one of the remote hosts is reachable, but not a standard port (aka tcp/22) so an entry in my ~/.ssh/config was containing both HostName (for the known FQDN of the host I had to point to, not the host i wanted to reach) and Port.

Host myremotehost
HostName my.public.name.or.the.one.from.the.bastion.with.iptables.rule
Port 2222

With such entry, I was able to just "ssh user@myremotehost" and was directly on the remote box. "ansible -c ssh  -m ping myremotehost" was happy, but in fact was not reaching the host I was thinking : running "ansible -c ssh -m setup myremotehost -vvv" showed me that ansible_fqdn (one of the ansible facts) wasn't the correct one but instead the host in front of that machine (the one declared with HostName in ~/.ssh/config). The verbose mode showed me that even if you specify the Port in your ~/.ssh/config, ansible will *always* use port 22 :

<myremotehost> EXEC ['ssh', '-tt', '-q', '-o', 'AddressFamily=inet', '-o', 'ControlMaster=auto', '-o', 'ControlPath=/tmp/ansible-ssh-%h-%p-%r', '-o', 'StrictHostKeyChecking=no', '-o', 'Port=22', '-o', 'User=root', 'myremotehost', 'mkdir -p /var/tmp/ansible-1351603527.81-16435744643257 && echo /var/tmp/ansible-1351603527.81-16435744643257']

Hmm, quickly resolved : a quick discussion with people hanging in the #ansible IRC channel (on irc.freenode.net) explained the issue to me : Port is *never* being looked at in your ~/.ssh/config, even when using -c ssh. Solution is to specify the port in your inventory file, as a variable for that host :

myremotehost ansible_ssh_port=9999

In the same vein, you can also use ansible_ssh_host , this one corresponding to the HostName of your  ~/.ssh/config.

Hope that it can save you time, if you encounter the same "issue" one day ...

While Chrome is an excellent browser, there isn't a quick and easy method to convince it to stop freaking out over self-signed or custom ssl certificates. For the majority of users this is probably a good thing, however for sys-admins or developer types there has to be a better way. This bash script takes the hassle out of importing certificates to make Chrome be quiet. The first option for the

I already know that i'll be criticized for this post, but i don't care . Strangely my last blog post (which is *very* old ...) was about a puppet dashboard, so why speaking about another tool ? Well, first i got a new job and some prerequisites have changed. I still like puppet (and I'd even want to be able to use puppet but that's another story ...) but I was faced to some constraints when being in front of a new project. For that specific project,  I had to configure a bunch of new Virtual Machines (RHEL6) coming as OVF files. Problem number one was that I can't alter or modify the base image so i can't push packages (from the distro or third-party repositories). Second issue is that I can't install nor have a daemon/agent running on those machines. I had a look at the different config tools available but they all require either a daemon to be started, or at least having extra packages to be installed on each managed node. (so not possible to have puppetd nor puppetrun or invoke puppet directly through ssh , as puppet can't even be installed, same for saltstack). That's why i decided to give Ansible a try. It was already on my "TO-test" list for a long time but it seems it was really fitting the bill for that specific project and constraints : using the 'already-in-place' ssh authorization, no packages to be installed on the managed nodes, and last-but-no-least, a learning curve that is really thin (compared to puppet and others, but that's my personal opinion/experience).

The other good thing with Ansible is that you can start very easily and then slowly add 'complexity' to your playbooks/tasks. I'm still using for example a flat inventory file, but already organized to reflect what we can do in the future (hostnames included in groups, themselves included in parents groups - aka nested groups). Same for the variables inheritance : at the group level and down to the host level, host variables overwriting those defined at the group level , etc ...)

The Yaml syntax is really easy to understand so you can have quickly your first playbook being played on a bunch of machines simultaneously (thanks to paramiko/parallel ssh). The number of modules is less than the puppet resources, but is quickly growing. I also just tested to tie the execution of ansible playbook with Jenkins so that people not having access to the ansible inventory/playbooks/tasks (stored in a vcs, subversion in my case) can use it from a gui.. More to come on Ansible in the future

Pulp gives you a very powerful admin cli utility in pulp-admin, however that power comes with a price. The command string can sometimes get a bit lengthy. Issuing iterative commands (bulk operations to all repositories for example) often require a bit of grep or awk piping in order to get things done. Fortunately pulp has implemented a very nice rest api that allows you create your own

TL;DR: Keep an eye on this project. It's going to save you EONS time because it handles the boring work for you. Testing and applying operating system updates is one of the more mundane tasks for sys-admins, and yet it can quickly become complex when you're dealing with vendor support or multiple versions of applications. Testing updates to determine if they're safe to push to production, while

Most people, me included, still consider IPv6 usage to be something not worth worrying about. This comes from the fact that most services are quite happy chugging along with just IPv4 access at both the service provider end and the service consumer end. However, what happens when an IPv6 option shows up ? Here are some numbers, many will find interesting, from the CentOS Mirrorlist service.


1st June 2012 : 0 hits on IPv6
11th June 2012 : We launched IPv6 access for mirrorlist.centos.org
+ 90 days ( midnight 13th Aug ):
--- 12,831,352 Hits to mirrorlist.centos.org were over ipv6
--- 711,014 unique IPv6's
--- with a usage average of 1.4 hits/sec
--- peak at 350 hits/sec

There are two, sometimes three, but always atleast two machines that respond to mirrorlist.centos.org requests over ipv6. Looking at the stats for one of these machines that has always been around, right since day one we get :

Total hits : 6,324,980
Of this 5,089,935 were CentOS-5 and 1,176,602 were CentOS-6 requests, the rest were invalid requests ( could be either CentOS-4 or for repos that dont exist )
Of the CentOS-5 hits, 61.24% of requests were to x86_64 repos
and from CentOS-6 hits, 27.05% of the reqests were to i386 repos



Interesting stuff, fairly large numbers.

Also worth noting here is that the numbers represent quite a skewed sample set, IPv6 is only really usable in some specific setups and in some specific environments / data centers. It does not represent an overall state of CentOS userbase, so please dont use these numbers to signify that.

- KB

We had a frantic call from a sometimes customer today.  Their self-administered WordPress-based website had a Trojan in it, and it was saturating their website traffic allocation.  "THE SITE WAS DOWN!!"  They had signed up at a CPanel mediated, shared hosting firm, and a plug-in they had installed turned out to contain a well-known trojan

We spent a couple of hours looking into it.  And then a couple hours looking into the WordPress security notification system.  Perhaps, I should say: non-notification system as to getting subscribed to a formal notification mailing list from the WordPress folks, proper

The WordPress model seems to be: treat your WordPress site as though it is a pet that needs daily feeding.  And to be 'put down' when you lose interest in it, move on, or forget about it -- Oops.  Log in daily as an administrator, and look for a notification
that you need to apply the 'latest and greatest' update.   Run the update process manually whenever it appears.  Oh yeah, did you remember to take a backup FIRST, and test that you can roll back to it if the 'update' breaks anything? Oops

This of course RULES OUT using a packaged approach to managing such sites, as the lag for stabilizing a new RPM package, accounting for potential database changes, and the like 'take too long'. Just unroll a tarball, and trust that it will not break any local customizations

I see fourteen open tabs in my browser panel still open, related to trying to track down a central and formal notification feed that I (or any person seeking to get 'push' notification) might subscribe containing only 'Security' notifications.  Weeding through the tabs, ...

• The 'Famous 5-Minute Install' for WordPress -- Nope, no useful outlink for hardening, nor to subscribe to notifications, beyond a pointer to a third-party Ubuntu appliance with an 'automatic security updates'.  That appliance's page has pointers to a tool to enable taking database backups, adding PHPMyAdmin, and Webmin.  Not good choices for a person caring about security
• Perhaps FAQ items tagged with: Security -- Nope, clearly incomplete, as for example a Google search turns up this third-party alert for version 3.3.2,  but the Release Notice does not get titled with: Security
• This bug (#10253) lingered for three years with a Security tag in their Trac issue tracker as to the current release series (3.4), and was amended ten days ago; But the latest release (for 3.4.2) was twenty days ago when this is written.  Should an update have been release?  Who Knows?
• Perhaps their FAQ Security -- Nope, no push notification link suggested there, but lots of clutter as to copyright infringement notification handling, and miscellaneous topics
• Perhaps watch the Releases News in an RSS reader - Oops, no sub-tag feed offered, and there has not been an "Important" Security release since December 2010, if one used that approach
• Run a Google search daily, and look for third-party commendary - Nope, although nuggets may be found, for it is not viable as: Not Authoritative, irregular and partial as to updates, and wading through search engine hit, or RSS feed clutter will kill your productivity

Clearly, one MUST configure the webserver to NOT permit off-site access to the credentials and configuration file: wp-config.php but I'll be darned if I can see instructions on the WordPress site, showing a novice administrator how to do this. In a shared hosting environment without 'root' level control, it is probably not even doable.  There is not hint of this rather elementary precaution on the official write-up concerning editting the file

A quick Google search for: turns up lots of vulnerable candidate installations, and a handy, dandy code fragment for parsing information out of potential victims so found, to automate take-overs. No criticism of the author of that code publishing his work; a knife can heal (as a scalpel), prepare dinner, or injure, depending on the intent of its holder

I see an official  recovery outline  suggestion, anyway

It is not clear if a cabal of Anonymous hackers, or simple network administration issues, caused the GoDaddy outage of Monday past. I guess it does not really matter

What really would have hurt is if the root domain server constellation had been compromised, to well and truly take down the internet. A Domain Registrar sends along updates to those root servers periodically, and GoDaddy's outage, from the extent of our involvement with them, simply impaired our ability to renew domains, and set new nameservers (NS records). As we had no urgent renewals pending, that is to say, not at all

We do not rely on GoDaddy for DNS services, and really, never have relied on them for production purposes. For PMman and for our ISP and COLO services, we run three geographically diverse nameservers for most of our purposes. We also run a few others for customers' needs (PTR records for a couple of datacenters we are in, testing, demonstration units)

The true 'masters' of our externally visible DNS servers are simply not accessible from the public internet. We push out updates to our public nameservers by cryptographically protected rdnc transactions. Those transactions are logged, and the information causing a given RDNC transaction are created by queries into a local database with a custom written LAMP control interface based on the FOSS tools that are in a stock CentOS install. Compared to manually editing zone files, checking variants in and out of a version control system, and so forth, this more readily provides us with scalability, traceability and auditability. Why, I caught a piece of lint in a zone file just last week, reading the overnight error report emails

We also retrieve the state of the generated zone files at the client public nameservers, and check them for consistency and coherency, essentially after each update, to prevent errors from propagating. ACLs, transaction logging and other checks provide more tracability, and we closed the mouse hole that that 'lint' crept in through in short order

As a result of the GoDaddy outage, a couple of our 'alumni' tech support folks who have moved on in their careers to other employment, gave us a call Tuesday, because they remembered how paranoid I am on making sure DNS is available. I appreciate the calls, and we've some new customers as a result

People have strong opinions about GoDaddy, sometimes for reasons of political correctness; I like them, by and large, because they provide a workmanlike product for a price that is hard to beat. They sure beat the heck out of the old Network Solutions rates. I have something like 500 domains that I administer and renew and most are there, although some are at other registrars for both historical and other reasons

And while Danica Patrick is not my cup of tea, she is not hard on the eyes, either

The question was asked in IRC today:

hello folks, is there any way to install packages from a list written by yum list installed? I've two CentOS 6.3 hosts and I like to get them with the same packages installed (also versions)

Here is a quick (and accurate) answer:

1. Take and test backups to taste
2. Run:
   

   rpm -qa --qf '%{name} \n' | sort > MANIFEST

   Note: that is a backslash n -- the html markup makes it hard to see the distinction
3. Then copy that MANIFEST to the second unit and run:
   

   yum -y install `cat MANIFEST`

   Note: and here, backtick around the cat to get a sub-shell
4. Finish by running:
   

   yum -y update

   on each unit

For extra credit, re-run the MANIFEST creator on each unit, and use diff to find any variances

Not a full posting, but this definitely struck me as worth linking to and/or resposting. blogs.gnome.org/otte/2012/07/27/staring-into-the-abyss/

July 9th, 2012 - Today the CentOS Project has released CentOS-6.3.  This release came 18 days after the upstream release of EL 6.3.  The major issues that we had with getting this release out were getting the i386 distribution to boot properly and adjusting the content of DVD-1 and DVD-2 of each architecture to allow for the most common install groups to come from DVD1 and not require DVD2.

Some of the major changes for this release include a move from OpenOffice 3.2.1 to LibreOffice 3.4.5 and the addition of Virtual to Virtual and Physical to Virtual (virt-v2v and virt-p2v) server migration tools to KVM Virtual Machines.  Please see the CentOS-6.3 release notes for more information on these and other features.  The Open H Website has a very good article on what's new in EL6.

We were pretty much ready to release the distribution by Friday, July 6th, but we had to then copy our trees to the several dozen internal mirror servers in the CentOS Project's infrastructure and then open those mirrors up to the more than 500 external mirrors that serve CentOS in more than 75 countries world wide.  While the CentOS Project is doing a point release, you can see what is happen by visiting our QA Development website and looking at the blog entries on the dashboard,

We hope you enjoy CentOS-6.3 !

The ELRepo Project has DRBD packages for CentOS-5 and CentOS-6, named drbd83-utils or drbd84-utils.  The CentOS Project does not want to maintain extra packages that exist in other places unless we need to change them ... so we are not going to create DRBD packages for CentOS-6.

Since CentOS-4 is being EOL'ed in less than a month, we are also not going to publish updates for the DRBD in CentOS-4.

This leaves the DRBD for CentOS-5 that are part of CentOS Extras.  Since these have been released for CentOS-5, we will continue to maintain the DRBD version 8.3.x  tree (drbd83) in CentOS Extras.

A new version of DRBD 8.3 (drbd83-8.3.12) has been released to the testing repository for CentOS-5.  You can see the details here:

DRBD 8.3.12 for CentOS-5

If you want to use DRBD 8.4.x for CentOS-5, rather than releasing it separately, the CentOS Project recommends that you use drbd84-utils from ELRepo (linked above).

For users who want to use the drbd83-8.3.12 version ... please test the version that is currently in CentOS Testing and provide feedback.  With enough feedback I will move the packages from testing to CentOS Extras.

NOTE:  These packages have now been moved to CentOS Extras and can be installed normally with yum.

There are some very nice feature improvements in gnome3, but if your used to the older gnome releases, odds are pretty good that you hate gnome3.  Don't let that stop you from using it though. With a few minor tweaks, you can have the best of both worlds.     To start, you'll need a few additional packages that you probably didn't install from the beginning, so lets grab them now. [root@

I've never listed my mobile number anywhere on the internet, and as far as I remember I've only ever shared it with friends and family. On the other hand, I've had the same number for years and its possible that its 'leaked'; But I still find it quite odd that people around the world manage to get their hands on the number, with no real effort. And that means I get calls.

Calls from people in Argentina at 4am UK time, wanting to know when php-5.4 is going to be released into CentOS-5. Calls from people in the UK, at 8am wanting to know if the httpd update released last night had a fix for CVE-XXX. Calls from people in India at 10pm UK time wanting to find out if the sound card on the motherboard they bought a few hours back, is supported on CentOS. A disgruntled passenger trying to check-in to their flight the next day, and the system throwing up a 'Apache on CentOS' page.

Some are a bit more alarming. eg. a call from people at a Large Defence Contractor in the USA asking who their 'CentOS Technical Account Manager' was and if I knew what the SLA terms were. Or the time when I got a call from a hosting company's Data Center saying there was a fire in the DC and they wanted to know if their CentOS backups were intact.

It's not something new, I've had these calls over the years from maybe 2008 or so. At one point, when it was really hectic with almost 10 to 12 calls a week, in 2010 I was seriously considering changing my number. Just doing the 'ignore if the number isnt in the address book' wasent scaling for me. But I didnt, the process of changing my number with everyone I knew was too much hassle, so I started giving people an alternative number and mostly started ignoring the 'popular' old mobile number. The number of such calls has now drastically reduced. I get maybe 1 or 2 in a week and in many cases I answer them and have had the odd interesting conversation. But realistacally, I think the time has come to change that number.

What I will, however, do is offer up a Voip line : +44-207-0999389 ; This terminates at a phone that I have on my desk. And I will try to make sure its turned on whenever I am doing CentOS stuff, or am in 'Open Source' mode. Go ahead, use that number - give me a call and if I am around, would love to have a chat. But please stop calling me on my mobile.

btw, I have tried to find my own number and failed to do so - even entering parts of the numbers into the various search engines does not bring up my mobile number. So, I have no idea where all these people suceed in finding it ?

- KB

One of the issues we have had in the CentOS Project is that at point release time, we were lagging behind a bit on getting the releases out.  Recently, the CentOS Project has addressed this issue in 3 major ways:

1.  We created the CR repository where we can, if there are delays, push out multiple updates that work together while we take a care of problems with packages that don't build, if we have issues getting the ISOs to properly build, etc. during the point release process.  This gives us an avenue to release pieces of the point release without having to release the whole thing.  We have actually not had to use the CR repository on the last 2 release cycles (CentOS-5.8 and CentOS-6.2) as we got each