Category Archives: Sysadmin

Systems Administration

airtel.in DNS broken

I wish I could say I don’t care about airtel.in, but because they are my ISP, I have to care :(.

The genius people at airtel.in CNAME-d the root record of their zone. 🙁

λ drill -t ns airtel.in @aaadel.mantraonline.com
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 55824
;; flags: qr aa rd ra ; QUERY: 1, ANSWER: 3, AUTHORITY: 1, ADDITIONAL: 0 
;; QUESTION SECTION:
;; airtel.in.   IN      NS

;; ANSWER SECTION:
airtel.in.      28800   IN      CNAME   www.airtel.in.
www.airtel.in.  7200    IN      CNAME   airtel.in.edgekey.net.
airtel.in.edgekey.net.  19572   IN      CNAME   e8827.a.akamaiedge.net.

;; AUTHORITY SECTION:
a.akamaiedge.net.       572     IN      SOA     n0a.akamaiedge.net. hostmaster.akamai.com. 1481797738 1000 1000 1000 1800

;; ADDITIONAL SECTION:

;; Query time: 46 msec
;; SERVER: 202.56.230.6
;; WHEN: Thu Dec 15 16:06:06 2016
;; MSG SIZE  rcvd: 174

Some less-strict resolvers seems to work with them, but others don’t. I mailed them (the WHOIS contact address) a month or two ago, but seems like it went to /dev/null. Today, when I went to access their looking glass, it again bit me. For now, I added lg.airtel.in to my hosts(5) file, where it will live, until it breaks.

Listing FreeBSD packages in order of dependency

So, I upgraded GHC (via pkg) few minutes ago, and didn’t upgrade any of hs- packages, which means now their registration information is gone from GHC’s cache. So as a workaround I just decided to invoke their corresponding register.sh which would result in their package being registered:

λ pkg info |awk '/^hs-/ { print $1; }' |xargs -n1 pkg info -l |fgrep register.sh |xargs -n1 sudo /bin/sh

This was ideal command except it didn’t work because package list (pkg info) was not topologically sorted. And RTFM-ing pkg info didn’t result in any option which could output topologically sorted listing. But thanks to pkg info -d which generates the list of dependencies, I hacked this, which worked:

λ pkg info |awk '/^hs-/ { print $1; }' |xargs -n1 pkg info -d |awk '/^[[:space:]]+hs-/ { print $1; }'  |sort |uniq -c |sort -k1 -nr |awk '{ print $2; }' |sed -re 's/-[[:digit:]._,]+$//g' |xargs -n1 pkg info -l |fgrep register.sh |xargs -n1 sudo /bin/sh

🙂

Duplicity

Recently I needed to do some backups on a host (not managed by me), and being somewhat paranoid, and lazy, I didn’t schedule backups for sometime. Recently while trying to be just paranoid, I decided it’s finally time to close this task pending for a while, so gave a shot to duplicity again. Its manpage is quite well written, and so is the software.

My backup provider offers SFTP space (courtesy: ProFTPD, I think). So to login to SFTP account, I created a SSH public key to be used by backup script, and converted it into RFC4716 format (ssh-keygen -e), and uploaded to remote host. For encrypting backups, I used GPG, so generated a GPG key locally, and copied the public key to the host which is to be backed up. Since decryption of backups is not needed, unless it’s to be restored therefore I don’t need to copy private GPG key on remote host.

Now with a duplicity command-lines similar to below, I was able to do full, and incremental backups respectively:

duplicity full --encrypt-key ${GPG_KEY} --log-file=${LOG_FILE} ${SOURCE_DIRECTORY_TO_BACKUP} scp://${BACKUP_USER}@${BACKUP_HOST}/${BACKUP_DIRECTORY}
duplicity incremental --encrypt-key ${GPG_KEY} --log-file=${LOG_FILE} ${SOURCE_DIRECTORY_TO_BACKUP} scp://${BACKUP_USER}@${BACKUP_HOST}/${BACKUP_DIRECTORY}

And to periodically clean old backups:

duplicity remove-all-but-n-full --force  --log-file=${LOG_FILE} ${NUMBER_OF_OLD_BACKUPS_TO_KEEP} scp://${BACKUP_USER}@${BACKUP_HOST}/${BACKUP_DIRECTORY}

That’s it. Put these things in a script in a cron job scheduled at appropriate times, I can now do encrypted backups. As an anonymous guy puts it:

It’s better to have dump-ed, and restored, than to never have dumped at all

Now go backup yourself!

Booting from LVM over mdraid

Recently while setting up a GNU/Linux host which had /boot over LVM over md RAID, I used following command-line to install GRUB2:

# grub-install --modules="biosdisk part_gpt raid chain part_msdos ext2 linux search help mdraid1x configfile ata normal" --boot-directory=/boot /dev/sda

Following is an excerpt from grub.cfg(5) menuentry block:

set root='(VG-Boot)'
linux /vmlinuz root=/dev/mapper/VG-Root ro  quiet
initrd  /initrd.gz

where VG is the name of volume group, and Boot, and Root are the names of logical volumes containing /boot, and / respectively.

HTH

Analysing tcpdump output on CLI

I recently needed to analyse a tcpdump output for an application layer protocol (mysql). Wireshark is there, but I wanted to be able to process the output with various text-process utilities. So I decided to give a shot to tshark(1), which is part of wireshark, and is like wireshark for terminals. With it, I was able to find get the list of SELECT queries from the captured stream, using following command-line:

% tshark  -r file.cap -d "tcp.port==$port,mysql" -Eheader=y -Tfields  -e ip.src -e tcp.stream -e mysql.query -R 'mysql.query contains "SELECT"'

Perfect!

Checkout specific patchset of FreeBSD sources

Following is a script to checkout FreeBSD sources (from its subversion repository) to specific patch-level as asked by a friend on IRC:

#!/bin/sh

if [ -z "$1" -o -z "$2" ]; then
        echo Usage: $0 release patchlevel
        echo e.g. To get 8.0-p5, $0 8.0 5

        exit 0
fi

RVER=$1
PVER=$2

SVNBASE=svn://svn.freebsd.org/base/releng
NEWVERS=sys/conf/newvers.sh
LIMIT=40
RETVAL=0

SVNLOG=$(mktemp -t $(basename $0))

if svn log -l $LIMIT $SVNBASE/$RVER/$NEWVERS >$SVNLOG; then
        BASEREV=$(awk 'BEGIN { sec=1; i=0; } /^r[[:digit:]]+/ { sec=0; revs[i++]=rev=$1; } /^(Security|Errata):/ { sec=1; } /^-+$/ { if(sec == 0) { print revs[i-'$PVER'-1]; exit 0; } }' <$SVNLOG)
        echo svn co -$BASEREV $SVNBASE/$RVER
else
        echo Error executing svn log
        RETVAL=$1
fi

rm -f $SVNLOG
exit $RETVAL

Obligatory screenshot:

chateau.d.if!abbe [~/bin] % co-freebsd-sources 7.0 4
svn co -r182740 svn://svn.freebsd.org/base/releng/7.0
chateau.d.if!abbe [~/bin] % svn cat -r 182740 svn://svn.freebsd.org/base/releng/7.0/sys/conf/newvers.sh|grep RELEASE-p
BRANCH="RELEASE-p4"
Download link: http://www.lostca.se/~abbe/scripts/co-freebsd-sources
SHA256 sum: c9958c4fd7cae5a5e9ff3fa84ba3af6adf38d9c8494b7913ad5b3a2f265a3f48

IRC proxy-ing

Sometime ago, I’ve received SSH access to a host which was quite restricted. Lots of executables are denied execution, as it was supposed to be only for learning, so no network access curl/wget/nc/socat/ssh -(L|D|R), or compilers, but then it’s got bash. Here is a tiny hack to connect to IRC from that host:

1. Create a file irc.sh on $host:

#!bash
exec 3<>/dev/tcp/irc.freenode.net/7070
cat <&3 &
cat >&3

2. Add following line to inetd/xinetd (or netcat):

sua     stream  tcp     nowait          $localuser    /usr/bin/ssh    ssh -i $privatekey -l $user $host bash irc.sh

3. Now connect your IRC client to localhost:sua (localhost:14001).

This is only a fun hack, and not something used to regularly circumvent access. 😛

Migration to OpenSMTPD

After being a FreeBSD port maintainer for mail/opensmtpd port for quite sometime, missing updates, and committing a buggy snapshot, I decided to start using OpenSMTPD myself. It was on my TODO list since a long time, with a pf-like configuration syntax, it was very tempting. Due to laziness, I was not switching to it, so yesterday I gave up (my laziness ofcourse :P), and following is my OpenSMTPD configuration:

listen on lo0
expire 3d

table aliases db:/usr/local/etc/mail/aliases.db
table secrets db:/usr/local/etc/mail/secrets.db
table personal { "user1@domain1.tld", "user2@domain2.tld" }

accept for local alias <aliases> deliver to mda "/usr/local/libexec/dovecot/deliver -o mail_location=mdbox:%{user.directory}/.mdbox -f %{sender}"

# my personal accounts
accept from local sender <personal> for any relay via smtp://localhost:8027

# my another personal account but goes through different MTA
accept from local sender user3@doman3.tld for any relay via smtp://localhost:8025

# my work account
accept from local sender work@workdomain.tld for any relay via tls+auth://work@localhost:8026 auth <secrets>

More details about my email setup in this mailing-list post. Back to being lazy again! 😉

OSX and SSL Certificate Bundle

Recently I’ve to use OS X as my computer (ofcourse temporarily ;)). I use postfix/fetchmail etc. to send/receive emails, and both of them are configured with SSL accounts. Unlike FreeBSD, or GNU/Linux distributions, OS X doesn’t ship with PEM certificate bundle which configuration of these programs expect to find. In order to get those programs going with certificates shipped with OS X, one can export all the certificates from OS X’s Keychain to a PEM file which they can then pass in the configuration.

I found this on internet somewhere I don’t remember. Posting here, in case someone else encounters similar problem.

HTH