Zabbix: “TLS handshake fail” workaround: Enable Jabber (XMPP) notifications via script (Cent OS)

 

Once you’ve been around for a while dealing with a medium sized company, you start to crave a system to control your machines and not get to work one morning to a crashed server or some other worse catastrophe.

Bad things happen, computers crash, disks fail, power supplies burn out (believe me, I’ve suffered a couple of them zapping and then leaving that characteristic burnt dust smell) – there’s nothing we can do about it.

There are a lot of tools to monitor your hardware: The traditional Nagios, PandoraFMS… Some are free, some other offer support.

Some years ago, I had a Nagios Setup and what I missed the most was a web UI to configure my devices: You had (and still have) to go through text files configuring each device: Once it’s done, it’s done, you don’t have to touch it anmore: But the learning curve can be very steep.

This is why some days ago I went ahead and tried Zabbix: It’s simple, looks great and gets the things done and oh, guess what: It’s got a UI for adding hosts and templates. You still will have to use the console, but not as much.


 

Anyway, let’s get to the point of this post: I was configuring the alerts: The e-mail ones are pretty straightforward, but I wasn’t able to get the jabber notifications to work

tls handshake error

There was no way to get around the tls handshake error.  I tried on CentOS, Ubuntu and OpenSUSE: All of them had the same issue.

At least, on CentOS, I figured out (well, some guy over at the Zabbix forums did) that the culprit of this was a library called eksemel. Its github page has not been updated for the past 6 years. Latest version, 1.5 seems to support newest (and supported) cypher systems, but 1.4, which is the version included in CentOS, does not.

I went on and tried to manually update it, found an rpm and started trying, but got entangled on a hell of dependencies. So, I had to find another way.

Note: this instructions are for CentOS. If you are using Ubuntu or other distro, you can do the same, but slightly changing the most distro-specific steps, such as repository addition, etc… also, bear in mind, some directory location may and WILL differ from this walkthrough.

Let’s get to it:

First, activate Software Collections (SCL)

sudo yum install centos-release-SCL

Once enabled, you’ll be able to install this piece of software that will let you send Jabber messages from the command line: sendxmpp

sudo yum install sendxmpp

This will also pull a bunch of dependencies.

Now, let’s get a jabber account ready, if you don’t have it: you’ll need an account from where to send the messages from, and another account for yourself, to receive those messages, if you didn’t have it before.

I chose Dismail. There’s also Jabjab, and many others: Feel free to browse this feature matrix and choose whichever you like most (the greener, the better, I guess..) https://gultsch.de/compliance_ranked.html

Now, go to your zabbix server to the alertscripts folder:

cd /usr/lib/zabbix/alertscripts

And create the script:

#!/bin/sh
echo “$3” | /usr/bin/sendxmpp -u <username> -j <domain> -p <password> -s “$2” “$1” -t

Where <username>, type in your jabber user name (without the @xxxx)

Where <domain>, type in the domain (what’s after the @)

Where <password>, well, your password.

You can put all this sensitive data in a text file somewhere secure and reference it, avoiding having plain text passwords in a script.

Once the script is created, save it, and give it execution permissions:

chmod +x sendxmpp.sh

You can go now to your web UI.

Under Administration -> Media types, create a new Media type

zabbix-xmpp-script-config

Name it as you wish- It’s important you select the type correctly “Script” and give the proper parameters to the script and in that particular order:

{ALERT.SENDTO}

{ALERT.SUBJECT}

{ALERT.MESSAGE}

And enable it.

 

As the final step, you will have to go over to Administration -> Users, edit your user and go to the “Media” tab.

There, Add a new media for the jabber script you created.

zabbix-xmpp-script-config-media

Select as “Type” the name you gave the media type on the previous screen, and set the destination address. Save, and you’re done! No more tls errors!

 

I’ll have to point out that even the Ubuntu appliance you download from the zabbix download page has the same tls handshake error issue.

I hope they implement a different approach some day so can use the integrated system that comes with the suite.

Thanks to the zabbix team, the people over at their forums and their subreddit for helping.

Advertisements

Running Owncloud on Docker

dockerManaging multiple appliances on the same server allways had its issues: Dependencies and updates are a risk you have to get over with in order to keep an up to date and secure service.

Docker is a technology that has been giving a lot to talk lately.

Imagine a world where you don’t have to worry about dependencies or configurations gone wrong: Docker is here for this purpose.

It’s not virtualization like we are used to with Virtualbox, VMWare or such. This is a technology involving completely isolated containers with different purposes. Completely abstracted from the underlying OS, so, in case an appliance becomes unstable, the other ones running ont he same server stay unharmed.

With this in mind, you can have a docker daemon running on a Ubuntu Server or CentOS, for example and be running multiple sub-systems (called containers) at the same time.

They offer a public repository of appliances on their web page . This is a public repository contributed by users from all around the world and organizations: Anyone can pack a new appliance and submit it to docker hub so anyone else can pull it on their machine and run it in minutes.

This is how you can get the simplest owncloud installation you can imagine.

First, you should have docker daemon running on your machine.

On Arch LINUX, follow their wiki: https://wiki.archlinux.org/index.php/Docker

For Ubuntu, they have a dedicated section on how to configure it: https://docs.docker.com/engine/installation/linux/ubuntulinux/

Once it’s working, you only have to pull the image:

docker pull owncloud

Optionally, create a persistent storage volume, so the data stays out of the docker image, and on your real OS’s filesystem:

mkdir /owncloudData

And finally, run the image on a container:

docker run –restart=always -d -p 80:80 -v /owncloudData:/var/www/html owncloud

Then, just point your browser to the machine where the docker daemon is running and you’ll find the initial screen where you can create your user and, optionally, select the database backend you want to use.

By default, it uses sqlite, which is a good choice for portability and if you want to keep a simple installation to use with few users, but you can modify this.

It couldn’t be simpler.

Socks5 tunnel over SSH on windows

A couple of days ago, I was trying to reach a certain page over the internet. The strange thing was that sometimes, the page seemed to be down so I let it rest for a couple of hours: they could be under maintenance and dropping connections. But after a couple of hours, the page was facing the same issues. So, I tried to connect over my phone’s 4g connection: To my surprise, the page was loading perfectly, so I tried tracing where the connection was being dropped:

On windows:

tracert examplewebpage.com

On Linux:

tracepath examplewebpage.com

After 9 hops, the connection was lost, there was no response from some IP already on the destination page’s sub-network, shile on my phone, I was still getting a perfect user eperience on the page. After switching to google’s dns and having the same set of issues, I figured out for some reason, the remote machine was dropping my ISP¡s connections, most likely because of some issue, so I decided to try tunneling my connection via my home connection, on a different ISP.

In order to do so, you must have a working SSH server with it’s ports properly mapped so it can be reached from outside (while SSH uses port 22 by default, I recommend using something different, like 2207 and map it to your internal 22 port: It’s a very tempting port for hackers).

Then, you just have to stablish the tunnel, on linux:

Let’s say your configuration goes like this:

  • Your home’s address (You can get a cheap solution for this with no-ip.com): myhomeaddress.ddns.com
  • Your ssh server user name to log in: user
  • Your ssh port: 2207
  • The local port you are going to use for socks connection: 8080

ssh -D 8080 -C -p 2207 user@myhomeaddress.ddns.com

Log in and keep the window open.

 

On windows, you will have to use a 3rd party software in order to make the tunnel.

I use kitty, a putty fork that i prefer for having extended features and a better sounding name in spanish rather than the original version, but you can use either one of them.

Then, configure it like you would usually do in order to stablish a ssh connection: Input your server’s address and port:

Then, go to tunnel settings and change these:

 

Once your connection is set you just have to redirect the connection from your browser. I used firefox because it’s easier to configure:

And you’re just done. Go to http://www.whatismyip.com to see your new IP.

Taking advantage of btrfs snapshots

One of the killer features of BTRFS are the snapshots.

In a nutshell, imagine you just installed your system and want to have a “copy” of that current state in the event of having a system crash in the future, so you can go back to that “stable” moment, avoiding all the fuzz of reinstalling and potentially losing data:

btrfs snapshot <source> <destination>

This will create that “copy” of the <source> btrfs subvolume on the desired <destination>.

You can create a snapshot inside of your subvolume: BTRFS is clever enough to know this and not create recursive snapshots.

This is useful for going back to previous states or just to recover data you have previously deleted (un)intentionally.

This feature is pretty neat, but it is quite useless when you can’t have it done automatically.

Sure, you can create a cron job to run the snapshot command every now and then, but for this, the guys over at SUSE  have already thought of it and created a very handy tool called snapper

To start using snapper (after pulling it from the repos or aur), you have to create configurations. Let’s start by creating a configuration for our root subvolume:

snapper -c root create-config /

However, snapper will not work if you don’t have a daemon, like cronie running, so install (if needed) and enable it

systemctl enable cronie

This will create, by default one snapshot every hour. You can list the current snapshots with the command:

snapper -c root list

And see the snapshots you have available

You can delete snapshots using

snapper -c root delete (id)

where ID, you can either enter one ID or a range of snapshots like “3-20” and snapper will delete all the snapshots in that range: Don’t worry if there’s no snapshot with an id 10 in this case: Snapper will skip unexisting snapshots and won’t fail.

 

Pretty nice, right?

Now, let’s take it up a notch: Let’s say your system fails,a nd you want to revert to a previous snapshot: Snapper has a build-in “snapper rollback” feature tht i didn’t manage to make work: Besides, I prefer when I do this kind of stuff manually: Helps you understand what is really going on 🙂

Just boot a live system and mount the root btrfs filesystem

mount /dev/sda1 /mnt

Now, you will have all your subvolumes under /mnt

Let’s say you created the btrfs filesystem at /dev/sda1 and created two different subvolumes: /mnt/root and /mnt/home

Snapper would have created snapshots under /mnt/root/.snapshots/#/snapshots/, being /mnt/root/.snapshots another subvolume.

You should first move this subvolume out of the root subvolume (you can have it on a separate subvolume at the same level of /mnt/root and /mnt/home, but let’s leave that for later on)

mv /mnt/root/.snapshots /mnt

Then, rename the “broken” root subvolume

mv /mnt/root /mnt/root.broken

Then, find a snapshot that you know was still working under /mnt/.snapshots/#/ and move it back to the top subvolume:

mv /mnt/root/.snapshots/119/snapshot /mnt/root

…and you’re done: unmount and reboot and you will be back at your working system.

Access owncloud via webdav from kde

Owncloud is the best solution if you want to have your own private cloud. I will probably write a step by step manual in the future about how to do this on arch.

One of the perks of being open software is that they try to keep it as compatible as possible: Instead of just having their own client, they also give you the option to connect via WebDav protocol.

This is the best choice when you want to open it on your file browser (i.e. kde’s dolphin)

ll you have to do is enter the url like so on the address bar:

webdavs://your-owncloud.url/remote.php/webdav

…and you will be prompted with a login dialogue.

Easy, ¿isn’t it?

Let’s go a step further: Let’s say you want to mount the dav folder on your FS, all you have to do is (prior to have the package davfs2 on arch installed) is:

mount -t davfs https://your-owncloud.url/remote.php/webdav /path/to/mountpoint

and/or add an entry to your /etc/fstab:

https://your-owncloud.url/remote.php/webdav /path/to/mountpoint davfs user,noauto,uid=username,file_mode=600,dir_mode=700 0 1

More info at the Arch Wiki Davfs page

 

Importing not-exported ZFS pool to new system

So, recently, I replaced my system’s HDD. I could have just copied the system over to the new disk, but that Arch installation has been with me for years without major issues and I wanted to start fresh, also because I was leaving behind the BIOS system and diving into UEFI.

To have a bit of background, I have 4 hard drives: One system HD with a windows partition and an arch linux partition and the other three are 3x2TB HDD forming a raidz (raid5) 4Tb setup where I can afford to lose one disk and still keep my data while I run to the nearest components shop and get me a new disk to replace the failed one.

ZFS keeps its information on a cache file at /etc/zfs in order to be able to re-assemble the volumes at boot time, but I just forgot to export the pool so it gets “released” for saying it somehow, from the old system, so I can import it on the new one.

The process for importing a non-exported pool is fairly easy:

zpool import -d /dev/disk/by-id -aN -f

This will scan the disks on your system and import the pools it can find. Verify the results with:

zpool status

Pretty simple, isn’t it? Well, not quite: When you install arch:

pacstrap /mnt base

“base” includes mdadm,which is a clever system that scans your partitions automatically at boot time and assembles any array it can detect, regardless of the configuration set in /etc/mdadm.conf, pulling information from the partitions each disk have: If there are partitions marked as array type, mdadm will understand there’s an array, not minding if the disk also has a, let’s say ZFS partition.

This would have not been a problem if my disks haven’t been a software array ever, but they did, and when I made the change to zfs I did not clear the partition information properly.

Before you are going to use mdadm or after you stop using it, you should issue the command

mdadm –zero-superblock /dev/sda

on every member disk of your array: This clears every partition information from the disk, avoiding future problems. Be careful, though: You can’t use this command after you put new data on your disk or you will lose all that data

There’s another way (also destructive, sorry) to do this. A more generic one with:

wipefs -a /dev/sdX

This will clear every (-a) data structure information on your disk: It will wipe the partition information and leave the disk partitionless.

Going back to my issues with ZFS: After re-importing the pools and the system generating the needed cache files for zfs import at boot time, my zfs pool was not surviving the reboot and instead, i found myself with a “/dev/mdxxx” device instead.

My first reaction after finding out was to remove the mdadm package completely.

In the future, I plan to wipe every disk of my array one by one, perhaps to avoid this issue the next time I might reinstall the system, perhaps pulling a device out of the array, issuing a wipefs command on it and then, re-assigning it to the array and performing a resilvering operation: I’m only wondering if the stress of 3 resilvering operations is worth the effort.

Backup your linux system to a compressed file

It’s well known how LINUX, compared to Windows, manages pretty well hardware migration. It involves 3rd party (usually paid) apps or some time editing registry settings and a bit of luck to get it right. However, on any *nix environment, all you have to do is create the new partitions on the target system, pack your files on the old (or just directly transfer them), deploy them on the new, edit /etc/fstab, and the bootloader reference to the disk you want to boot from and perhaps a

mkinitcpio -p linux

to regenerate the boot images and you’re done!

This is a little command I use to make a backup of my whole system from time to time in case something goes wrong:

tar cvpzf /home/backup-$(date +%Y-%m-%d_%H%M%S).tgz –exclude=/proc –exclude=/lost+found –exclude=/mnt –exclude=/var/cache/pacman –exclude=/sys –exclude=/home –exclude=/.snapshots /

The output would be a file under /home with the date on it, so you ca keep old versions easily and basically, I skip every volatile and external directory from the backup. Finally, I want to keep the btrfs snapshots out of the picture as well, hence the last –exclude option.

Fix deluge-web error with default certs when connecting via https

I use deluge on a headless server as a seedbox. The way to connect is via http, using the deluge-web service that comes with the deluge standard package in arch linux. As usual with every http connection, you should enable https whenever possible, just to get that basic security level, even though you (most of the times) will use self-signed certificates that will give you the typical security warnings when connecting.

For reasons I still don’t know (perhaps because the certificates I was using were quite old already), a setup I have been using for years now, suddenly started giving errors upon connecting that didn’t give me the choice to accept the security risks and connect regardless, so I had to re-generate the certificates.

It’s a pretty easy task: you just have to navigate to the deluge configuration folder where the certs are stored at:

/srv/deluge/.config/deluge/ssl

And regenerate new certificates like so:

openssl req -new -x509 -nodes -out deluge.cert.pem -keyout deluge.key.pem

And then go back to the deluge web UI settings and reference them on the Preferences -> Interface tab.

Hope it helps, since it took me a couple of hours to figure it out!

Migrate BTRFS setup to a new (bigger or equal) disk: The dirty way

Imagine you are in the situation where your btrfs disk is getting filled up, or is just about to fail or you just want to install a bigger HD, or even better: You are migrating to a SSD disk setup.

Migrating partitions with traditional filesystems such as ext4, or FAT32 is easy, since you just have to format a partition on the new disk with the same type and just

cp -a /mnt/source /mnt/destination

Being /mnt/source your mountpoint for the old disk partition and /mnt/destination the new one.

But with btrfs, the thing gets a little more complicated, since the contents of those partitions are not just plain and simple files and folders: You have subvolumes, snapshots, etc… and they behave in a special way or have specific commands to manage them.

There is a solution, a fast solution, in case you wan to migrate without all the fuss of replicating the subvolumes, use btrfs send and receive, etc….

You can just add the new disk’s partition to the btrfs pool, getting a raid1 data structure, then reduce the data structure to raid0 and finally, removing the old disk from the array.

First, create the new btrfs partition on the new disk, I’m going to assume /dev/sda2 is the btrfs partition on the old disk, while /dev/sdb2 is the btrfs partition on the new one.

First, we create the btrfs partition:

mkfs.btrfs /dev/sdb2

Then, we add the partition to our array: Let’s suppose you mounted /dev/sda2 on /mnt, we would add the new partition like so:

btrfs device add /dev/sdb2 /mnt

You will be warned by a message saying all the data on /dev/sdb2 will be destroyed, and this is because everything on this partition, including the UUID, will be identical to what you had on the original one at /dev/sda2

Afterwards, we have to convert the data structure to raid0:

btrfs balance start -dconvert=raid0 -mconvert=raid0 -sconvert=raid0 /mnt -f

We will probably get a warning if we don’t specify the “-f” option: BTRFS does not like to have single system data structures if you have more than one disk, but it’ll be fine.

NOTE: These operations might take a long time to complete, depending on the amount of data on your brtfs partitions(s)

Finally, once the balance concludes, you can remove the “old” disk’s partition:

btrfs device delete /dev/sda2 /mnt

Btrfs will take a really long time on this final step, since it has to move every chunk of data remaining on /dev/sda2 over to /dev/sdb2, but once you’re done, you can safely reboot and remove the old disk.

I must say again, this is a “dirty” and not so elegant method, but it does the job and is the best way in case you are in a rush or the subvolume/snapshot structure is way too complicated. Keep in mind the new partition must be same size or bigger than the old one for this to work without problems.

Enjoy!