Script day – Shutting down multiple servers at once
A system administrator in my company recently approached me with a problem – how to shutdown multiple Linux servers at the same time from a central location. Apparently this is something that people in the MS-Windows world use all kinds of applications, like the Remote Shutdown Tool from Microsoft (though I don’t understand how they handle the authentication – this tools doesn’t seem to require any authentication so it appears that any person with network access can shutdown any computer).
Anyway, apparently searching the web for “Linux remote shutdown” yields no useful results (or so I’ve been told), but frankly – when you have standard UN*X tools at your fingertips, a remote shutdown tool is simply typing ssh root@server shutdown -r now at your local console. But still, for people who want a “tool” – read on.
If you want to control multiple servers, it does gets a bit more tricky – you don’t really want to start sshing to server1, start the shutdown, ssh to server2, start the shutdown, lather, rinse, repeat.
So lets suppose you have a file with a list of all your servers. I’m assuming its a text file that looks something like this:
server1 1.2.3.4 server2 1.2.3.5
So here is a little something that scans such a list, calls each server to shut it down and waits 30 seconds for the server shutdown to complete. The assumption here is that you have exchange keys with each server so you can login as root to each server without the need to type in a password. If this is not the case, this script will attempt to publish your DSA key to the remote server so it will work better next time 🙂 .
Here is the script, with some inline documentation. Just copy and past it into a text file and make it executable. The server file is expected to be in a file named lan-servers.txt but its easy to change this.
#!/bin/bash
# where the server list sits
LAN_LIST="$HOME/lan-servers.txt"
# Function to check if the server responds to ping
function testServer() {
if ping -c1 $ip>/dev/null 2>&1; then
return 0
else
return 1
fi
}
# get a list of IPs for the servers
function getServers() {
cat $LAN_LIST | egrep -v '^(#|\s*$)' | perl -ple 's/\s+/ /g;' | cut -d' ' -f 2
}
# go over all servers
for ip in $(getServers); do
# check if the server is alive
if ! testServer $ip; then # if its not then its already down
echo "Shutting down $ip: already down"
continue; # to next server
fi
# check if I can log in to the server with no password
if ! ssh -o "NumberOfPasswordPrompts 0" root@$ip '/bin/true' >/dev/null 2>&1; then
# if I can't, try to do a key exchange (this will require an interactive password prompt)
cat $HOME/.ssh/id_dsa.pub | ssh root@$ip 'mkdir -p .ssh; cat >> .ssh/authorized_keys'
# check if I can now log in to the server with no password
if ! ssh -o "NumberOfPasswordPrompts 0" root@$ip '/bin/true' >/dev/null 2>&1; then
# if I still can't then something is broken
echo "Failed to complete key exchange with $ip - skipping this server"
continue;
fi
fi
# start to shutdown the server
echo -n "Shutting down $(ssh root@$ip hostname)"
# halt the remote server
ssh root@$ip halt
count=0
# wait until its down or 30 seconds have passed
while testServer $ip; do
sleep 1; echo -n "."
count=$(( $count + 1 )) # count up to 30
if [ "$count" -gt 30 ]; then # if the server did not shutdown in 30 seconds, something is broken
echo -n "Server did not shutdown properly!"
break;
fi
done
echo ""
done
And thats it. Of course if you really want an application we can complicate it a lot more if we want – add a GUI, some command line switches, a web interface – what not. But the basic functionality is something that a sysadmin should be able to hack in a few minutes.
This story highlights one issue I have with MS-Windows: you need special apps to handle all kinds of simple administration tasks – for example my Brother uses a special program for batch renaming of files. This functionality is completely unneeded on Linux because your simple shell is so much more powerful and an advanced user1 can write much more complex renaming commands then what a GUI can offer, using simple bash commands.
The point is that some computer administration tasks are simple enough that you don’t need special tools – but what is considered “simple” is subjective also depending on your OS of choice. With MS-Windows simple stops at about the process of moving an entire directory tree. With Linux simple can be much much more interesting.
- the kind of user that needs batch renaming is advanced enough for my taste [↩]
Why not just use clusterssh ?
http://freshmeat.net/projects/clusterssh/
Or just use Konsole with its “send input to all windows” feature.
There are a few reasons:
* This script actually make sure that a server was shutdown by pinging it. It will also elegantly skip hosts that are already down.
* Its automated – of course it is limited to just doing a shutdown, but it does this automatically with a single command – no need to run a full application with UI and everything. It will do its job cleanly and report the results in a simple way.
* It does not require X – you can run it from a VT or from a putty console on windows or anything
* I like writing scripts 😉
Real Windows administrators learn to use the Windows shell, CMD.EXE, and VBScript for the CScript engine, and the ones who are more open to the world also use Cygwin tools. Plus, I have been unemployed for more than a year now, so things might have improved since last time I administered a Windows machine — I recall having heard of a new Windows shell, and some POSIX tools coming from Microsoft.
See my reply to Eran below about Powershell. Regarding scripting – sure, but its like saying “you don’t need to install applications to do stuff – you can write the applications yourself”.
The point of the article is that a decently competent user can achieve more without getting into “developing applications” because the scripting tools are so handy that you don’t even feel like your are writing a script: want to convert all the underscores in your files to dashes? Just run
for file in *; do mv “$file” “${file/_/-}”; done
right at your command line – its more complicated to think about then to write it!
But the reason I use Magic File Renamer is that I wanted something that will make all the pictures I have look like this and all the audiobooks look like that and all the music albums look another way. And I thought of writing something myself but then there’s this cool tool that does it for me, remembers what I want and is perfectly good at its job.
Microsoft came out with Powershell not long ago but I haven’t tested it myself.
Well, I have tested it. They have some good things, like strong syntax, but they messed some other things completely – like pipes as object streams? Come on!
Regardless: CMD.exe is really primitive (like an early 80s UNIX shell), while Powershell is an external software package (like cygwin) you have to go and install it. If you have ever used Bash seriously and then tried to use cygwin or Powershell you’d know that its not on the same level. The reason is that its not only the shell – its about the entire system: Linux comes with tons of tools and applications that are geared to work nicely as components in a shell environment, while with MS-Windows you don’t even have a normal pager.
Compared to what you can get on MS-Windows (including cygwin), Linux is like working in a state-of-the-art workshop with more tools then you knew existed (and 5 sizes of each) compared to working in your garage with your ill fitting work desk, where half the time you can’t find the exact tool that you need and have to kludge it and the rest of the time your tool is not the exactly the size or shape that you need and you still have to kludge it.
You should stick to linsux because you sound like an idiot when talking about Windows. I’m sure that’s because you are an idiot.