Executing multiple commands in SSH session against multiple machines

Hey,

it’s not uncommon for me to have to execute a quick command against a set of machines. Naturally, the easiest way of performing that against a single machine is using ssh:

readonly private_key='./key.rsa'
readonly command='echo test'
readonly ip='10.0.0.2'
readonly user='ubuntu'
readonly port='22'

echo "$command" | ssh \
        -i ${private_key} \
        -p ${port} \
        ${user}:${ip}

Having the command properly wrapped in terms of variables now it’s just a matter making it a method, looping through a list of machines and executing the command.

# Executes a given command via SSH
#       $1:             IP of the machine;
#       $2:             port of the machine to 
#                       connect to.
#       $USER:          user to use when executing
#                       the commands.
#       $SCRIPT:        global variable with the
#                       full multiline script to
#                       execute.
#       $PRIVATE_KEY:   global variable with the
#                       full path to a private
#                       key that is authorized by
#                       the machines.
execute_script_in_machine () {
        local ip=$1
        local port=$2

        echo "$SCRIPT" | ssh \
                -o "StrictHostKeyChecking=no" \
                -i "$PRIVATE_KEY" \
                -p $port \
                $USER@$ip
}

The list of machines might be an array that we just iterate through:

readonly TARGETS=(
        "127.0.0.1:2200"
        "127.0.0.1:2201"
        "127.0.0.1:2202"
)

As we provide the port argument separated from the ip argument we must split the strings during the iteration. Thankfully to bash we are covered by some internal operators. For instance:

#!/bin/bash

readonly TARGETS=(
        "firstIP:firstPORT"
        "secondIP:secondPORT"
)

# when iterating over an array we must 
# get each element from the array accessor
# `VARIABLE[@]` and not only `VARIABLE`.
main () {
        local ip
        local port

        for target in "${TARGETS[@]}"; do
                ip=${target%:*}
                port=${target#*:}

                printf "ip=$ip\tport=$port\n"
        done
}

main

leads to the following:

bash test.sh 
ip=firstIP      port=firstPORT
ip=secondIP     port=secondPORT

which essentially means that now we need to plumb these two pieces (the parsing of the target list and the command execution) and have the full working script:

#!/bin/bash

readonly USER="ubuntu"
readonly PRIVATE_KEY="./key.rsa"
readonly SCRIPT='echo -----STARTING;
  echo "WhoAmI? $(whoami)";
  echo -----DONE;
'
readonly TARGETS=(
  "127.0.0.1:2020"
  "127.0.0.1:2021"
  "127.0.0.1:2022"
)

main () {
        local ip
        local port

        for target in "${TARGETS[@]}"; do
                ip=${target%:*}
                port=${target#*:}
        done
}

execute_script_in_machine () {
        local ip=$1
        local port=$2

        echo "$SCRIPT" | ssh \
                -o StrictHostKeyChecking=no \
                -i "$PRIVATE_KEY" \
                -p $port \
                $USER@$ip
}

main 

That’s it!

I included an extra section after some feedback. It describes an alternative called mpssh that does the job of executing commands over SSH across many machines. Make sure you check it out!

Closing thoughts

This is just a quick script that might be handy if you want to inspect the state of something across many machines. I really don’t advise to make use of this as a way of provisioning machines as there are much better solutions out there (like ansible) which take a more declarative approach, are less distribution-specific and provides the great benefit of idempotence.

Having some bash skills are nice and can really save some time if need to get something done quickly using standard Unix tools. If you’re interested in knowing more about some bash/shell stuff, make sure you subscribe to the mailing list.

I don’t have many resources to share regarding this article but I always end up reading something here and there from the advanced bash-scripting guide on TLDP. I’d not stress too much on a guide as at least in my case most of what I know comes from solving stuff when needed. After a while, you get the gotchas and become good enough to get to the goal.

Have a good one!

finis

Update - mpssh as an alternative

Hey, a reader pointed out that there are some other very lightweight solutions that do the job very well. The alternative I liked the most is mpssh.

If you’re a MacOS user and uses brew: brew install mpssh. Under Linux, clone the repository ndenev/mpssh and then run make from the root (assuming you have at least gcc and make).

SYNOPSIS
        mpssh \
                [-besvV] \
                [-o directory] \
                [-u username] \
                [-f hosts] \
                [-p procs] \
                <command>

   The mpssh utility executes multiple parallel ssh binary 
   instances in order to connect to a list of hosts (specified 
   in the hosts file) and execute the given <command> on each 
   of them.

So, first thing to do is create a hosts file. There you can specify in each line the ip that you want to connect to. For instance:

echo "10.0.0.1
10.0.0.2
10.0.0.3"  > /tmp/hosts

Next step is configuring ~/.ssh/config: mpssh seems to now have an easy way of specifying a private key to use so you have to specify an IdentityFile in the ~/.ssh/config file. For instance:

Host *
    IdentityFile /tmp/key.rsa

With that we can already execute a command across those machines:

mpssh --user ubuntu --nokeychk --file /tmp/hosts 'echo "test"'

MPSSH - Mass Parallel Ssh Ver.1.3.3
(c)2005-2013 Nikolay Denev <ndenev@gmail.com>

  [*] read (3) hosts from the list
  [*] executing "echo "test"" as user "ubuntu"
  [*] strict host key check disabled
  [*] spawning 3 parallel ssh sessions

10.0.0.1 -> test
10.0.0.2 -> test
10.0.0.3 -> test

  Done. 3 hosts processed.

Aside from just executing the commands in all of them, it can have a ratelimit in terms of parallelization (e.g, given a list of 100 machines, execute in parallel 3 at a time). It’s also possible to output the result of the execution of each machine in a separate file. Pretty good!