Hey,

This week I wanted to make the logs of a Docker build easier to read, and remembering how some tools prefix their outputs with a name, I thought that doing so would be a good way to go.

In such scenario, I had the following structure:

.
├── Dockerfile          # The Dockerfile that instruct the 
│                       # image building process
│
└── scripts             # directory full of scripts to be run
    ├── first.sh        # Ideally, each script would have its name
    ├── second.sh       # used as the prefix in the logs when they
    └── third.sh        # get executed.

For this example, let’s assume that each of these scripts output a dummy text.

cat ./scripts/{first,second,third}.sh
#!/bin/bash
echo "hello, first here"
#!/bin/bash
echo "hello, second here"
#!/bin/bash
echo "hello, third here"

Knowing how to run these three scripts would give a hint to how to proceed.

Here I had some options:

  • use find and `xargs together;
  • use find with its -exec argument;
  • using a for together with variable expansion.

Although all of them are capable of achieving the goal, I’d always first give a try to find piping to xargs - there are some caveats to running for loops over files that might break your script in ways you don’t expect (given that there’s a variable expansion going on, files named after command-line arguments could break your execution later).

# Search for all the files under the `./scripts`
# directory that end with `.sh`.
#
# Pipe the result of the search to `xargs`, which
# then executes the command supplied in the positional
# arguments replacing the replacement string supplied
# to `-I` (that is, when it finds `lol.sh`, it executes
# `bash lol.sh`.
find ./scripts -name "*.sh" \
        | xargs -I {} bash {}
hello, first here
hello, third here
hello, second here

However, given that we’ll need to make use of piping to have our outputs prefixed, that’d mean making xargs call an extra shell for that.

# If we want to pipe the result of `bash` to another
# command, then we need to use an extra shell to process
# the piping operator and set up the piping itself.
find ./scripts -name "*.sh" \
        | xargs -I {} /bin/sh -c 'bash {} | wc -c'
      18
      18
      19

So, in this case, using a for loop seems to be the way to go (knowing that none of our files there are going to break us):

for script in *.sh; do
        bash $script | wc -c
done
      18
      19
      18

Now, to solve the problem we introduced before (prefixing the output of the executions), we can leverage the fact that we can take the output produced by those executions and then mutate them accordingly such that the final result has a prefix.

Naturally, sed is a great candidate for mutating streams of strings.

# Create a stream of multiline text
printf 'haha\nhaha\nhaha\n'
haha
haha
haha

# Create a stream of multiline text and pipe
# it to `sed`.
#
# Using the `s/regular expression/replacement/`
# we can provide a regular expression that matches
# the very initial part of the string and then 
# replace that by a prefix.
printf 'haha\nhaha\nhaha\n' | sed 's/^/prefix: /'
prefix: haha
prefix: haha
prefix: haha

Putting all together, we can make use of script name in the place of prefix in the sed rule and achieve our goal:

# Iterate over all the script files under ./scripts.
# 
# Given that the expansion will result in names that
# include the `./scripts` prefix and the `.sh` suffix,
# we can make use of bash's prefix and suffix replacement
# syntax to get rid of that and end up with the pure name.
#
# Once clean names are got, then it's time to use in `sed`.
for script in ./scripts/*.sh; do
  name=${script#./scripts/}
  name=${name%.sh}

  bash $script | sed "s/^/[${name}] /"
done

[first] hello, first here
[second] hello, second here
[third] hello, third here

As an extra tip, if you happen to have your output being forwarded to a log collector and you see many lines (that should be separate) comming together, it might be the case that sed is performing some buffering. You can get rid of such behavior by executing sed with stdbuf.

That’s it!

Please let me know if you have any questions or if you spot anything wrong. I’m cirowrc on Twitter.

Have a good one!

finis