Working with lists of items in the Shell

Another useful idiom for command-line junkies to learn is

list-of-something | do-something-with-each-item

This can be expressed in a number of different ways. My go-to way is the following:

list-of-something | while read f;do do-something-with-each-item $f;done;

The real beauty is this can be built up incrementally, testing out the commands until one has got it right.

For example, to confirm the syntax of a the do-something-with-each-item, I prefix it with an echo to see what command is going to be run i.e.

list-of-something | while read f;do echo do-something-with-each-item $f;done;

If the list-of-something is very large or to confirm that I have the right subset of the list, I add some filters i.e.

list-of-something | grep something | head | while read f;do echo do-something-with-each-item $f;done;

The benefit of this incremental/iterative way of building up a command is that it is quick – up-arrow, edit the command and enter to try again.

The keystrokes to master and memorise are

while read f;do $f;done;

These are then embellished by adding a list-of-something | before the while, and an echo do-something-with-item before the $f.

Frequency counts in a Shell

For command-line junkies, a useful idiom to learn is

  list-of-something | sort | uniq -c | sort -nr

which will give a frequency count of list-of-something sorted in descending order.

e.g. to show the frequency count of the 4-starting-letters of words

  cat /usr/share/dict/words |\
     cut -c 1-4 |\
     sort | uniq -c | sort -nr |\

which will show something like

  2043 over
  1334 unde
  1323 inte
  1078 anti
  1000 supe
   951 semi
   731 unco
   700 poly
   648 para
   618 peri

The classic story about this relates to Doug McIlroy and Donald Knuth More Shell, Less Egg

Vim – Capturing Output from Commands

One of the most useful things about Vim is its command-line mode. Using comamnd-line mode one can run external commands and capture the output and apply it to the buffer.

The external command is able to take text from the buffer, process it and replace the text.

Find checksum of lines of text

 :1,4 ! md5
 :% ! md5

Reformat JSON using jq

  :% ! jq .

One can also run an external command and have its output added to the current buffer.

Get list of files and insert into current buffer

  :. ! ls -latr

Capture output of a `curl` command and insert it into the current buffer

For example if one wants to insert the output from a curl command at the current line, one can do something like the following

  :. ! curl -s
  :. ! curl -s

For more information, see the Vim help :he filter and :he :.

Back Again in 2019…

Time to get this show on the road again.

The to-write list is brimming with ideas. And I have a process that should work. But it is a matter of finding the time and commitment to put pen to paper and see what happens.

Reading the following two posts got me motivated again.

And of course then is Brian Harmer’s “Wysiwygpurple's Blog” where he talks about putting your hands on the keyboard and doing something. If I could do the something similar with my photography, that would be unreal.

But let’s get technology on the way first…

Good Teaching

Over the years I have come to appreciate the difference a good teacher makes. A good teacher is someone whose teaching affects the way you think and one you remember for life.

Recently, I had to create a mashup of photos and video. Previously for this type of task I used iMovie 9.0.9 (or was it iMovie ’11). Anyway I fired up the latest version of iMovie and, ughh, the interface had changed once again and become super simple, hiding or removing some useful functionality. I was too hassled to try and ‘discover’ it.

So I went and bought big brother Final Cut Pro X. Mainly because I already had bought some books on special and had a premium subscription to

So I did the right thing and started working through the Final Cut Pro X 10.2 Essential Training with Ashley Kennedy . And this is the course that has inspired me to write about “Good Teaching”.

The thing about teaching a complex course is

  • the teacher has to have a thorough understanding of the material so that they can provide context, then drill down into the detail and then jump back out to an overview to provide context.

  • the course has be presented in a structured manner so that we can see over the overview and get to the actual detail quickly.

Not only does this course cover the details of Final Cut Pro X, but the way that Ashley does it is amazing. She works on two different video presentations, Chocolat and Letterpress. The primary aim of the course is to teach you how to use Final Cut Pro X. However the way she works on these two videos not only shows you how to use Final Cut Pro X, but also gives some insights in how to produce a video.

From coming up with a storyline, organising supporting material and then assembling the pieces together to provide final impact.

To cap it off, the actual video of the course is also fantastically done. If you want to get going quickly, Chapter 2: FCP X QuickStart provides this quick overview showing how one can get a video produced.

As a keyboard junkie, another hightlight for me was the extensive emphasis on keyboard shortcuts. Whenever, an action was done, the keyboard shortcut was mentioned and a flyout would show what key was being used.

The course is well structured and the transcript is also provided in the web version.

Vim – Running Commands on Matching Lines

Over the years I have used many editors. There was VM/CMS Xedit, Burroughs CANDE, ISPF/PDF editor, the IBM E, E2, E3 series of editors which influenced SlickEdit somewhat. There was a brief foray into PC editors including MultiEdit and the Borland C IDE. On OS X, I have used TextMate, SlickEdit, Sublime and now Vim.

One of the most amazing features in Vim is its command-line mode and bears similarity to that found in ISPF/PDF Editor, Xedit, SlickEdit and E series of editors. Instead of manipulating the content directly, one can issue commands to do the dirty work.

In particular the global command is really useful.

For example. given the poem below, assume that one wants to apply a change on all lines containing the word ‘was’, say uppercase the entire line

  Mary had a little lamb,
  His fleece was white as snow,
  And everywhere that Mary went,
  The lamb was sure to go.

  He followed her to school one day,
  Which was against the rule,
  It made the children laugh and play
  To see a lamb at school.

One could enter the following

 :g/\/ norm 0gU$

and end up with the following.

  Mary had a little lamb,
  And everywhere that Mary went,

  He followed her to school one day,
  It made the children laugh and play
  To see a lamb at school.

Basically this can be read as “For every line that contains the word was (g:/<was>/, execute a series of normal mode commans (norm) i.e. go to the beginning of the line (0) and then upper case to the end of the line (gU$)”.

Beyond Compare – It really Is…

Back in the day, I was a big fan of SlickEdit and have bemoaned the fact that it was not available on OS X.

One of its great features was the file and directory compare facility. Until last year or so, I thought it was the best.

Colleagues raved about this Windows app called Beyond Compare and how good it was. Windows only, so I didn’t pay much attention. Then there was a beta for OS X and I tried it out.

And boy does it rock.

  • Comparing directories or files, no problem at all.
  • Comparing log files and getting it to ignore differences such as log times.
  • Comparing zip, jar or war files to find differences without having to first unpack them.
  • Comparing tar files that have been gzipped and have different checksum due to gzip timestamp.

It has stopped my pining for SlickEdit.

The States of Git

One of the things I like about git, is the concept of working directory, index and repository. The index, or staging area, is really useful for gathering related changes before committing to the repostitory.

The GitGuys have a good description of the difference between these three.

Sometimes we will have changes in the working directory and/or the index. To compare differences between these different states:

Command Useful Alias Working
Index Repo (HEAD)
git diff gd
git diff --cached gdc
git diff HEAD gdh

If you want to diff a particular file, add -- filename to end of command e.g.

  git diff -- src/sql/update_20140125a.sql
  git diff --cached -- src/sql/update_20140125a.sql

To see contents of a file in a particular state

State Command
Working Directory cat filename
Index git show :filename
Repo (HEAD) git show HEAD:filename

And to get a good understanding of how git works, take a look at Git from the inside out