Moriarty Naps

When we talk about Command Line Interfaces (CLI), we’re referring to text-based commands for our computer.

Think of it like playing one of those 80’s text adventure games If you want to try it out yourself, the spirit of the 80’s lives on in corners of the web. where you have to enter “Move North” to explore. If instead you type “Prance Left”, nothing will happen because the computer only understands very, very specific inputs.

Which is a pain. But! for GIS work it can be a godsend.

It’s not a cure-all for all GIS woes, but you can save yourself a lot of future headaches by having them now learning the magic of Command Line Cartography

What we’re covering here

• What is a command line interface (CLI)?
• When is this a useful approach?
• An introduction to CLI navigation
• How to get setup to run GIS commands
• Some basic CLI carto commands via Mapshaper

Repeatability

A staple of GIS work is that almost every action either permanently changes the original shapefile, or you save another one. This adds up fast between projections, clipping, buffers, data joins, etc. If you realize you need to fix or update something from early on in the process, this can be a huge time drain redoing in QGIS or Arc.

CLI’s biggest advantage is that each step is written out and repeatable. You can make the needed tweaks to early work, re-run it and be on your merry way.

Documentation

Hand in hand with repeatability is that you’re incidentally creating perfect documentation for your GIS process. If you need to come back to a subject six months later you’ll know exactly what you did and how you did it.

Batch Processing

If you want to operate over 100 or so files we can iterate commands over a folder of files. So, if we have multiple shapefiles, we can easily execute GIS operations over each of them.

Mix & Match

Want to automate more than just GIS operations? Perfect. We can bring in additional CLI to also fetch data from websites, or text operators to tweak csv’s. We can also run scripts written in Javascript, python, or whatever have you. The world’s your oyster!

You become Neo

Not as important as the above things, but you will literally become Neo from the Matrix when you run a command that makes your computer screen flood with fast moving text.

To start yelling orders at your computer, you’ll need to open up it’s Terminal. If you’re on a Mac, you can do so by searching for an application called “Terminal”. If you’re on Windows, it’s called the “Command Prompt”.

Where Am I?

When you start up your terminal, you’ll see a guide-less void of a text box. First let’s figure out where we are:

Type pwd and hit enter. You’ll see the terminal reply with where it’s is currently looking at. Should be something along the lines of “/Users/yourUsernameHere”.

Just like when you navigate folders and files on your desktop, this terminal is open to a specific folder on your computer. Unlike your desktop, however, we don’t have any folder graphics to click on.

Entering ls will list off all the files & folders that are inside wherever your terminal is currently looking. Basically a text version of what you usually see what you look at folders in finder.

This will create a folder named newfolder. mkdir will create a folder with whatever name you enter after the initial command. Quite a few commands work this way – the first word evokes a command and the words that follow help define it.

This will only create that folder. It will not move you into it.

Unlike our previous commands, mkdir will not cause your terminal to reply with a message. It can be a bit hard to get used to at first, but many commands don’t automatically say “hey, I did the thing and it worked, hooray!”.

To confirm that the folder you just made exists, you can run the ls command again. You should see your newly created folder listed now. You did the thing and it worked, hooray!

Moving Left

Files on your computer are tucked away into folders, and the route to them is refereed to as the file’s ‘path’.

You’re already familiar with how file paths work from navigating website URLs. On your computer they work much the same way.

This will take you to your home directory, aka your computer’s home folder regardless of where you are.

This will move you into your newly created folder. Like mkdir, you can enter any text after cd and it will take you to that folder, if it exists.

This will take you to your current folders parent. So, if you’re currently at “home/code/disco-party” and run cd .. you’ll move to “home/code/”. Think of this as your ‘back’ command.

Let the Adventure Begin!

OK. Now that we can navigate around, we can get started.

Or well, sorry, one more step. Then we can get to the GIS.

All of the above commands from the previous section are built into modern operating systems. Commands to do GIS operations however, are not. We’ll need to install some libraries to help out with that work.

There are a lot of options QGIS is built with and upon many of em’! for GIS work but for the purposes of this tutorial we’ll rely on the workhorse library Mapshaper.

To get it installed, we’ll want to do the following:

1. Install Node.js
2. Type the command npm. If it returns a block of text starting with "Usage: npm <command>", we know we're good. If you get an error, copy and paste that error into Google. Someone else has likely had the same issue!
3. Run the command npm install -g mapshaper. This will install mapshaper on your computer globally so it can be accessed anywhere! Neat.
4. If you get an error message that talks about denied access or permissions, you may need to run it as sudo install -g mapshaper. Sudo is a prefix you can add to commands that tells your terminal you're an administrator of the computer, so it'll ask for your admin password. In general, only run sudo if you need to. No other commands in this tutorial should require it.

Finally, you need a geographic file to work with. You can use anything you’d like for the following examples, but I’ll leave one here for consistencies sake.

1. Download this file. (4.9 MB). This is Natural Earth Data's 10m country geography, renamed 'countries.shp' to save us time typing out the file name. Also included is a csv of countries that red panda's call home.
2. Locate your home folder (the one cd navigates to by default) and create a folder there called Carto.
3. Put the contents in that new Carto folder. I'd recommend just doing this via finder. This will just help us stay on the same page for where our working files are located.
4. Finally, cd via your terminal into your newly created Carto folder.

Let’s go!

Change file formats

First, let’s change the file format. Shapefiles are fine, but given that they require four to six different files for one database things get messy fast. Let’s convert to a topojson Added bonus: topojson is a wonderfully small format, so you’ll save disk space working it. for the time being.

mapshaper countries.shp -o countries.json format=topojson

All commands will roughly follow this form: The first word mapshaper tells our Terminal to use the mapshaper library of commands. Next we type in the name of the file we want to operate on. -o is shorthand for “output”, which we follow by stating what the output file should be called. Finally, we tell it what format we want it to kick out as.

If you type ls now, you should see the new file created! Neat, eh?

Mapshaper can translate between the following formats:

• shapefile, geojson, topojson, json, dbf, csv, tsv, svg

Projections

Next let’s project our countries. With mapshaper -projections you can see a full list of projections Mapshaper does by default. Let’s do a quick projection to Robinson:

mapshaper countries.json -proj robin -o countries-projected.json

If you want to check out these files after they’ve gone through these commands you can open them in [insert favorite gis software here].

Filter Geography

Let’s say you only need to filter your geography by countries with a low population. First let’s see what attributes our file has with:

mapshaper countries.json -info

Looks like POP_EST is what we’ll want to go with. We can filter our features with:

mapshaper countries.json -filter 'POP_EST < 250000' -o countries-lowpop.json

The part of the text here between the hyphens is a Javascript boolean test. For each feature it asks “is the POP_EST attribute less than 250000?” and if it doesn’t, the feature is excluded from the output. This’ll work on for filtering any attribute that is a numeric value.

You can also filter by strings. Let’s get just countries in South America:

mapshaper countries.json -filter '"South America".indexOf(CONTINENT) > -1' -o south-america.json

This does the same test as before, this time with a bit of Javascript called indexOf. This looks at the value of CONTINENT and if it matches “South America” returns a 1. Because 1 > -1, the statement is true and that feature’s saved. This’ll also work for any attribute that is a string.

Projections, cont.

In addition to the default projections mapshaper has, you can also use proj4 strings for more specificity.

epsg.io is a great resource for getting proj4 string defintions, letting you search either by projection name or country. Once you’ve selected a projection, just scroll down the page to the Export menu and click Proj.4 to get a defintion.

Just paste that string after -proj, and you’ll be good to go:

mapshaper south-america.json -proj +proj=poly +lat_0=0 +lon_0=-54 +x_0=5000000 +y_0=10000000 +ellps=aust_SA +towgs84=-57,1,-41,0,0,0,0 +units=m +no_defs -o south-america-projected.json

Join CSV Data to Geography

One last example– joins. Let’s join my csv of red panda habitats to the original countries.json dataset.

mapshaper countries.json -join native-red-pandas.csv keys=NAME,COUNTRY -o countries-with-pandas.json

Here we need to put the path to the data.csv following -join. keys= tells mapshaper which attributes to join by, the former being the field in the GIS file. The later is what field to join by in the csv.

Chaining it all together

Doing these step by step will work fine, but you may notice a lot of repetition in the commands structure. We can actually run a lot of these commands together likeso:

mapshaper countries.json \ -join native-red-pandas.csv keys=NAME,COUNTRY \ -filter '"yes".indexOf(PANDAS) > -1' \ -clean \ -proj +init=EPSG:32645 \ -o countries-with-pandas.json

For terminal commands, if you write a command on a new line, it will understand it as two separate commands. The \ character tells it to ignore the new line and instead read it as one uninterrupted command.

You might notice that I snuck one more new command in there; clean. Sometimes the data we’re working with has minor topological errors in it– or we create them along the way. Mapshaper’s clean function helps tidy up those stray points and mis-aligned borders.

At the end, you should end up with a json of just the five countries that are the current home of red pandas. Fast, eh?

If, somehow, they discover Red Panda’s also exist natively in Ireland, we only need to update the csv with that information and re-run the final script. It will work the exact same- only this time including Ireland in the final output. For practice, try it out! The projection may be a bit off… but you can also tweak that!

If you’ve made it this far, kudos! Hope it’s helped demystify CLI ever so slightly.

In the next part, I’ll introduce Makefiles , which are an old school way of running commands from a saved file vs. having to copy and paste into the terminal to run. We’re gonna build some CLI robots!

Til then,

–D.M. cd ~./ny/new-york/brooklyn/boerum-hill
Feb.18th, 2019