Just as Mario: Using the Plan9 plumber utility

Flattr this
Note: It's best to open the videos in full screen. Also I have added a few line breaks or readability in the code snippets that will make them not work correctly. It's not hard to find where they are, if you run into any problems let me know.

If you've been following this blog, you'll know I've been using Acme and related Plan 9 from User Space utilities lately. One of its pieces is the plumber. And knowing what it does may be tricky:

Plumbing is a new mechanism for inter-process communication in Plan 9, specifically the passing of messages between interactive programs as part of the user interface. Although plumbing shares some properties with familiar notions such as cut and paste, it offers a more general data exchange mechanism without imposing a particular user interface.



What is the plumber?

It's somewhat hard to explain at first. I'll try to give you a simple example from Mac OS. In Mac OS you can find a command line utility named "open" (/usr/bin/open). From its manual entry you can read:
open -- open files and directories
Simple and easy. And it just does what you'd expect:

  • open something.jpeg will open this image in Preview (or its default opener)
  • open afile.txt will open this text file with its default opener, too.
  • open http://web.something will open the webpage in the default browser.
I think you get the idea. Open just opens something. If you use the command line in Mac OS, you have probably used it on and off to quickly open images or pdfs. I know I have.

The Plan9 plumber is like open on steroids. Why?

Plumbing superpowers

The strength of the plumber is two sided:
  • Completely configurable
  • System wide (kind of)

Completely configurable

The plumber works with rules, and you can just write your own cool roles in a file ~/lib/plumbing What does a rule look like? Here is a rule to open an image using the Mac OS open utility:

type is text
data matches '[a−zA−Z0−9_–./]+'
data matches '([a-zA-Z¡-￿0-9_\-./]+)
  \.(jpe?g|JPE?G|gif|GIF|tiff?|TIFF?|ppm|bit|png|PNG)'
arg isfile $0
plumb start open $file

Rules work in a trigger-and-fire way. A set of tests make the trigger, if the text you pass to the plumber matches, the rule is fired. In this case:

  • The type of the data has to be text (in fact is the only type the plumber handles)
  • The content of the message has to match this regular expression [1]
  • The content of the message has to match this regular expression, with groupings
  • The argument group $0 (the whole regexp match) has to be a file, then the variable $file holds its full path
  • THEN we run "open $file"

[1] This is needed. From the plumb manual section 7 (man 7 plumb or right click plumb(7) inside acme):
The first expression extracts the largest subset of the data around the click that contains file name characters; the second sees if it ends with, for example, .jpeg. If only the second pattern were present, a piece of text horse.gift could be misinterpreted as an image file named horse.gif.
This relatively simple example is big enough to allow us to make pretty cool things with the plumber. Although this example only allows us to type plumb filename.jpg and the image will be opened via "open".

But we may get as fancy as we like, for example:

# isbn10 search through Amazon
type is text
data matches '([0-9][0-9][0-9][0-9][0-9][0-9]
  [0-9][0-9][0-9][0-9])'
plumb start open 'http://www.amazon.com/s/?field-keywords='$1

  • Type has to be text, as always
  • Match exactly 10 digits (Plan9 regexes don't have brace count)
  • Then search the ISBN through Amazon
Now we can run plumb 0836213122 to find what book this ISBN points to. Neat, isn't it? Well, it just gets better.

System wideness of the plumber

When I say the plumber works system wide, I mean that in the Plan9 ecosystem (or in plan9ports) anything the plumber recognises as "special" can be easily plumbed. For example, I'm typing this in acme, thus I can right-click this ISBN (0836213122) to switch to Chrome and find what this book is. And if I'm in a 9term window, I can middle click-and-release to get the same. It's neat, but of course it gets neater: right clicking in an expression filename:lineno gets you to this match, making grep a whole lot more useful.

For me, its main strength is inside acme, since it is almost the only piece of Plan9 I use with regularity (I use zsh which is pretty good as it is.) As such, using the plumber we can add even more awesomeness to this editor:

type is text
data matches 'ag:"?([a-zA-Z¡-￿0-9_\-./]+)"?'
arg isdir .
plumb start zsh -c 'ag --nogroup --nocolor --search-files "'$1'" '$dir' 
  | plumb -i -d edit -a ''action=showdata filename='$dir'/ag/'$1' '''

This lets me write ag:word or ag:"something longer" to search for word or "something longer" recursively in the current working directory using The Silver Searcher.

The Silver Searcher is a source code file searcher, similar to ack. Ack and tss are both faster than grep because they omit .ignore, backup and non-source files. Of course this can be overriden, but since I usually work only with source when using grep, it is a great tradeoff.

The --search-files is a hidden option, without it line numbers are not rendered when ag is ran from another program (see this github issue).

Or I can add the following to use my wikipedia quick searcher script (based on this neat commandlinefu hack):

type is text
data matches 'def:"?([a-zA-Z¡-￿0-9_\-./]+)"?'
plumb start zsh -c 'wikiCLI.sh '$1' 
  | plumb -i -d edit -a ''action=showdata filename='$dir'/wikipedia/'$1' '''

So I can write and use def:europium in case of need. Of course, it is far more handy to use acme's 1-2 chords for this: select europium with button 1, select wikiCLI.sh (the name of my script) with button 2 and without releasing, tap button 1...



As you can guess, this won't work in a 1-button Mac laptop without external mouse, since you can't press button 1 while pressing button 2... After all, button 2 is button 1 while pressing Alt. To make it work, I patched devdraw in plan9ports:

diff -r 1bd8b25173d5 src/cmd/devdraw/cocoa-screen.m
--- a/src/cmd/devdraw/cocoa-screen.m Tue Mar 19 14:36:50 2013 -0400
+++ b/src/cmd/devdraw/cocoa-screen.m Sat Apr 06 20:02:44 2013 +0200
@@ -847,7 +847,9 @@
  case NSFlagsChanged:
   if(in.mbuttons || in.kbuttons){
    in.kbuttons = 0;
-   if(m & NSAlternateKeyMask)
+   if(m & NSControlKeyMask)
+    in.kbuttons |= 1;
+                        if(m & NSAlternateKeyMask)
     in.kbuttons |= 2;
    if(m & NSCommandKeyMask)
     in.kbuttons |= 4;

And now you can press Ctrl while having Alt pressed to send a "button 1" chord. Simple and innocuous patch.

Final remarks

As you can see, the plumber can be useful in every day tasks, enhancing acme. Together with the ability to execute text, the plumber makes acme and incredibly different text editor. It's not vim, it's not emacs. It is acme, in its own right a very powerful piece of software. I hope these posts I'm writing lately are making you have an itch to use acme and have a look at what Plan 9 has to offer.

Extensibility in the Acme text editor

Flattr this
Text editors. You hate them or love them. Praise them with religious zeal, and attack them with the same power. I've been an emacs user for the last 8 years, getting as deep as I could without checking the source. And the past few months I have started using evil-mode in emacs, to get some taste of vim in my daily editing (mostly text objects.)

There's still a third contestant in editor-land, for me. It is Acme, the odd editor from Plan9 from Outer Space, the even-more-odd operating system from Bell Labs. There's no need to install Plan9 and fight against your current hardware. If you are in any kind of Unix derivative (Mac OS, Linux) you can install Plan9 from User Space, a port of most of Plan9 to work in user space (as you may guess.) Plan9 is a whole different thing from other Unix systems, and Acme is an incredibly different beast from any other editor you know.

I can start with a screenshot of it:

This is how this post looks like. You can see an adict window by the side
This is Acme. I hope you like this shade of yellow and this shade of blue. There's no way to change it without getting into the source code and recompiling. It may be sound odd, but I kind of like it. It's refreshing. In emacs and vim it is very easy to get a beautiful colour scheme (I use solarized-dark everywhere I can,) but this means you can choose. And choosing means a decision, with pros, cons and whatever. Just screw it and pick blue and yellow.

Once you are used to it, you have to face something "worse." If you come from emacs or vim this will sound just horrible. Wait for it. Everything is done with the mouse. Yes, you read that well. No keyboard shortcuts (well, there are a few, I'll get into these in a short while.) Mouse clicking, moving and chording. The likes. I know this will sound stupid, a waste of time, prone to carpal tunnel syndrome. Let me go on for a while.

Emacs and vim users alike like to bash any other editor in the grounds of speed. I can refactor faster than you can, is almost the motto. Watch how fast I type, thus how fast I change code. I'm one of these, I usually don't even have to think when I'm doing "something" in emacs or vim and changing stuff. But then again, how often I'm changing stuff?

Emacs and vim make easy changing what's there. Multiple-marks, text objects, quick jumps. All this is there just to make changing stuff fast. Agree? Ok, go on. If you don't, no problem. Go on anyway.

Now the revelation: most of the time I'm writing, I'm creating new stuff, not rewriting or moving old stuff. Shocking? Watch your own coding/writing habits. Yes, I change what's in text strings (ci", in vim) and is incredibly fast. In Acme, you can double-click after the first quotation mark (or just before the last) to select everything inside a pair of delimiters (a pity it is not smart enough to understand dollar-delimiters as used by LaTeX.) But the point is not that speed. What are you changing this string for? Did you wait to think about it or you just changed it, compiled it, checked it and went back to square one?

Pause: the file servers

Acme and Plan9 follow a special philosophy: in some sense, everything is a file. And most programs (I could say all, but I'm not that into Plan9 to be sure) act as file servers. Acme is just one of these: everything you can see in an Acme session is a file. For example, this text I'm writing (you saw it in the previous screenshot) has window ID 10. So...

acme/10/       is the directory associated to this text
acme/10/addr is a file with an address position for text insertions
acme/10/body is a file with the contents of the editing window (can't overwrite)
acme/10/ctl is a "file" (socket-like) that allows you to send commands to the window
acme/10/data is a file with the data of the editing window (can overwrite) 
acme/10/errors is a file with data spat by commands executed by this window
acme/10/event is a "file" (socket-like) where you can read/write the editing session
acme/10/tag is a file holding the contents of the tag (the menu above)
acme/10/xdata is a file with the data (addr bound) of the editing window

What does this mean? It means I can write code in any language that can manipulate this text. Read this out loud: I script something, and make it work with the text I'm editing. Think about indenting, linting, type-checking, done against the working copy, not the real file. With output to a special buffer associated to the file. Extending the editor is just a matter of writing a program.

Back into Acme

Back into Acme. In Acme there's no GUI: text is the user interface. TUI. Every window is composed of two pieces: the text in the window (its "body") and a tag above it (in blue.) If you want to copy something, select it and middle-click on Snarf. Then put your cursor where you want to paste and middle-click on Paste. Done. Of course, Snarf and Paste can be anywhere. In the tag menu, in the text you are editing or even in another document. They are just words. Words that do the work. 

But the same works with shell commands. I can type date and middle-click it, to get the current date in a new buffer. Same goes with ls. Or even |md5sum to calculate the checksum of some text. Or append something to a window. For example, there's an easy way to make small queries to Wikipedia via the command line (see http://www.commandlinefu.com/commands/view/2829/query-wikipedia-via-console-over-dns) I wrote a script to do it, sitting in my path, so I can now type here <wikiCLI.sh Acme with the cursor a few lines below this... and middle click.

Result:
"Acme (\; , the peak, zenith, prime) denotes the best of something. Acme or ACME may also refer to: Acme Corporation, a fictional company in the cartoon world of Looney Tunes, ACME Detective Agency, a fictional detective agency from the Carmen Sandiego seri" "es of computer games and television shows, Acme (album), the sixth album by the Jon Spencer Blues Explosion, Acme Novelty... http://en.wikipedia.org/wiki/Acme"

On button clicking

I use a Macbook (almost 5 years old already, and still kicking.) And as you may guess, it only has one button. So, how do I manage to use middle and right clicking with ease? Well, easy. Or almost. Pressing alt while clicking simulates middle click, command while clicking simulates right clicking (in Acme, not in general.) Easy, since alt is in the middle and command just right to it. 

Problem is, chording is most awesome with a real 3-button mouse. Why? I'd rather search for wikiCLI.sh Acme (text editor) to get:

;; Truncated, retrying in TCP mode.
"Acme is a text editor and graphical shell from the Plan 9 from Bell Labs operating system, designed and implemented by Rob Pike. It can use the sam command language. The design of the interface was influenced by Oberon. It is different from other editing " "environments in that it acts as a 9P server. A distinctive element of the user interface is mouse chording... http://en.wikipedia.org/wiki/Acme_(text_editor)"

Doing so is a little more troublesome: select "Acme (text editor)" and select the command with the second button (to execute) finally click: clicking sends the last selection as argument 1 to the program. Doing so with a Mac trackpad is impossible: there's no way to simulate a left-click while middle-clicking. I also find there's a glitch here: there's no way to redirect the output of the command. It's either overwriting the selection of the argument, or goes to the +Errors window.

Selections, regexes and other furry animals

How can I select everything? :0,$ and right-click. Done. Want to replace all instances of acme for Acme? Middle click this: Edit ,s,acme,Acme,g The sam syntax is easy but... odd. The first , is to select everything, s to replace acme for Acme, g for global. Easy? Not much, but Acme is just different. And works. Edit is the command to execute an editing command, by the way.

You can also do regex searches. Like :/interface/ or :/click[i|.]/. And you can get fancy, by doing filename:<search-or-position>. For example, acmescripting:/interface/ in another window would open acmescripting in the first instance of interface. And acmescripting:20 opens it, selecting line 20. As you can see, filename is implied to be current file in case of doubt. Also, this kind of referencing works nice with most compilers and linters.

Keyboard shortcuts

There are a few keyboard shortcuts, even if Acme is mouse-centric. Press Esc, and all text written since the last click is selected. In addition to this, we have the other standards:

C-U –> Delete from cursor to start of line.
C-W –> Delete word before the cursor.
C-H –> Delete character before the cursor.
C-A –> Move cursor to start of the line.
C-E –> Move cursor to end of the line.

Nothing more, nothing less. Minimal, isn't it?

Scripting power

Finally, I want to show some scripting power of acme. I introduced the concept of the filesystem a few paragraphs ago. Now, let's see how it can be used. Let's say for example I'm an avid C programmer, and like to have code neatly indented. Well, an option is to write a script that uses the indent command line program to indent the text in the window. How? Now comes a trivial example, not written in the best way possible

#!/bin/zsh
#9indent
echo "WinId is: " $winid
echo -n "1,$" | 9p write acme/$winid/addr
echo "Selected whole contents for overwriting with 'write'"
9p read acme/$winid/body | indent -st | 9p write acme/$winid/data

First is to check which is the ID of the window. When invoking a command, its environment has it in a variable, aptly named winid. To overwrite the contents of the file, we set the addr to the whole file with the selector 1,$. We do so by piping to 9p with the command write. 9p is the middleman, allowing us to read and write files in 9P servers, like the ones acme and other Plan9 programs offer. Finally we get the body, indent it (-st is the command to use stdin in indent) and pipe it to data. Done! This indents a well-formed C file as expected.

Add this file to your path and add 9indent to your tag. Ready to indent by middle-clicking.

A slightly more complex example is to generate the output of a Markdown file. The code is as follows:

#!/bin/zsh
echo "WinId is: " $winid
format=$(9p read acme/$winid/tag)
echo "Tag is " $format
echo "Format in tag"
case $format in
    *"latex"* )
        echo "latex ouput selected"
        format="latex" ;;
    *"groff-mm"* )
        echo "groff-mm output selected" 
        format="groff-mm" ;;
    *"odf"* )
        echo "odf ouput selected" 
        format="odf" ;;
    *"html"* )
        echo "html output selected" 
        format="html" ;;
    * )
     echo "Unrecognized format, defaulting to html"
     format="html" ;;
esac
echo -n "1,$" | 9p write acme/$winid/addr
echo "Selected whole contents for overwriting with 'write'"
9p read acme/$winid/body | peg-markdown --to=$format | 9p write acme/new/body
echo "Wrote the html-markdowned version to a new buffer"
last=$(9p ls acme | sort -g | tail -n 1)
echo "Get last created buffer"
echo -n "clean" | 9p write acme/$last/ctl
echo -n "0,0" | 9p write acme/$last/addr        
echo -n "dot=addr" | 9p write acme/$last/ctl
echo -n "show" | 9p write acme/$last/ctl    
echo "Moved to beginning"

This is slightly more complex, at least on the shell side. It checks the tag for one of the accepted formats for peg-markdown and then creates the formatted output in a new window, by writing to acme/new/body. Then I want the cursor to be at the beginning of this file, not at the end (as is the default.) It was slightly tricky, but the best way was to sort in numerical order and get the last-created window (that's this tail -n 1) then to set the address at 0,0 and set the dot (selection) at address by writing at the control file. Then the command show makes the window show the selected position: 0,0. Done! Intersped among all this is a "clean" command, to make this new window to close.

Here you can see a video of these scripts in a sample use (and you'll see how I miss a middle click - execute -  for a right-click - open)



Dirty, Clean, Put

A window can be dirty or clean. It is clean when the contents and the disk file are the same. It is dirty when it is being edited. The best way to know if it is dirty is if you see "Put" in your tag menu, just beside the vertical bar. By middle-clicking Put (or Putall in the main tag) you save this file and mark it as clean. 

Also, making a window clean makes closing it quicker (middle click in Del.) Dirty windows need to be Put, or you have to Del again. 

That's all, folks (for now)

I have yet to introduce the plumber, a mechanism that allows you to open arbitrary files (using rules) from within acme. For example, I can open pdf files by right-clicking on them (i.e. some.pdf) but instead of using page (the Plan9 image viewer) I use MacOS Preview. I was forced to do so, since page can't handle all the fonts in a LaTeX generated PDF, so for me it's useless. I'll probably write how I configured the plumber in my next Acme installment.

In some sense, the plumber is like a system-wide, app-deep "open" mechanism. In Mac OS, you can "open" almost anything from the command line. If you open an URL, your default browser opens it, if you open an image, Preview handles it. Plumbing is like "open 3.0" but it is hard to manage :/

Below you can see another video with a simpler scripting: browsing reddit from the command line, inside acme. The Python code snippet that gets Reddit data is available in this gist: reddi.py

Creating New Text Objects in Evil-Mode (Vim emulation layer in emacs)

Flattr this

The links to Practical Vim are affiliate links to Amazon. Beware!

So... last January I was in a flight to London, preparing for an intense, 12 days course on traditional shoemaking (English hand-welted shoes, improving our knowledge at The Fancy Puffin.) And my flight read was Practical Vim. Most of my readers are already aware I'm an emacs guy, so the main question is why?

I love knowing many tools. Programming in many languages, knowing enough about any thing. I couldn't stand all these people saying Vim was superior (even though most had never touched emacs,) and since knowledge is power, I thought a little more (for me) could not hurt (mwahahahaha!) I picked Practical Vim and started reading it a few days before leaving, and finished it during the few non-shoemaking moments of that course.

When I was back to having a computer, I was torn between what I learnt about Vim and looked cool and all those things I already loved about emacs. I have MacVim a few tries, writing short pieces, writing, testing, rewriting. Clearly there was no way I could give up gnus, org-mode (even though I'm not using it that much lately) and all my complex configuration options. So... how come? Even though modal editing feels weird, text objects are cool. Being able to "say" di( and get it to delete inside a pair of parentheses is way cool. Basically this is what sold me as a great feature.

I decided to use evil-mode in emacs. Best of both worlds: I still had all my emacs tools AND text objects and some modal editing. I missed having numbered lines, although I just googled for it... And there's linum-mode (since emacs 22!!) which gives them. But I miss ex commands: being able to move any line anywhere just by issuing :source m destination is powerful. Evil-mode supports some ex commands, just not copy or move. Sigh!

Today was the first day I added evil-mode to my .emacs file, and the day started by working in a LaTeX file. In a LaTeX file, most things are enclosed in pairs of dollars, like $x=1$. These dollar delimited expressions are what TeX parses as formulas, and they appear as neat images in your target (usually PDF) file. So, most of the things I need to change, delete or move around are enclosed by dollars. So:
  • change what's inside a pair with ci$
  • delete the whole formula with da$
  • delete just the contents of the formula with di$
  • copy the formula with yi$
Useful, isn't it? Problem is, evil-mode does not have $ as a text object! Luckily it offers the tools to make new text objects... Getting it to work was harder than expected, though. I was given several choices of functions I could use to create my text object. The best option seemed to be using evil-regexp-range that allows you to give a regexp for the first delimiter and another one for the second. But all tries I have done resulted in it marking the first occurrence in the buffer. No luck. After trying repeteadly with it, I gave up and used instead evil-inner-object-range, where I had to wrestle with point, mark, excursions and the like. When I managed to get it working I just decided I would not touch it again (even if I'm pretty sure the save-excursion calls are redundant, as well as most move and mark commands...) It works, don't touch it.

(defun rb-first-dollar-exclusive (&optional arg)
(interactive)
(save-excursion
(setq rb-temp-first (re-search-backward "\\$" nil t 1))
(message "First %s" rb-temp-first))
(goto-char (+ 1 rb-temp-first)))

(defun rb-second-dollar-exclusive (&optional arg)
(interactive)
(save-excursion
(setq rb-temp-second (re-search-forward "\\$" nil t 1))
(message "Second %s" rb-temp-second))
(goto-char (- rb-temp-second 1)))

(evil-define-text-object evil-inner-dollar (count &optional beg end type)
(evil-inner-object-range count beg end type
#'rb-second-dollar-exclusive
#'rb-first-dollar-exclusive))

(define-key evil-inner-text-objects-map "$" 'evil-inner-dollar)

(defun rb-first-dollar-inclusive (&optional arg)
(interactive)
(save-excursion
(setq rb-temp-first (re-search-backward "\\$" nil t 2))
(message "First %s" rb-temp-first))
(goto-char rb-temp-first))

(defun rb-second-dollar-inclusive (&optional arg)
(interactive)
(save-excursion
(setq rb-temp-second (re-search-forward "\\$" nil t 1))
(message "Second %s" rb-temp-second))
(goto-char rb-temp-second))

(evil-define-text-object evil-outer-dollar (count &optional beg end type)
(evil-inner-object-range count beg end type
#'rb-second-dollar-inclusive
#'rb-first-dollar-inclusive))

(define-key evil-outer-text-objects-map "$" 'evil-outer-dollar)


(Blogger is killing indentation, damn) Enjoy, though!

An introduction to the magic of Google Tag Manager

Flattr this
If you want to read a quick overview of this in Spanish, check my post about GTM at DoctorMetrics.

Google has just unveiled a new tool: Google Tag Manager. I have spent a few hours playing with it (both before and after official release.) And it's awesome! Or at least, it has quite a lot of awesomeness, hidden behind a seemingly simple interface.

Its main feature is the fact that you can get away with just a piece of included code. You don't need to include your Analytics code on top and your DoubleClick counters in all the pages you need. There's just one thing you can't do: change the content of your page (via document.write)

So, what's Google Tag Manager good for (exactly)?

It centralizes the management of page tagging and tag execution based on rules. A simple example: you want to track how many people click on the Atom subscribe link in your Subscribe page with event tracking via Google Analytics. This  is straightforward with Google Analytics, in fact: you only need to set the correct event for onclick in the page source like this:

_gaq.push(["_trackEvent", "SubscribeRSS", "Sidebar"]);



But what happens when you want to change this tracking category, or add a new tracking to another link? You have to edit all required pages in your site. If the site is well structured, this will be easy, but not all websites are well-structured. Or you are not the webmaster. This is the main point: what if you are just an external consultor? Exchanging 15 emails to get the IT staff to add this onclick can easily drive you nuts (and cost you and your employer many hours) You don't have to get nuts any more.

After creating a container and adding the GTM code to your site, you just need to add the required page selector in Google Tag Manager's rules:

url> contains> "subscribe"




As easy as that... But you also need to add the event tracker to that link. How? The simplest way I've come up is with some basic jQuery magic. Include jQuery (which you can do for all pages, not just subscribe!) and then set the gaq push event, by selecting the correct link via the href attribute of the link. You can select based on text, or anything you fancy, of course.

<script 

  src="//ajax.googleapis.com/ajax/libs/jquery/1.8.0/jquery.min.js" type="text/javascript">
</script>

<script type="text/javascript">
$('a[href$="http://www.mostlymaths.net/feeds/posts/default"]').click(
  function(event){     _gaq.push(["_trackEvent", "SubscribeRSS", "Atom"]);   });
</script>

With jQuery you can go as far as to dynamically change the link to add campaign tracking, to any link in the page:

$('a[href$="http://www.mostlymaths.net/feeds/posts/default"]').attr("href",

   "http://www.mostlymaths.net/p/subscribe.html?utm_source=GTM&utm_medium=Redirect&utm_campaign=Testing")

The options available are almost limitless: enjoy your new GTM goodness! And feel free to contact me for more implementation details or ideas.