A Simple Example of Customizing GNU Emacs
*****************************************

Introduction
============

What Is This For?
-----------------

The purpose of this tutorial is to demonstrate the process of customization by having the reader work through an example. It does not explore any part of GNU Emacs in great detail.

There's homework at the end, so that you can apply what you've learned, using a now-familiar chunk of code. (see exercises)

What Is This?
-------------

This file is something like a transcript, with commentary, of a GNU Emacs customization.  It is written somewhat in the style of the GNU Emacs tutorial, so that you can---and should---follow along and do the customization yourself.

Who Is This For?
----------------

This particular customization was one of the first I did.  It is not a contrived example---it's what I actually did, recorded as I did it (though since modified for new versions of Emacs).  It's probably most suited to a person who knows Emacs, C, and some form of Lisp reasonably well.

How to Use It?
--------------

First, make a private directory and run Emacs there.  Run info and select (customizing-emacs).  (see (info) for information about info.)  (You may also read this tutorial in hardcopy.  Hardcopy saves screen space, but you'll still need an info version so that you can copy text from it.)

Read through the text.  Some parts of the text explain Emacs code---read the explanation and also the actual code in another window.  Other parts of the text explain what I did to browse through Emacs code.  Do the same browsing---and any other browsing that strikes your fancy.  Browsing is fundamental to understanding.  Other parts of the text have actual large chunks of new code, often following motivation and explanation. Copy that code into the working file (named below); install it and use it as explained in the text (usually in paragraphs after the body of the code).

Notation
--------

For the rest of this document, I'll assume that GNU represents the location of the Emacs directory on your system.

Author Information
------------------

This text is copyright © 1986 by Brian Marick. Permission is granted to make and distribute verbatim copies, provided the copyright notice and this permission notice are preserved on all copies.

Please send comments on this tutorial to marick@gswd-vms.arpa or ihnp4!uiucdcs!ccvaxa!marick.

The Problem and a Proposed Solution
===================================

I make heavy use of the csh(1) cdpath feature. cdpath is a variable used by the shell in this way:

     If you try to change your working directory to a relative pathname that is not a subdirectory of the current working directory, the csh will then check if the pathname is a subdirectory of any of the directories in the cdpath.

     For example, suppose you are in /mnt/marick and your cdpath contains /lisp/testing.  If you type cd tests and /mnt/marick/tests exists, you will change to that directory.  If /mnt/marick/tests does not exist, but /lisp/testing/tests does, you will change to that directory.

cdpath can cause confusion when you use the shell within an Emacs buffer.  Emacs calculates the buffer's current working directory by trying to imitate the actions of the directory-changing commands.  Since Emacs doesn't know about the cdpath, it can't know that cd tests means cd /lisp/testing/tests.  Consequently, after you type cd tests, Emacs's idea of the current working directory differs from the shell's.

One possible solution would be to let Emacs know about the cdpath. However, that wouldn't work for another style of directory-changing. Many people use csh aliases to change directories.  For example, they place the following in their .cshrc file:

     alias tests 'cd /lisp/testing/tests'

so that typing tests to the shell changes to the /lisp/testing/tests directory.

The most general solution seems to be to have all directory-changing commands and aliases print the directory stack after they do their work: that is, that they all do what dirs(1) does.  pushd and popd already do that; this solution requires that users of cd alias it with

     alias cd 'cd \!*; dirs'

and that other special aliases do the similar thing.

Implementing The Solution
=========================

What we're concerned with is what Emacs does when it sends a directory-changing command to the shell, which is surely a special case of what it does when sending any command.  A first step is to look at the function that RET invokes when in a shell buffer.

Call up a shell buffer with M-x shell RET.  Inside the buffer, type C-h k RET to see what function is bound to RET.  You will see that the function is shell-send-input. We need to find where that function is defined.

Using dired on the GNU Emacs Lisp directory is one way to find possible source files.  (We'll see much better ways later.) Do that by typing C-X d. Answer GNU/lisp/*.el to the minibuffer prompt.

Browing through the directory, we see this line:

     -r--r--r--  1 marick      13150 Jul 18 15:10 shell.el

Such a file looks a likely candidate, so visit it (by typing f when the cursor is on that line).  (Don't view it---we'll want to do cursor-positioning, and view-mode turns off keys like Control-f and Control-e.)

Explanation of shell-send-input
-------------------------------

shell-send-input is indeed in this file.  Find it. Here (briefly) is what it does:

     The first fourteen or so lines are concerned with acquiring the text to send to the shell.  That text is all the text on the current line, less the shell prompt.  (There are two cases: if the current line is at the end of the buffer, it is newline terminated.  If it is within the buffer, it is copied to the end.)

     Before sending the text to the shell, shell-send-input calls shell-set-directory, which must be the code that checks for directory-changing commands.

     Notice that the call to shell-set-directory is wrapped within a condition-case and a save-excursion.  A condition-case catches any errors signalled by the enclosed code. In this case, an error causes a call to the shell-set-directory-error-hook.  The default value for that hook function is ignore, which silently ignores the error.

     A save-excursion prevents any changes made to the mark and point >from affecting their values outside the enclosed code.

     We could, at this point, try to make use of the shell-set-directory-error-hook to solve our problem, but instead let's continue on our original course and examine shell-set-directory.

     In general, directory-changing commands manipulate the shell-directory-stack.  Note that this stack is not the same as the shell's stack (as printed by dirs).  The shell keeps the current directory on the stack; Emacs keeps only old directories. (We must remember to maintain this convention when we make changes.)

     In particular, the code is split into several cases.

     If the command matches the shell-popd-regexp and the popd has no arguments, the Emacs cd function is used to change to the top directory on the stack and the stack is popped.

     If the command matches the shell-pushd-regexp, one of two things happens:

      1. If the pushd has no arguments, that means ''switch the top two elements of the shell stack''.  That's done by popping and changing to the top directory on the Emacs stack and pushing the previously-current directory on the stack.

      2. If the pushd has an argument, the argument is read and variables and special metacharacters are expanded to find a directory to move to.  The previously-current directory is pushed on the stack, and the current directory is changed. If, however, the directory doesn't exist (function file-directory-p returns nil), the Emacs shell-directory-stack is not changed, nor is the current directory.  (Note that this is what would happen if Emacs tried to execute pushd testing in our stock example.)

               TANGENT: the substitute-in-file-name function is the first one whose purpose hasn't been reasonably obvious.  We might want to look at it to see what it does.  First, create a tag file in this directory by executing the etags program with

                    M-! etags GNU/lisp/*.el RET

               then position the cursor over the use of substitute-in-file-name and type Meta-..  You'll be asked for a tag to find; typing just a carriage return will take you to the definition of substitute-in-file-name.  Next, you'll be asked which tag table to visit---make sure you visit the one you've just created.

               Confusion!  The search fails.  One possible explanation is that this function is defined in C, not Lisp.  We need to make our tag table more comprehensive.  Here's a way of doing it:

                    M-! etags GNU/lisp/*.el GNU/src/*.[ch] RET

               Now M-. RET shows the function.  Reading substitute-in-file-name reveals that it can signal errors, which explains a need for the condition-case, which catches and (by default) ignores them.

     Continuing with the explanation of the function:

     If the command matches the shell-cd-regexp, one of two things happens:

      1. If the command has no arguments, Emacs changes to the home directory.  Note that the shell-directory-stack needn't change.

      2. If the command has an argument, it is expanded and Emacs changes to that directory.

Returning to shell-send-input: after the call to shell-set-directory, we notice the three lines of code that actually send the command to the shell.

It appears that what we need to do is combine and rearrange the code in this way:

 1. Check and remember if the command matches any of the three regexps.

 2. Send the command to shell.

 3. If we saw a matching command,

      1. Read what the shell just printed.

      2. If it wasn't an error message,

           1. Parse the string into a list of strings.

           2. Change directory to the first one and throw it away.

           3. Substitute and expand the remaining strings.

           4. Make that the shell-directory-stack.

Subproblem 1: How to read what the shell prints.
------------------------------------------------

Looking at the code that sends to the shell, we see that it sends through last-input-end.  The text between last-input-end and a following newline ought to be what we want.  The code to extract the shell's output ought to look something like the code that's already used to extract the argument part of a directory-changing command.

We should write this code now, before going on.  We begin by editing a file called custom-shell.el, which will be used for all our changes to Emacs shell interaction code.  We will

 1. copy in the code for shell-send-input and change its name to test-shell-send-input,

 2. remove the code that parses directory-changing commands, since we'll be letting the shell do that,

 3. and add code that extracts what the shell prints (the dirs-style directory stack) and displays it.

One question:  how to display what's extracted?---how to debug? If you look at the code for shell-send-input, you'll see that it expects to be run when a RET is typed, since it searches for that character.  That makes it inconvenient to call test-shell-send-input in any other way.

An obvious---and generally useful---thing to do is display the result in the minibuffer.  Fortunately, we remember that changing the directory in a shell window already prints the new directory in the minibuffer.  So we look at (using tags) the cd function, which leads us to pwd, which shows us the message function, which is what we want.

The final version looks like this, where lines marked with ;NEW are new.

     (defun test-shell-send-input ()
       (interactive)
       (end-of-line)
       (if (eobp)
           (progn
             (move-marker last-input-start
                          (process-mark (get-buffer-process (current-buffer))))
             (insert ?\n)
             (move-marker last-input-end (point)))
         (beginning-of-line)
         (re-search-forward shell-prompt-pattern nil t)
         (let ((copy (buffer-substring (point)
                                       (progn (forward-line 1) (point)))))
           (goto-char (point-max))
           (move-marker last-input-start (point))
           (insert copy)
           (move-marker last-input-end (point))))
       (let ((process (get-buffer-process (current-buffer))))
         (send-region process last-input-start last-input-end)
         (set-marker (process-mark process) (point))
         (accept-process-output process))                            ;NEW
       (save-excursion                                               ;NEW
         (message "%s"                                               ;NEW
                  (buffer-substring last-input-end                   ;NEW
                                    (progn                           ;NEW
                                      (goto-char last-input-end)     ;NEW
                                      (skip-chars-forward "^\n")     ;NEW
                                      (point))))))                   ;NEW

Most of the NEW lines correspond well to what we planned to do.  We print a message from the end of the input through the next newline.  The save-excursion prevents the changes to the point from contaminating where the user left the cursor.

What's very different is use of accept-process-output.  In retrospect, of course, it's obvious why it's necessary, but finding out in the first place was some work.  The reason is that send-region sends commands to the process but doesn't wait for the process to respond.  So if we later tried to skip past a line of the shell's output, we'd get a null string, because there isn't any output yet.  accept-process-output (which we discovered by looking at send-region through the tag table and browsing through the file that contained it) waits for process output.

Note that the real version of shell-send-input should only use accept-process-output when a directory-changing command has been seen. We don't want to block on any arbitrary line of input sent to the shell, only on directory-changing commands.

Try playing around with the code; simulate what's done during debugging. (For example, remove the call to accept-process-output.) To execute the code, do the following:

 1. Evaluate the definition with C-M-x. (Note:  C-M-x expects to find (def at the beginning of the first line in the function, so remove the leading blanks.)

 2. Create a shell, and install this new function as the function bound to RET with

          M-x local-set-key RET RET test-shell-send-input

 3. Type things  to the shell.  (Try echo wombat and dirs.)

Subproblem 2: Parsing shell output.
-----------------------------------

The only remaining problematic part of our original plan is parsing the output from the shell into a list of strings.  Since strings are a basic datatype, the routines to manipulate them are probably written in C.

Grepping for string functions with

     M-x grep 'DEFUN.*string' GNU/src/*.c

yields a quite a number of functions, including

     ../src/fns.c:362:DEFUN ("substring", Fsubstring, Ssubstring, 2, 3, 0,

Browsing through fns.c, we discover a number of useful functions --- notably elt, to extract characters from a string.  With those two functions and the special form while that we remember seeing during earlier browsing, it's easy to write a function like we need:

     ;;; This code expects a string of directories.  Each one
     ;;; (including the last) should be terminated by a single
     ;;; space, in the style of the csh(1) "dirs" command.  That
     ;;; given, it's straightforward to iterate down the string,
     ;;; extracting substrings at each space and pushing them on
     ;;; a list.  That list (which is now in reverse order) should
     ;;; have substitute-in-file-name and expand-file-name applied
     ;;; to each element, to get a list of absolute pathnames
     ;;; without metacharacters.

     (defun parse-and-expand-directories (shell-directory-stack)
       (let ((result nil))
         (let ((first-char 0)		; first character of directory.
               (pos 0)			; current position rover.
               (len (length shell-directory-stack)))	; Stop here.

           (while (< pos len)
             (cond ((eql (elt shell-directory-stack pos) ?\ )
                    (setq result
                          (cons (substring
     			    shell-directory-stack first-char pos)
                                result))
                    (setq first-char (+ pos 1))))
             (setq pos (+ 1 pos))))
         (mapcar (function (lambda (directory)
                             (expand-file-name
                               (substitute-in-file-name directory))))
                 (nreverse result))))

We install this function with C-M-x, as we did with shell-send-input. We then test it by switching to the *scratch* buffer, which is a Lisp-Interaction mode buffer, and typing the following, followed by a LFD.

     (parse-and-expand-directories "~ ~wombat /lisp/bin /tmp/$USER ")

The result looks correct:

     ("/mnt/marick" "/mnt/wombat" "/lisp/bin" "/tmp/marick")

Back to the Main Problem:
-------------------------

We're now ready to write the new version of shell-send-input.  One remaining question is how to detect when the shell signals an error. Note that we depend on a successful directory-change sending back a blank-terminated line.  A couple of simple tests show that error messages generally are not blank terminated.  So we'll let the presence or absence of a trailing blank determine whether an error has occurred.  (This is, of course, somewhat shaky design---if it proves inadequate, we'll do something else.)

After some debugging with a shell window, we arrive at this function:

     ;;; New version of shell-send-input.  The old version could get
     ;;; confused about the current working directory.  This version
     ;;; expects directory-changing-commands to print a
     ;;; ``dirs(1)''-style message after finishing.  This function
     ;;;  then reads that line, parses it, and installs it as the
     ;;; value of shell-directory-stack.
     ;;; Things to know:
     ;;; 1.  We detect errors in the directory-changing command by
     ;;; looking for a missing space at the end of the output line.
     ;;; Dirs puts one there, error messages don't generally have one.
     ;;; This is shaky.
     ;;; 2.  There's really no point to having a shell-directory-stack
     ;;; any more, nor to having three different directory-changing
     ;;; regexps.  These are retained for compatibility, also in case
     ;;; there's a use for them someday.

     (defun shell-send-input ()
       "Send input to subshell.
     At end of buffer, sends all text after last output
      as input to the subshell, including a newline inserted at the end.
     Not at end, copies current line to the end of the buffer and sends it,
     after first attempting to discard any prompt at the beginning of the line
     by matching the regexp that is the value of shell-prompt-pattern if
     possible.  This regexp should start with \"^\"."
       (interactive)
       (end-of-line)
       (if (eobp)
           (progn
             (move-marker last-input-start
                          (process-mark (get-buffer-process (current-buffer))))
             (insert ?\n)
             (move-marker last-input-end (point)))
         (beginning-of-line)
         (re-search-forward shell-prompt-pattern nil t)
         (let ((copy (buffer-substring (point)
                                       (progn (forward-line 1) (point)))))
           (goto-char (point-max))
           (move-marker last-input-start (point))
           (insert copy)
           (move-marker last-input-end (point))))
       (let ((directory-change-possible
                (save-excursion
                  (progn
                    (goto-char last-input-start)
                    (or (looking-at shell-popd-regexp)
                        (looking-at shell-pushd-regexp)
                        (looking-at shell-cd-regexp))))))
           ;; Now send the command to the shell.  If a directory change is
           ;; possible, wait for a reply.
           (let ((process (get-buffer-process (current-buffer))))
             (send-region process last-input-start last-input-end)
             (set-marker (process-mark process) (point))
             (if directory-change-possible
                 (accept-process-output process)))

           ;; If we might have changed the directory, parse the shell's answer.
           (if directory-change-possible
               (let ((shell-answer
                      (save-excursion
                        (buffer-substring last-input-end
                                          (progn
                                            (goto-char last-input-end)
                                            (skip-chars-forward "^\n")
                                            (point))))))
                 ;; Test to see whether we've been given a directory list or
                 ;; an error message.  Determined by whether there's a trailing
                 ;; blank on the line.  If no error, set up the
                 ;; shell-directory-stack.
                 (cond ((eql (elt shell-answer (- (length shell-answer) 1))
                             ?\ )
                        (setq shell-directory-stack
                              (parse-and-expand-directories shell-answer))
                        (cd (car shell-directory-stack))
                        (setq shell-directory-stack
                              (cdr shell-directory-stack))))))))

Test shell-send-input.  Notice that shell-send-input displays the new directory on the message line.  When you try to change to a nonsense directory, it displays nothing.  You can see that the Emacs buffer's directory is still the same as the shell's by typing dirs to the shell and typing

     C-h v default-directory RET

to Emacs.

When you're finished testing, place the functions parse-and-expand-directories and shell-send-input into the file custom-shell.el, compile it with

     M-x byte-compile-file RET

load it with

     M-x load-library RET custom-shell RET

and try it out some more.  It still works.

Now we need only install the file in some standard directory and find a way to load it.  The easiest thing to do is just load shell and custom-shell in our .emacs file.  (Slightly better is to just load custom-shell and have custom-shell.el include the line

     (require 'shell)

which causes the loading of shell.elc.)

Alternatively, we can only load custom-shell.el when a shell process is started.  This is done with the following shell-mode-hook:

     (setq shell-mode-hook
           '(lambda ()
              (require 'custom-shell)))

We use require instead of load because the shell-mode-hook is executed whenever we switch to the shell buffer with

     M-x shell

We don't want to reload it quite so often.  Note that we must put a

     (provide 'custom-shell)

in custom-shell.el.

Exercises
=========

 1. There is a flaw in the shell code.  Suppose, for example, that I have b aliased to popd.  What happens when I type

          bm <pattern> <some-enormous-set-of-files>

     (bm is a fast string-search program.)

     You will see that Emacs waits for the command to complete, thinking it's a directory-changing command.  Fix this problem.

 2. What happens if there's extra white space between the prompt and the command?  Fix this problem.

 3. Find a better way to detect shell error messages.

 4. Our implementation assumes that accept-process-input will return a complete line.  What if it doesn't?

 5. Now that you're familiar with the shell command, give users the ability to easily create multiple shell buffers.  (For example, the shell command might take a prefix argument, or it might prompt for the name of a shell.)

A General Problem
=================

Now that you've finished this customization, what do you do with it?  I think that's a general problem: often you want to modify particular functions within a standard Emacs Lisp file, but you don't want to change that file.  Neither do you want to copy the entire standard file, since that would isolate you from further changes to it, changes that might, for example, fix bugs.

A solution might be to have the new file shadow the old one in the following way:

 1. Put the new file in a new directory.  Give it the same name as the shadowed file.  It will now be loaded in preference to the old file if you have the new directory early in the load-path.

 2. To avoid having to include all of the old file in the new one, arrange for the new file to load the old file as it itself is being loaded.

Dick King has suggested another way to accomplish much the same thing. He suggests that defun come in two versions, aggressive and passive.  Aggressive defuns behave as defun does now. (That is, a new defun overrides an old one with the same name.)  Passive defuns, when read from a file, would not override old definitions.

Shadowing changes could be made with aggressive defuns. Thus, they would be immune to load order---we would not have to carefully arrange things so that our version of shell-send-input is loaded only after all of shell.elc is loaded.