Have you heard about literate programming? It's a technique that lets you write a computer program the same way as if you were writing a piece of literature. And while it may seem exotic and unnecessary, there are some cases when it's quite useful. I believe I have found one of them. In this post, I will describe a solution for writing a tutorial in the spirit of literate programming by using nodejs-based, language-agnostic, literate programming tool.
Background
Literate programming is a technique of writing computer programs in natural (spoken) language. When using this technique, documentation receives most of programmers attention while code is treated as a formality. Code snippets are typically embedded in the essay and get extracted for compilation during project build.
The main idea is to regard a program as a communication to human beings rather than as a set of instructions to a computer.
~ The CWEB System of Structured Documentation by Donald E. Knuth and Silvio Levy
First literate programming tool, developed by Donald Knuth, was called WEB and worked with Pascal only. Short after, Knuth developed CWEB, which is WEB for C language. In response to WEB, Norman Ramsey created noweb—a literate programming tool compatible with any programming language. It started a language-agnosticity trend in the field.
Research
So, I was faced with a task of writing a tutorial on using hyper-text-slider with node and browserify. Since I have this habit of making my life more exciting, and since writing a tutorial is an inherently dull task, I started thinking. From this thinking emerged a question—Is it possible to automatically test a tutorial?
After few minutes of Googling around, I had my answer—literate programming.
HyperText Slider is a web slideshow component that utilizes CSS3 transitions to control background and foreground animations of changes between slides. It has declarative (HTML-only) API which is very easy to use, and JavaScript API for fine-grained control. Using hyper‑text‑slider with node and browserify means using a node-base build system for front-end technology stack (HTML, CSS and JavaScript). I needed a literate programming solution that supports front-end stuff and has a plugin for gulp (my build utility of choice for node-based projects).
As it turns out, number of literate programming solutions for JavaScript in suprisingly big (npm search literate gives 288 results).
I really disliked docco which seemed to be the most popular one.
It was advertized as quick and dirty yet it was still too complicated for my taste.
After trying a few, I ended up choosing writ.
It does only one thing—extracts any code from Markdown
(with very basic support for macro-definitions).
Solution
- Documentation (tutorial), human-written, contains snippets of code needed for a website to be built. It is a single source of information (actual sources). Tutorial is written in markdown and hosted on Github which automatically renders markdown files into nice HTML documents.
- Code snippets are extracted from the documentation by invoking writ from a gulp script. Writ produces one output file for each markdown file, so the tutorial must have been divided into multiple pages—3 source files (HTML, Sass and JavaScript), a shell script that sets up the project, and a gulpfile that builds generated sources. There actually are 2 gulpfiles in the project. First that invokes writ (human-written), second that builds the project (generated). Extracted JavaScript and Sass code is linted by invoking linter plugins after writ. All generated sources are checked into the source control so that the project can be cloned and built.
- Project is built by invoking gulpfile script which was extracted from the documentation. Building process produces browser-runnable files (HTML, CSS, JavaScript), which are also checked into the source control. Github Pages is active with master branch set as source so that compiled project can be run in the browser.
This way, I'm killing three birds with one stone. The project is:
- A tutorial on creating a web slideshow using hyper-text-slider, node and browserify,
- An example project which can be cloned and compiled,
- A website which shows live results of the compilation.
Configuring Writ
There are two gulpfiles in the project. Generated one has the default name (gulpfile.js).
Handwritten one must be named differently (.writ.gulpfile.js).
NOTE
Below paragraphs require basic understanding of gulp. If you're not familiar with it, please read Build Configuration chapter from the tutorial. If you know gulp, it may still be useful to read it as it contains soure code of generated gulpfile.
We need gulp, gulp-util, gulp-writ, gulp-rename2, gulp-eslint, gulp-stylelint,
merge-stream and del modules (they must be installed with npm install in order to make
this gulpfile work).
'use strict';
var gulp = require('gulp');
var gutil = require('gulp-util');
var writ = require('gulp-writ');
var rename = require('gulp-rename2');
var eslint = require('gulp-eslint');
var stylelint = require('gulp-stylelint');
var merge = require('merge-stream');
var del = require('del');
var child = require('child_process');
The project contains following folders:
doc/containing the tutorial,src/containing generated sources,dist/containing build output files (browser-comsumable).
As it will be used as a dependency of generate, clean task must be defined first.
Deleting files with del module requires using gulp in asynchronuous mode
(task function has a callback which is called by del).
We want to delete all files generated in previous run of writ which
are contained in src folder, but without deleting the folder itself.
gulp.task('clean', function(callback) {
return del([ 'src/**/*', '!src/' ], { force: true }, callback);
});
The tutorial consists of following files:
doc/0_introduction.markdown,doc/1_setup.sh.md,doc/2_gulpfile.js.md,doc/3_script.js.md,doc/4_index.html.md,doc/5_style.scss.md.
First file (0_introduction.markdown) does not contain any code snippets
so it has different extension. This way, it is still properly rendered on Github
yet not catched by writ's glob pattern.
Each file is prefixed with a number which indicates reading order of the tutorial.
Numbers also keep the files properly ordered when listed on Github.
gulp-writ plugin removes .md extension but we need to use rename2 in order
to remove number prefixes. setup.sh and gulpfile.js will be placed in project main folder while all generated sources will be placed in /src.
function removeNumberPrefix(pathObj, filePath) {
return pathObj.dirname(filePath) +'/'+ pathObj.basename(filePath).replace(/^[0-9]+_/, '');
}
gulp.task('generate', [ 'clean' ], function() {
var sources = gulp.src([ 'doc/*.md', '!doc/*.sh.md', '!doc/*gulpfile.js.md' ])
.pipe(writ().on('error', gutil.log))
.pipe(rename(removeNumberPrefix))
.pipe(gulp.dest('src/'))
;
var setupAndConfig = gulp.src([ 'doc/*.sh.md', 'doc/*gulpfile.js.md' ])
.pipe(writ().on('error', gutil.log))
.pipe(rename(removeNumberPrefix))
.pipe(gulp.dest('.'))
;
return merge(sources, setupAndConfig);
});
Generated sources are ready to be linted.
gulp.task('lint:sass', [ 'generate' ], function() {
return gulp.src([ 'src/*.sass' ])
.pipe(stylelint({
reporters: [ { formatter: 'string', console: true } ],
}))
;
});
gulp.task('lint:javascript', [ 'generate' ], function() {
return gulp.src([ '*.js', 'src/*.js' ])
.pipe(eslint())
.pipe(eslint.format())
.pipe(eslint.failAfterError())
;
});
gulp.task('lint', [ 'lint:sass', 'lint:javascript' ]);
Linted sources are ready to be built. In case of this tutorial, building the project
means invoking another gulp process with generated gulpfile.
gulp.task('dist', [ 'lint' ], function() {
child.execSync('gulp');
});
Running Writ
Invoking generate task extracts sources from the tutorial.
gulp --gulpfile=.writ.gulpfile.js generate
Invoking dist task extracts sources, lints them and builds the project.
gulp --gulpfile=.writ.gulpfile.js dist
Above command is equal to invoking lint task and then running gulp with generated gulpfile.
gulp --gulpfile=.writ.gulpfile.js list
gulp
Invoking dist task after changing the documentation will propagate changes
into src/ and dist/ folders. It should be invoked before each commit.
Wrap Up
After setting everything up, all that remains is writing the tutorial. It will be automatically tested in a sense that contained code snippets will be linted and checked if they compile successfully. Resulting program must still be run by a human, but it's much less work than manually testing each step of the tutorial.
Implementing a watch task would add more automation into the process
It's there in my tutorial's .writ.gulpfile.js,
but as it's not essential, I haven't include it this blogpost.
I must say that writing a tutorial using literate programming was an exciting experience. Writ is a great, simple tool for literate programing in Markdown. I really recommend using it. I'm curious of your opinion on literate programming and writ. If you have any thoughts to share or questions to ask, please do in the comments below.
References
- literateprogramming.com is a website that gathers literate-programming-related white paper fragments,
- In his original paper on literate programming, Donald Knuth describes first attempt of implementing this concept in a framework called WEB,
- Literate Programming—a book by Donald Kunth—contains a collection of his papers on this technique alongside some new material,
- Cunningham & Cunningham Wiki contains an interesting discussion on literate programming.
- Haskell Wiki contains information on how to do literate programming by using only Haskell's native features,
- Github Pages Help website contains information on how to set a source branch for Github Pages in your project,
- Tutorial described in this blogpost is hosted on github.