Literate Programming (lp) Syntax

Normal Form

lp blocks are special fenced code blocks within a markdown page and look like so:

LP Source:

 ```bash lp addsrc fmt=mk_console
 echo "Hello ${User:-World}"
 ```

Result:

$ echo "Hello ${User:-World}"
Hello World

or more general:

 ```<lang> lp[:mode] [block level (=header) parameters]  <---- lp block header
 statement 1 [per statement parameters]          
 (...)                                                   <---- lp block body
 [statement n]                                  
 ```

• mode: The mode parameter determines the block processing mode. Default is shell mode ("bash") may also be supplied via a keyword parameter:
  lang lp:foo equal to lang lp mode=foo.

• lang: The word before "lp" is for Markdown not for LP. It is declaring the code block formatting language, and is not relevant for LP itself.

Short Form

There is a short form, for blocks without a body:

 `lp:<mode> [parameters]`

In the short form the mode must be added to lp:. See below for more about the short form.

Detection

• We parse the markdown source line by line.
• We have a state variable fenced, initially False.

Normal Form

Block Start

1. fenced is False
2. A line must start with n spaces ( n ≥ 0 )
3. Followed by 3 backticks. fenced is set to True.
4. Followed by a word (any letter except space), the formatting language
5. Followed by "lp" then a space OR "lp:" then a word, the mode parameter

All following lines are the block body - until:

Block End

1. fenced is True
2. A line starts with m spaces ( m = n )
3. Followed by 3 backticks
4. Followed by any number of spaces. fenced is set to False

Example

An Admonition With An Inner LP Block

LP Source:

 ```bash lp addsrc
 echo "hello world"
 ```

Result:

CmdOutput

$ echo "hello world"

$ echo "hello world"
hello world

Summary

In general this means that LP blocks look like normal fenced code blocks except:

• The first (header) line has more parameters than just the formatting language
• They are also detected within HTML, if the criteria are met:

HTML Example:

 <div style="color:gray">
 ```_ lp:python
 import time; now = time.ctime(); show(f"Hello from python, at <b>{now}</b>!")
 ```
 </div>

Result:

Hello from python, at Mon Apr 11 14:55:32 2022!

See here regarding the show function.

Short Form

1. fenced is False
2. A line must start with n spaces ( n ≥ 0 )
3. Followed by 1 backtick
4. Followed by lp:
5. Followed by k characters except space, with k ≥ 1 )
6. The line ends with a backtick, followed by any number of spaces.

Parametrization

Evaluation is parametrized by keys and values, which may be given via the environment, per page, per block and per statement, with priority in this order.

Here is the list of supported parameters, for the default mode: bash.

Environment

A parameter foo may be set to value bar for all lp blocks within your docs set like so:

LP_foo=bar mkdocs build
export LP_foo=bar; mkdocs build 

# and for convenience also the lower case form:
lp_foo=bar mkdocs build 

Page Level

You may specify a parameter for all blocks within a specific markdown page like so:

Normal Form:

 ```my_lang lp:page foo=bar # or: ```my_lang lp mode=page foo=bar
 ```

Short Form:

`lp:page foo=bar`

i.e. by specifying the keyword "page" as mode

Position matters

The parameters in mode=page headers are only valid for all lp blocks below the declaration.

• You may set to a different value, mid-page.
• You may use more than one page block.

Block Level

This sets the value for just one block:

 ```bash lp foo=bar 
 echo Hello
 ```

Statement Level

This set the value for just one statement:

 ```bash lp
 echo Hello # lp: foo=bar
 echo World # while executing this, foo is NOT set (except when defined elsewhere)
 ```

Parameter Syntax

Parameters may either be delivered python signature compliant or "easy args" style:

 # easy args (=true is default):
 ```bash lp foo=bar f=1.23 foo=true bar asserts="foo and bar"

 # python signature compliant form:
 ```bash lp foo='bar', f=1.23, foo=True, bar=True, asserts="foo and bar"

When easy args parsing fails, then python signature mode is tried.

Easy Args Conventions and Restrictions

• No spaces in key=value allowed for easy args, except when value between double or single quotes (" or ').
• mykey allowed, identical to mykey=true
• Casting canonical: 1.123 considered float, 42 considered int, true considered bool, else string

Presets

These keywords will be dynamically replaced within headers:

key	value
dir_repo	Set to directory of the repository
dir_project	Set to project root directory

Examples:

    ```bash lp session=project    cwd=dir_repo # easy args style
    ```bash lp session='project', cwd=dir_repo # sig args style

Available Environment Variables

If you want to work with/generate assets relative to your docu, these should be practical:

$ env | grep LP_ # any env var starting with lp_ or LP_ is put into the session
LP_DOCU_FILE=/home/runner/work/docutools/docutools/docs/features/lp/syntax.md
LP_DOCU_ROOT=/home/runner/work/docutools/docutools/docs
LP_PROJECT_ROOT=/home/runner/work/docutools/docutools
LP_DOCU_DIR=/home/runner/work/docutools/docutools/docs/features/lp

These are also put into new tmux sessions (at new_session):

$ env | grep LP_                                      
LP_DOCU_FILE=/home/runner/work/docutools/docutools/docs/features/lp/syntax.md   
LP_DOCU_ROOT=/home/runner/work/docutools/docutools/docs                         
LP_PROJECT_ROOT=/home/runner/work/docutools/docutools                           
LP_DOCU_DIR=/home/runner/work/docutools/docutools/docs/features/lp
$ env | grep TMUX                                     
TMUX=/tmp/tmux-1001/default,2649,24     
TMUX_PANE=%24

You can reference any env var as a "dollar var" within your header args.

Short Form for LP Plugins Without Body

Some plugins do not need a body to evaluate. Then you can also express lp blocks via the short form:

 `lp:<mode(plugin name)> [header params like normal]`

Example:

 `lp:lightbox match=img`

Must be within separate lines

The short form can not work as inline statement (i.e. between other words in a line). You (still) need to have exclusively one statement per line.

If you do have a (short) body, you may supply it via the body parameter:

 `lp:python body=print(42)`

Evaluation

Statements

Within an lp block can be more than one statement contained. We run the statements consecutively.

Staments can be given per line or a in a structured way:

 ```bash lp
 echo hello # lp: asserts=hello
 echo world
 ```

is equivalent to

 ```bash lp
 [{'cmd': 'echo hello', 'asserts': 'hello'},
  {'cmd': 'echo world'}]
 ```

Hint

In structured mode you can omit the list notation, if there is just one command and simply supply a dict with cmd and parameters.

Multiline Commands / Here Docs

Here Docs come handy when you have dynamic variable from a previous command, which you need to set into a file.

Like this, i.e. use the '> ' indicator at beginning of lines, which should be run as single commands.

LP Source:

 ```bash lp addsrc session=DO asserts=foobarbazXsecond_line
 foo () { echo bar; }
 cat << EOF > test.pyc
 > foo$(foo)baz
 > second_line
 > EOF
 cat test.pyc | sed -z "s/\n/X/g" | grep 'foobarbazXsecond_line'
 ```

Result:

CmdOutput

$ foo () { echo bar; }
$ cat << EOF > test.pyc
foo$(foo)baz
second_line
EOF
$ cat test.pyc | sed -z "s/\n/X/g" | grep 'foobarbazXsecond_line'

$ foo () { echo bar; }
$ cat << EOF > test.pyc                 
> foo$(foo)baz      
> second_line       
> EOF               
$ 
$ cat test.pyc | sed -z "s/\n/X/g" | grep 'foobarbazXsecond_line'         
foobarbazXsecond_lineX

Picky Syntax

• The second space is requred after the '>'
• No lines with content may start with a space

Special Statements

• wait 1.2: Causing the process to sleep that long w/o logging (tmux only). You can attach and check output.
• send-keys: C-c: Sends a tmux key combination (here Ctrl-C).

Exceptions

On Execution Exit Codes

Whenever a statement fails, we stop evaluation and except.

Hint

If you want to continue even when a statement fails, than add the usual || true after the statement.

Example

CmdOutput

$ ls /notexisting || true # would except here without the `|| true`
$ ls /etc |grep hosts

$ ls /notexisting || true # would except here without the `|| true`
ls: cannot access '/notexisting': No such file or directory
$ ls /etc |grep hosts
hosts
hosts.allow
hosts.deny

On Timeouts

In session mode, we expect commands to complete within a given time, otherwise we raise.

On Assertions

We can also force normally exitting commands to fail, when certain strings are not occuring in the result, see the asserts parameter. That way you can turn lp into a stateful (sessions) or stateless functional testsuite.