~/bin: apod-set-bg

Astronomy Picture Of the Day shows a new, usually desktop sized, picture every day. apod-set-bg is a shell script to download the daily image and set it as the desktop background. It also downloads the explanation as a separate html file.

Requirements: curl, tidy, xmlstarlet, feh.

#!/bin/sh

IMG_DIR="$1"

APOD_URL=http://apod.nasa.gov/apod/ 

URL_CONTENT=`curl $APOD_URL | tidy -q -asxml`

IMG_URL=$APOD_URL`echo "$URL_CONTENT" | xml sel -N html='http://www.w3.org/1999/xhtml' -t -v '//html:img/ancestor::html:a/attribute::href' -`

EXPLANATION=`echo "$URL_CONTENT" | xml sel -N html='http://www.w3.org/1999/xhtml' -t -c '/html:html/html:body/html:center[2]' -c '/html:html/html:body/html:p' -`

FILE_NAME=`basename "$IMG_URL"`

FILE_PATH="$IMG_DIR/$FILE_NAME"

curl -o "$FILE_PATH" "$IMG_URL" || exit 1

echo "$EXPLANATION" > "$FILE_PATH.html"

ln -sf "$FILE_PATH" "$IMG_DIR/latest.jpg"
ln -sf "$FILE_PATH.html" "$IMG_DIR/latest.html"

feh --bg-fill "$FILE_PATH"

Put the following in your crontab to run daily.

@daily ID=apod DISPLAY=:0.0 ~/bin/apod-set-bg ~/bg/ >/dev/null 2>&1

tvnz-grab in factor

The previous post in factor. I wrote it so my Windows using friends could have a single-binary solution. Download the zip archive and unzip somewhere in the windows path; C:\windows\system32 will do.

Usage: tvnz-grab <episode-page-url>. It will download all parts of the episode into the current directory.

Unfortunately, this only works for NZ content. Foreign content uses Adobe’s encrypted RTMP protocol. To get at episodes using that, check out rtmpsuck.

Imports first.

! Copyright (C) 2010 Jeremy Hughes.
! See http://factorcode.org/license.txt for BSD license.
USING: accessors assocs combinators.short-circuit fry
http.client io.streams.byte-array kernel namespaces make
regexp sequences xml xml.data locals splitting strings
io.encodings.binary io io.files command-line http system
math.parser destructors math math.functions io.pathnames
continuations xml.traversal ;
IN: tvnz-grab

Because I intend to extend this program into a small Qt demo, it is necessary that any words used to display UI information, dispatch on the type of UI.

SYMBOL: ui
SINGLETON: text

And the display methods themselves…

HOOK: show-progress ui ( chunk full -- )
HOOK: show-begin-fetch ui ( url -- )
HOOK: show-end-fetch ui ( -- )
HOOK: show-page-fetch ui ( -- )
HOOK: show-playlist ui ( seq -- )
HOOK: show-fatal-error ui ( error -- )

M\ text show-progress uses the symbols bytes and count to keep track of the number of bytes downloaded and the proportion of progress-bar fill respectively.

: print-bar ( full chunk -- )
    count [
        [ swap / 50 * round ] dip [
            - CHAR: =
             >string write
        ] [ drop ] 2bi
    ] change ;

M: text show-progress
    swap bytes [ + [ print-bar ] keep ] change flush ;

M: text show-begin-fetch
    "Fetching " write print "[" write flush ;

M: text show-end-fetch
    "]" print flush ;

M: text show-page-fetch
    "Fetching TVNZ page..." print flush ;

M: text show-playlist
    length "Found " write number>string write " parts." print
    flush ;

M: text show-fatal-error
    dup string? [ print ]
    [ drop "Oops! Something went wrong." print ] if 1 exit ;

Failed HTTP request errors need to be wrapped in a user friendly explanation.

: wrap-failed-request ( err -- * )
    [
        "HTTP request failed: " % [ message>> % ]
        [ " (" % code>> number>string % ")" % ] bi
    ] "" make throw ;

The playlist parameter in each episode’s web page is in a section of code looking like this.

var videoVars = {
    playlist: '/content/3685181/ta_ent_smil_skin.smil',
    config: '/fmsconfig.xml',
    ord: ord
};

Given this code could change unpredictably, we’ll use nothing more robust than a regular expression to get at the playlist path.

: get-playlist ( url -- data )
    http-get [ check-response drop ]
    [ R/ (?<=playlist: ').*(?=')/ first-match ] bi* [
        "http://tvnz.co.nz" prepend http-get [
            [ check-response drop ]
            [ wrap-failed-request ] recover
        ] dip
    ] [ "Could not find playlist at address." throw ] if* ;

The playlist is an XML file of which only the video elements concern us.

<video src="path-to-segment.flv" systemBitrate="700000"/>

700000 appears to be the highest bit rate so that is what we’ll go for.

parse-playlist takes the output of get-playlist and returns a list of URLs to episode segments.

: parse-playlist ( data -- urls )
    bytes>xml body>> "video" "700000" "systemBitrate"
    deep-tags-named-with-attr
    [ [ drop "src" ] [ attrs>> ] bi at ] map [ ] filter ;

Each segment is downloaded in chunks.

: call-progress ( data -- )
    length response get check-response
    "content-length" header string>number show-progress ;

: process-chunk ( data stream -- )
    [ stream-write ] [ drop call-progress ] 2bi ;

: get-video-segment ( url -- )
    [ show-begin-fetch ] [ ]
    [ part-name binary  ] tri
    [ '[ _ process-chunk ] with-http-get drop flush ]
    with-disposal show-end-fetch ;

: get-video-segments ( urls -- )
    [ get-video-segment ] each ;

grab-episode is where the action starts.

: (grab-episode) ( url -- )
    show-page-fetch get-playlist parse-playlist dup
    show-playlist [
        0 bytes count [ set ] bi-curry@ bi get-video-segments
    ] with-scope ;

: grab-episode ( url -- )
    [ (grab-episode) ] [ nip show-fatal-error ] recover ;

For the complete program see my github.

Saving TVNZ “ondemand” episodes

The code in my previous post will work for episodes from TVNZ’s ondemand service; but there is a more direct approach.

Each episode page stores a URL to the episode’s playlist in a javascript property "playlist". The URL points to an XML file containing the URLs of four or more video parts and no ads. The following script downloads the episode page, grabs the playlist URL, downloads the playlist, and parses it. It returns a newline separated list of video URLs.

You will need to have curl and xmlstarlet installed.

#!/bin/sh

if [ "$1" == "" ]; then
    echo "usage: $0 episode-url"
    exit 1
fi

EPISODE="$1"

PLAYLIST=http://tvnz.co.nz`curl "$EPISODE" | grep -Po "(?<=playlist: ').*?(?=')"`

curl "$PLAYLIST" | \
    xml sel -N smil=http://www.w3.org/ns/SMIL \
        -t -m "//smil:video[@systemBitrate='700000']" -v '@src' -n -

I have saved it as tvnz-grab and call it like so:

tvnz-grab <episode-page-url> | xargs curl -O

The four or five parts can be played as a single file using mplayer’s "-fixed-vo" option.

Saving flash video on linux

On linux, the flash-player caches video in files under /tmp. Each video gets a file called /tmp/Flash[random-string]. These are easy enough to find and rename to [meaningful-string].flv. However, some players delete these files on completion, which is a pain given they are the files containing the video we want to save.

Hard linking these files solves the problem. The Flashxxx files will be deleted by the player, but their contents will sill be available at whatever location we made the hard link.

Some players split their content into multiple files, so we want this hard linking process to be automatic.

I have the following script saved as lnflv in my ~/bin.

#!/bin/sh

cd /tmp
for x in Flash*; do
    count=`find . -xdev -samefile "$x" | wc -l`
    if [ $count -gt 1 ]; then
        echo "Not linking $x. Already linked."
    else
        echo "Linking $x."
        ln "$x" "dl-$x.flv"
    fi
done

Run it every thirty seconds like this: watch -n30 lnflv. Each flash file not already hard linked will be linked to /tmp/dl-Flashxxx.flv.

Order by time and rename video parts with something like this:

alpha=a
for f in `find . -size +10M -name dl-Flash\* | xargs ls -tr`; do
    cp "$f" [name]$alpha.flv
    alpha=`echo -n $alpha | tr 'a-y' 'b-z'`
done

Change [name] to the name of whatever you are watching. "-size +10M" filters out files smaller than 10MB; this screens out short ads and is only required for video streams that embed them.

Old blog: Map in Java

(Found in my years old defunct blog. Less relevant now that Java is getting lambdas.)

Scheme

(map fn lst ...)

;;  ==> list
      

map is a higher order function that applies the function fn of n arguments to each element in the lists lst ...

Java has no map. Enhanced for isn't adequate for the following reasons:

• No return value: Since for is a control structure, not a function, it doesn't return anything useful. This means a data structure has to be created beforehand, and added to at each iteration of the for loop.
• No function arguments: This is not for's problem. Functions (methods) are not first class in Java.
• One iterable only: The enhanced for loop can only iterate over one Iterable data structure. map can operate on any number of lists.

Examples

Simple one list mapping

Scheme

(define squares (list 1 4 9 16))

(define (sqrts nums)
  (map sqrt nums))

(sqrts squares)

;;  ==> (1 2 3 4)
      

Java

/* No generics for clarity.  Assume also the existance of a static
   method `toList' that takes an Array and returns an ArrayList. */

List squares = toList(new Integer[] {1, 4, 9, 16});

List sqrts(List nums) {
    List out = new ArrayList();
    for (Integer i : nums)
        out.add(Math.sqrt(i));
    return out;
}

sqrts(squares);

//  ==> [1, 2, 3, 4]
      

Multiple list mapping

Scheme

(define even (list 2 4 6 8))
(define odd (list 1 3 5 7))

(define (sums . lists)
  (apply map + lists))

(sums even odd)

;;  ==> (3 7 11 15)
      

Java

/* Assume the existance of a static method `isSameLength' that tests
   whether all Lists in an array are the same length. */

List even = toList(new Integer[] {2, 4, 6, 8});
List odd = toList(new Integer[] {1, 3, 5, 7});

List sums(List... lists) {
    List out = new ArrayList();
    if (!isSameLength(lists))
        throw new RuntimeException("Lists must have same length.");
    int numargs = lists.length;
    int length = lists[0].size();
    for (int i = 0; i < length; i++) {
        int val = 0;
        for (int k = 0; k < numargs; k++)
            val += lists[k].get(i);
        out.add(val);
    }
    return out;
}

sums(even, odd);

//  ==> [3, 7, 11, 15]
      

The problem with this is that it isn't generalised.

Map in Java

First we'll need a something that acts as a first-class function.

Java

public interface Proc {

    public Object apply(Object... args);

}
      

Next, a static map method.

Java

public static List map(Proc proc, List... lists) {
    List out = new ArrayList();
    if (!isSameLength(lists))
        throw new RuntimeException("`map' requires lists of identical length");
    int numargs = lists.length;
    int length = lists[0].size();
    for (int i = 0; i < length; i++) {
        Obj[] args = new Obj[numargs];
        for (int k = 0; k < numargs; k++)
            args[k] = lists[k].get(i);
        out.add(proc.apply(args));
    }
    return out;
}
      

Now, the multiple lists example again.

Java

List even = toList(new Integer[] {2, 4, 6, 8});
List odd = toList(new Integer[] {1, 3, 5, 7});

Proc sum = new Proc() {
    public Object apply(Object... args) {
        return (Integer)args[0] + (Integer)args[1];
    }
};

List sums(List... lists) {
    return map(sum, lists);
}

sums(even, odd);

//  ==> [3, 7, 11, 15]
      

Much shorter, but with some limitations:

• It doesn't use generics, and therefore
• requires casting in Proc implementations; and
• it is hardcoded to return an ArrayList.

Parametrized map

First, Proc is changed.

Java

public interface Proc1<R, A> {

    public R apply(A arg);

}

public interface Proc2<R, A, B> {

    public R apply(A argA, B argB);

}

public interface ProcN<R, A, B,..., N> {

    public R apply(A argA, B argB,..., N argN);

}
      

And a parameterized map.

Java

public static <R, A, X extends List<R>> X map(Proc1<R, A> proc, X out, List<A> list) {
    for (A a : list)
        out.add(proc.apply(a));
    return out;
}

public static <R, A, B, X extends List<R>> X map(Proc2<R, A, B> proc,
        X out, List<A> listA, List<B> listB) {
    if (!isSameLength(listA, listB))
        throw new RuntimeException("`map' requires lists of identical length.");
    int length = listA.size();
    for (int i = 0; i < length; i ++)
        out.add(proc.apply(listA.get(i), listB.get(i));
    return out;

// And so on up to `map<R, A, B,..., N, X>.
      

The example would now be:

Java

List even = toList(new Integer[] {2, 4, 6, 8});
List odd = toList(new Integer[] {1, 3, 5, 7});

Proc2<Integer, Integer, Integer> sum = 
    new Proc<Integer, Integer, Integer>() {
        public Integer apply(Integer argA, Integer argB) {
            return argA + argB;
        }
    };

List sums(List listA, List listB) {
    return map(sum, new ArrayList(), listA, listB);
}

sums(even, odd);

//  ==> [3, 7, 11, 15]
      

The cost of parameterizing

The obvious limitation of the generic implementation is that arbitrary numbers of arguments are no longer supported in ProcN or map.

Reduce

reduce is easily written on top of Proc2.

Java

public static <R, A> R reduce(Proc2<R, R, A> proc, List<A> list, R initial) {
    int size = list.size();
    R out = initial;
    for (A a : list)
        out = fun.apply(out, a);
    return out;
}

List nums = toList(new Integer[] {1, 2, 3, 4});

Integer squareSum(List list) {
    return reduce(new Proc2<Integer, Integer, Integer>() {
        public Integer apply(Integer a, Integer b) {
            return a + (b * b);
        }
    }, list, 0);
}

squareSum(nums);

//  ==> 30
      

Why I like Factor

Factor is a relatively new language and implementation in the tradition of Forth, Lisp, and Smalltalk. Like Forth, Factor is concatenative and uses a postfix syntax. Also like Forth, Factor emphasises small procedures and constant refactoring. Like many Lisp and Smalltalk implementations, Factor compiles code into a loadable image. Like Lisp, Factor can perform arbitrary computation at compile time using macros while parse time evaluation allows the creation of new syntax forms.

Unlike Forth, Lisp, and Smalltalk, Factor is modern and unencumbered. Lisp suffers from an ossified specification, Smalltalk has lived inside its own image for so long it's out of touch with the rest of the world, and Forth provides few of Factor's higher level features like garbage collection and dynamic code updating.

Factor’s focus on correctness and efficiency is not commonly found in other modern ‘dynamic’ languages. The results speak for themselves.

1. Factor vocabularies are compiled to machine code. When using the interactive listener the code is compiled before execution (like SBCL).

2. Factor is fast. Not C fast, but SBCL fast.

3. Almost all structures are modifiable at runtime. This includes classes at any position in the hierarchy as well as FFI bindings. When reloading a modified vocabulary, definitions no longer present will be deleted from the running image. These changes are propagated to all dependent code and the changes to the running image are kept consistent.

4. A simple foreign function interface. No quirky header files or IDL necessary; everything is done from within Factor. No reloading of Factor necessary either: create bindings incrementally and test them as you go.

5. Deployment images. Factor’s deployment tool only loads code the resultant binary will run. Minimal image size for a hello world program is a few hundred kilobytes.

6. Common Lisp style condition system. Don’t let stack unwinding lose an exception’s context. Recover anywhere between where the exception is caught and where it was thrown.

Why having a competent designer pays off

In contrast to other popular languages like Ruby and Python, Factor does not suffer from limiting creeds like, “There should be one way to do it,” or from BDFLs who don’t see the value in useful language constructs discovered as early as the 70s. Here's the payoff...

7. Correct lexical scoping. Usually in Factor one uses its postfix syntax and data flow combinators, however it also sports a locals vocabulary defined entirely in factor which implements lexical scoping and lambdas. Despite the fact that this is an addition in a language where lexical variables are not the default, Factor's lexical scoping is correct and its lambdas are uncrippled. Python 2.x is unable to rebind a variable in a parent scope (3.x overcomes this by introducing a new keyword, ugh!), and Ruby only recently (1.9) gave blocks their own scope.

8. Usable higher order functions. Combinators are used in Factor all the time. For some unfathomable reason Ruby allows only one block argument per function while in Python the use of such esoterica as map and reduce is discouraged.

9. Tail call optimization. Yep, that’s right, that helpful recursion thingy. Another thing Python doesn't have because, oh I don’t know, ask Guido. Come to Factor and free your mind from loopiness!

10. Low, hi, and FFI. Factor scales well from highly abstracted garbage collected code, to micro optimization using inline ASM. Along the way, the FFI allows Factor quotations to be used as C callbacks, and provides factor-side memory allocation should you need to store values on the stack or in a memory pool.

11. Methods are orthogonal to classes. Generic words and their methods are defined outside classes. Adding a generic word to a preexisting class is as simple as defining it. Extensibility without nasty monkey patching or name clashes, and a nice fit with a hierarchy that can contain anything from tuples, to predicate classes and C structs. No hideously bloated base-class needed (I'm looking at you Smalltalk). If Programmer Pooh adds a display-in-opengl method to object he need not modify core code, nor will his method clash with any present or future methods on object.

Why having a smart community pays off

Smart, or at least willing to learn new things.

12. No whiners! Try educating a bunch of Java programmers in the utility of higher order functions or tail-recursion...

13. Macros, syntax, and combinators, oh my! If you don’t whine, you get cool stuff. Macros akin to those in Common Lisp, but hygienic like Scheme, because if you don’t have variables, you can’t capture them. Syntax words like Lisp’s reader macros, but better because no dispatch character craziness is necessary. Observe the usefulness of regular expressions that are syntax checked at parse time, or the EBNF vocabulary which compiles an inline EBNF description into parsing code. For higher order goodness check out Factor’s unique dataflow combinators. This is not your ancestor’s stack shuffling!

More stuff I like

14. Live documentation. Factor has live documentation ala Emacs, but prettier and better linked. The documentation is contained in separate files to the code it describes. For those of us who hate hunting for scratchings of code amid screeds of API comments, this is a good thing.

An acceptable Lisp?

Someone once wrote about Ruby being an acceptable Lisp. They were wrong. Ruby doesn’t have macros, reader macros, native compilation, or a REPL from which everything can be modified. (Oh, but it does have three different kinds of lambda!)

Factor isn’t an acceptable Lisp either. Factor is a mighty fine Lisp. Factor is better than Lisp. It has all the things that make Lisp great and more. Factor will make your code beautiful. Factor will cook you breakfast. Factor reads like English, (lisp (like (not))). All hail Factor!

Summation

I like Factor because it hits the sweet spot of pandering to nothing but being a great language. It lifts much of the good stuff from great languages of yore and gives them an improved spin. It hasn’t yet succumbed to the whining hordes of mediocrity. It is written by a team that really know what they are doing. If that doesn’t describe your language, then give Factor a try.

Multichannel play record with jack

The brief: play a signal through 24 audio ports and simultaneously record from another 24 ports. These ports came from three daisy chained Presonus FP10’s which were connected by firewire to a linux box. This hardware was accessed by the ffado jack drivers. However, none of the following is specific to that hardware setup.

What follows therefore is a walk through of the code to an executable called recapture. Note that the code is under the GPL.

Simultaneous play and record using jack

24 inputs an 24 outputs, simultaneous play and record. The jack audio server is written in C and can be run as a realtime process under a linux kernel with the RT patch applied. There are few bindings to other languages, most likely because of these realtime constraints. C was the most sensible language in which to implement recapture.

Includes and macros

/*
 This program is free software; you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
 the Free Software Foundation; version 2 of the License.

 This program is distributed in the hope that it will be useful,
 but WITHOUT ANY WARRANTY; without even the implied warranty of
 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 GNU General Public License for more details.

 recapture.c 
 2009 Jeremy Hughes

 Play a signal through one port and simultaneously record from another.
*/

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <signal.h>
#include <sndfile.h>
#include <math.h>
#include <pthread.h>
#include <getopt.h>
#include <jack/jack.h>
#include <jack/ringbuffer.h>

Standard includes plus sndfile.h, jack/jack.h and jack/ringbuffer.h. jack.h contains the general API and ringbuffer.h contains jack's implementation of a circular buffer. libsndfile is used for reading and writing audio data.

const char* usage =
 "usage: recapture [ -b bufsize ] [ -i <inports> ] [ -o <outports> ] infile outfile\n"
 "            <inports> and <outports> are `,' separated\n";

#if 1
#define DEBUG(...) (fprintf(stderr, "recapture: "), fprintf(stderr, __VA_ARGS__))
#else
#define DEBUG
#endif
#define MSG(...) (fprintf(stderr, "recapture: "), fprintf(stderr, __VA_ARGS__))
#define ERR(...) (fprintf(stderr, "recapture: error: "), fprintf(stderr, __VA_ARGS__))

Usage notice and some useful macros.

#define MAX_PORTS 30

A sensible number given 24 input and output ports. To be a truly general program this would need to be a variable able to be overridden by a command line argument.

Structs and typedefs

typedef jack_default_audio_sample_t recap_sample_t;

jack_default_audio_sample_t is a little long.

typedef enum _recap_status {
 IDLE, RUNNING, DONE
} recap_status_t;

This program starts two extra threads. The main thread sets up a number of callbacks which the jack process runs in realtime. The two extra threads are a read thread and a write thread. This split is necessary because reading and writing cannot occur within a realtime thread without wrecking its realtime guarantees. The read and write threads are connected to the jack thread by a ringbuffer each.

Before initial synchronization the read thread is IDLE, once reading has commenced it is RUNNING, and when reading is finished it is DONE. The jack processing thread also uses IDLE and DONE.

typedef struct _recap_state {
 volatile int can_play;
 volatile int can_capture;
 volatile int can_read;
 volatile recap_status_t reading;
 volatile recap_status_t playing;
} recap_state_t;

A single instance of this struct is shared among all three threads. Play and record start when can_play, can_capture, and can_read (which correspond to the three threads) are all true; they finish when reading and playing are both DONE.

typedef struct _recap_io_info {
 pthread_t thread_id;
 char* path;
 SNDFILE* file;
 jack_ringbuffer_t* ring;
 long underruns;
 recap_state_t* state;
} recap_io_info_t;

Both read and write threads receive an instance of this struct as their only argument.

typedef struct _recap_process_info {
 long overruns;
 long underruns;
 recap_io_info_t* writer_info;
 recap_io_info_t* reader_info;
 recap_state_t* state;
} recap_process_info_t;

An instance of this struct is passed to the callback responsible for processing the signals.

Global values

const size_t sample_size = sizeof(recap_sample_t);

This works out to be the size of a 32 bit float. Not bad given the sample size of CD audio is 16 bits.

pthread_mutex_t read_lock = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t ready_to_read = PTHREAD_COND_INITIALIZER;
pthread_mutex_t write_lock = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t ready_to_write = PTHREAD_COND_INITIALIZER;

Locks and condition variables for each IO thread.

recap_process_info_t* proc_info;

Needs to be global for signal handling.

/* read only after initialization in main() */
int channel_count_r = 0;
int frame_size_r = 0;
int channel_count_w = 0;
int frame_size_w = 0;
jack_nframes_t ring_size = 65536; /* pow(4, 8) */
jack_port_t* recap_in_ports[MAX_PORTS];
jack_port_t* recap_out_ports[MAX_PORTS];
jack_client_t* client;

Variables that are not changed subsequent to initialization and therefore do not need to be in thread specific structs.

channel_count_r and channel_count_w hold the number of read and write signals. The frame_size of each is calculated by multiplying the channel count by the sample size. The default ring_size can be overridden by command line argument. recap_in_ports and recap_out_ports will hold the jack ports this client connects to.

Helper functions

static size_t array_length(char** array) {
 int i = -1;
 while (array[++i] != NULL);
 return i;
}

There is probably a more standard if not more general way to do this, but a) I couldn't find it, and b) this function does the job.

typedef recap_sample_t (*next_value_fn) (void*);

int uninterleave(recap_sample_t** buffers, size_t length, size_t count,
                next_value_fn next_value, void* arg) {
 int status = 0;
 int i;
 int k;
 for (i = 0; i < count; i++) {
   for (k = 0; k < length; k++) {
     recap_sample_t sample = next_value(arg);
     if (sample == -1) {
       status = -1;
       break;
     } else {
       buffers[k][i] = sample;
     }
   }
   if (status) break;
 }
 return status;
}

recap_sample_t next_value(void* arg) {
 recap_sample_t sample;
 jack_ringbuffer_t* rb = (jack_ringbuffer_t*) arg;
 size_t count = jack_ringbuffer_read(rb, (char*) &sample, sample_size);
 if (count < sample_size) sample = -1;
 return sample;
}

libsndfile expects multichannel data to be interleaved. uninterleave() uses a supplied function to read interleaved data and writes it to an array of output buffers (one per channel).

next_value() simply reads the next sample_size bytes from the supplied ringbuffer.

typedef int (*write_value_fn) (recap_sample_t, void*);

int interleave(recap_sample_t** buffers, size_t length, size_t count,
              write_value_fn write_value, void* arg) {
 int status = 0;
 int i;
 int k;
 for (i = 0; i < count; i++) {
   for (k = 0; k < length; k++) {
     status = write_value(buffers[k][i], arg);
     if (status) break;
   }
   if (status) break;
 }
 return status;
}

int write_value(recap_sample_t sample, void* arg) {
 int status = 0;
 jack_ringbuffer_t* rb = (jack_ringbuffer_t*) arg;
 size_t count = jack_ringbuffer_write(rb, (char*) &sample, sample_size);
 if (count < sample_size) status = -1;
 return status;
}

interleave() reads data from an array of input buffers and uses the supplied function to write it interleaved.

write_value() simple writes sample_size bytes to the supplied ringbuffer.

static void recap_mute(recap_sample_t** buffers, int count, jack_nframes_t nframes) {
 int i;
 size_t bufsize = nframes * sample_size;
 for (i = 0; i < count; i++)
   memset(buffers[i], 0, bufsize);
}

After playing has finished, jack output must be muted (zeroed) otherwise jack will continue playing whatever is left in the output buffers, looping through the last cycle of whatever signal was being played. Definitely not what you want.

Signal handling

static void cancel_process(recap_process_info_t* info) {
 pthread_cancel(info->reader_info->thread_id);
 pthread_cancel(info->writer_info->thread_id);
}

static void signal_handler(int sig) {
 MSG("signal received, exiting ...\n");
 cancel_process(proc_info);
}

static void jack_shutdown(void* arg) {
 recap_process_info_t* info = (recap_process_info_t*) arg;
 MSG("JACK shutdown\n");
 cancel_process(info);
}

Signal handing. cancel_process() ensures things are cleaned up nicely. jack_shutdown() is a callback that the jack process calls on exit.

Thread abstraction

typedef int (*io_thread_fn) (recap_io_info_t*);
typedef void (*cleanup_fn) (void*);

#define FINISHED -1

static void* common_thread(pthread_mutex_t* lock, pthread_cond_t* cond, io_thread_fn fn, cleanup_fn cu, void* arg) {
 int* exit = (int*) malloc(sizeof(int*));
 memset(exit, 0, sizeof(*exit));
 int status = 0;
 recap_io_info_t* info = (recap_io_info_t*) arg;
 pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);
 pthread_cleanup_push(cu, arg);
 pthread_mutex_lock(lock);
 while (1) {
   if ((status = fn(info)) != 0) break;
   pthread_cond_wait(cond, lock);
 }
 pthread_mutex_unlock(lock);
 *exit = status;
 if (status == FINISHED) *exit = 0;
 pthread_exit(exit);
 pthread_cleanup_pop(1);
}

This abstracts out the common parts of setting up a thread and its loop. The supplied io_thread_fn function is executed every iteration until it returns non zero. The supplied cleanup_fn is called whenever the thread is exited. After each iteration the thread waits until it is signalled to continue.

typedef int (*io_test_fn) (recap_io_info_t*);
typedef size_t (*io_size_fn) (recap_io_info_t*);
typedef int (*io_body_fn) (void*, size_t, recap_io_info_t*);

static int io_thread(io_test_fn can_run, io_test_fn is_done, io_size_fn ring_space, io_body_fn body, recap_io_info_t* info) {
 int status = 0;
 if (can_run(info)) {
   size_t space = ring_space(info);
   if (is_done(info)) {
     status = FINISHED;
   } else if (space > 0) {
     void* buf = malloc(space); /* TODO malloc and memset once only */
     status = body(buf, space, info);
     free(buf);
   }
 }
 return status;
}

Further abstracted out is the code common to the read and write threads. io_test_fn checks when the thread is finished and should exit; io_size_fn returns how much read or write space is available; io_body_fn contains code specific to reading or writing.

It would be good to malloc void* buf only once at the beginning of the thread to ensure no pagefaults. However, since the allocation does not occur in a realtime thread and no overruns or underruns (dropouts) were observed in testing, it was not a high priority to fix.

Thread implementation

static int reader_can_run(recap_io_info_t* info) {
 return info->state->can_read;
}

static int writer_can_run(recap_io_info_t* info) {
 return info->state->can_capture;
}

static int reader_is_done(recap_io_info_t* info) {
 return 0;
}

static int writer_is_done(recap_io_info_t* info) {
 return jack_ringbuffer_read_space(info->ring) == 0 && info->state->playing == DONE;
}

static size_t reader_space(recap_io_info_t* info) {
 return jack_ringbuffer_write_space(info->ring);
}

static size_t writer_space(recap_io_info_t* info) {
 return jack_ringbuffer_read_space(info->ring);
}

Read and write implementations of the above typedefs.

static int reader_body(void* buf, size_t space, recap_io_info_t* info) {
 static int underfill = 0;
 int status = 0;
 sf_count_t nframes = space / frame_size_r;
 sf_count_t frame_count = sf_readf_float(info->file, buf, nframes);
 if (frame_count == 0) {
   DEBUG("reached end of sndfile: %s\n", info->path);
   info->state->reading = DONE;
   status = FINISHED;
 } else if (underfill > 0) {
   ERR("cannot read sndfile: %s\n", info->path);
   status = EIO;
 } else {
   sf_count_t size = frame_count * frame_size_r;
   if (jack_ringbuffer_write(info->ring, buf, size) < size) {
     ++info->underruns;
     ERR("reader thread: buffer underrun\n");
   }
   DEBUG("read %6ld frames\n", frame_count);
   info->state->reading = RUNNING;
   if (frame_count < nframes && underfill == 0) {
     DEBUG("expected %ld frames but only read %ld,\n", nframes, frame_count);
     DEBUG("wait for one cycle to make sure.\n");
     ++underfill;
   }
 }
 return status;
}

Reader implementation of io_body_fn. It is impossible to tell if the first underfill is caused by IO problems or by reaching the end of the sound file. If the frame count for the following cycle is zero, it is assumed that the end of the file has been reached; otherwise if underfill is greater than zero it must be an IO issue so the thread exits.

The use of static int underfill means no more than one reader thread can be active during the lifetime of the program.

static int writer_body(void* buf, size_t space, recap_io_info_t* info) {
 int status = 0;
 sf_count_t nframes = space / frame_size_w;
 jack_ringbuffer_read(info->ring, buf, space);
 if (sf_writef_float(info->file, buf, nframes) < nframes) {
   ERR("cannot write sndfile (%s)\n", sf_strerror(info->file));
   status = EIO;
 }
 DEBUG("wrote %5ld frames\n", nframes);
 return status;
}

Writer implementatino of io_body_fn.

static int reader_thread_fn(recap_io_info_t* info) {
 return io_thread(&reader_can_run, &reader_is_done,
                  &reader_space, &reader_body, info);
}

static int writer_thread_fn(recap_io_info_t* info) {
 return io_thread(&writer_can_run, &writer_is_done,
                  &writer_space, &writer_body, info);
}

Read and write implementations of io_thread_fn. Due to the earlier abstraction these definitions are simple.

static void io_cleanup(void* arg) {
 recap_io_info_t* info = (recap_io_info_t*) arg;
 sf_close(info->file);
}

static void io_free(recap_io_info_t* info) {
 jack_ringbuffer_free(info->ring);
}

io_cleanup() is passed to common_thread as the thread cleanup callback. io_free() is used at the end of main() to free resources which the jack thread may still be using after the IO threads exit.

static void* writer_thread(void* arg) {
 return common_thread(&write_lock, &ready_to_write,
                      &writer_thread_fn, &io_cleanup, arg);
}

static void* reader_thread(void* arg) {
 return common_thread(&read_lock, &ready_to_read,
                      &reader_thread_fn, &io_cleanup, arg);
}

Functions to start writer and reader threads.

Main jack callback

static int process(jack_nframes_t nframes, void* arg) {
 recap_process_info_t* info = (recap_process_info_t*) arg;
 recap_state_t* state = info->state;

The main jack callback. This function must read nframes of signal from the connected input ports and write nframes of signal to the connected output ports. The size of nframes is determined by the caller (jack).

if (!state->can_play || !state->can_capture || !state->can_read)
   return 0;

No point reading or writing anything to jack’s buffers if everything isn’t ready to go.

recap_sample_t* in[MAX_PORTS];
 recap_sample_t* out[MAX_PORTS];
 int i = 0;
 jack_port_t* prt;
 while ((prt = recap_in_ports[i]) != NULL) {
   in[i++] = (recap_sample_t*) jack_port_get_buffer(prt, nframes);
 }
 in[i] = NULL;

 i = 0;
 while ((prt = recap_out_ports[i]) != NULL) {
   out[i++] = (recap_sample_t*) jack_port_get_buffer(prt, nframes);
 }
 out[i] = NULL;

Get the signal buffers of each input and output port. It is recommended in the jack documentation that these are not cached.

if (state->playing != DONE && state->reading != IDLE) {

If state->reading is IDLE there is nothing to play yet, and if state->playing is DONE it is time to exit the program.

jack_ringbuffer_t* rring = info->reader_info->ring;
   size_t space = jack_ringbuffer_read_space(rring);
   if (space == 0 && state->reading == DONE) {
     state->playing = DONE;
     recap_mute(out, channel_count_r, nframes);
   } else {
     int err = uninterleave(out, channel_count_r, nframes, &next_value, rring);
     if (err) {
       ++info->underruns;
       ERR("control thread: buffer underrun\n");
     }
   }

This, the guts of the processing is simply uninterleaving the file data and writing it to the buffers of the appropriate output ports. Jack handles the rest.

jack_ringbuffer_t* wring = info->writer_info->ring;
   int err = interleave(in, channel_count_w, nframes, &write_value, wring);
   if (err) {
     ++info->overruns;
     ERR("control thread: buffer overrun\n");
   }
 }

Similarly simple. Interleaving the input port data and writing to the writer thread’s ringbuffer.

if (pthread_mutex_trylock(&read_lock) == 0) {
   pthread_cond_signal(&ready_to_read);
   pthread_mutex_unlock(&read_lock);
 }

 if (pthread_mutex_trylock(&write_lock) == 0) {
   pthread_cond_signal(&ready_to_write);
   pthread_mutex_unlock(&write_lock);
 }
 return 0;
}

Data has been written to the writer thread’s ringbuffer and removed from the reader thread’s ringbuffer, so signal each thread to begin another iteration.

Thread setup and running

static int setup_writer_thread(recap_io_info_t* info) {
 int status = 0;
 SF_INFO sf_info;
 sf_info.samplerate = jack_get_sample_rate(client);
 sf_info.channels = channel_count_w;
 sf_info.format = SF_FORMAT_WAV | SF_FORMAT_PCM_32;
 if ((info->file = sf_open(info->path, SFM_WRITE, &sf_info)) == NULL) {
   ERR("cannot open sndfile \"%s\" for output (%s)\n",
       info->path, sf_strerror(info->file));
   jack_client_close(client);
   status = EIO;
 }
 DEBUG("opened to write: %s\n", info->path);
 DEBUG("writing %i channels\n", channel_count_w);
 info->ring = jack_ringbuffer_create(sample_size * ring_size);
 memset(info->ring->buf, 0, info->ring->size);
 info->state->can_capture = 0;
 pthread_create(&info->thread_id, NULL, writer_thread, info);
 return status;
}

Set up resources for the writer thread then create the thread. This means opening a (multichannel) WAV file to write to, creating a ringbuffer, and touching all its allocated memory to prevent pagefaults later on.

static int setup_reader_thread(recap_io_info_t* info) {
 int status = 0;
 SF_INFO sf_info;
 sf_info.format = 0;
 DEBUG("opened to read: %s\n", info->path);
 if ((info->file = sf_open(info->path, SFM_READ, &sf_info)) == NULL) {
   ERR("cannot read sndfile: %s\n", info->path);
   status = EIO;
 }
 channel_count_r = sf_info.channels;
 frame_size_r = channel_count_r * sample_size;
 DEBUG("reading %i channels\n", channel_count_r);
 if (sf_info.samplerate != 44100) {
   ERR("jack sample rate must be 44100 (is %i)\n", sf_info.samplerate);
   cancel_process(proc_info);
 }
 info->ring = jack_ringbuffer_create(sample_size * ring_size);
 memset(info->ring->buf, 0, info->ring->size);
 info->state->can_read = 0;
 pthread_create(&info->thread_id, NULL, reader_thread, info);
 return status;
}

Same purpose as the previous function but for the reader thread. Opens a (multichannel) WAV file to read from, creates a ringbuffer, and touches all its memory.

Similarites between these two functions could probably be extracted into a general setup_io_thread() function.

static int run_io_thread(recap_io_info_t* info) {
 int* status;
 int ret;
 pthread_join(info->thread_id, (void**) &status);
 if (status == PTHREAD_CANCELED) {
   ret = EPIPE;
 } else {
   ret = *status;
   free(status);
 }
 return ret;
}

Joins to a thread and returns its exit status.

Port registration and connection

static jack_port_t* register_port(char* name, int flags) {
  jack_port_t* port;
  if ((port = jack_port_register(client, name, JACK_DEFAULT_AUDIO_TYPE, flags, 0)) == 0) {
    DEBUG("cannot register port \"%s\"\n", name);
    jack_client_close(client);
    exit(1);
  }
  return port;
}

Register the named port with the jack process.

static void connect(const char* out, const char* in) {
 if (jack_connect(client, out, in)) {
   DEBUG("cannot connect port \"%s\" to \"%s\"\n", out, in);
   jack_client_close(client);
   exit(1);
 }
}

static void connect_ports(char** in_names, char** out_names) {
 int i;
 char* s;
 for (i = 0; i < channel_count_w; i++) {
   char prt[32];
   sprintf(prt, "recapture:input_%i", i);
   char* shrt = prt + strlen("recapture:");
   register_port(shrt, JackPortIsInput);
   recap_in_ports[i] = jack_port_by_name(client, prt);
   if ((s = in_names[i]) != NULL) connect(s, prt);
 }
 recap_in_ports[i] = NULL;
 for (i = 0; i < channel_count_r; i++) {
   char prt[32];
   sprintf(prt, "recapture:output_%i", i);
   char* shrt = prt + strlen("recapture:");
   register_port(shrt, JackPortIsOutput);
   recap_out_ports[i] = jack_port_by_name(client, prt);
   if ((s = out_names[i]) != NULL) connect(prt, s);
 }
 recap_out_ports[i] = NULL;
}

Initialize recap_in_ports and recap_out_ports with this client’s in and out ports, and connect them to the supplied jack ports.

Set callbacks and handlers

static void set_handlers(jack_client_t* client, recap_process_info_t* info) {
 jack_set_process_callback(client, process, info);
 jack_on_shutdown(client, jack_shutdown, info);
 signal(SIGQUIT, signal_handler);
 signal(SIGTERM, signal_handler);
 signal(SIGHUP, signal_handler);
 signal(SIGINT, signal_handler);
}

Set jack callbacks and signal handlers.

Run jack client

static int run_client(jack_client_t* client, recap_process_info_t* info) {
 recap_state_t* state = info->state;

 state->can_play    = 1;
 state->can_capture = 1;
 state->can_read    = 1;

 int reader_status = run_io_thread(info->reader_info);
 int writer_status = run_io_thread(info->writer_info);
 int other_status = 0;

 if (info->overruns > 0) {
   ERR("recapture failed with %ld overruns.\n", info->overruns);
   ERR("try a bigger buffer than -b %" PRIu32 ".\n", ring_size);
   other_status = EPIPE;
 }
 long underruns = info->underruns + info->reader_info->underruns;
 if (underruns > 0) {
   ERR("recapture failed with %ld underruns.\n", underruns);
   ERR("try a bigger buffer than -b %" PRIu32 ".\n", ring_size);
   other_status = EPIPE;
 }
 return reader_status || writer_status || other_status;
}

Run reader and writer threads, and return their status.

Looks like can_play, can_capture, and can_read could be collapsed in to one field.

Argument parsing

static void split_names(char* str, char** list) {
 int i = 0;
 char* s = strtok(str, ",");
 while (s != NULL) {
   list[i++] = s;
   s = strtok(NULL, ",");
 }
 list[i] = NULL;
}

Port names given on the commandline are comma separated.

static void parse_arguments(int argc, char** argv, char** in_names, char** out_names) {
 char* optstring = "b:i:o:h";
 struct option long_options[] = {
   { "help", 0, 0, 'h' },
   { "bufsize", 1, 0, 'b' },
   { "inports", 1, 0, 'i' },
   { "outports", 1, 0, 'o' },
   { 0, 0, 0, 0 }
 };

 int c;
 int longopt_index = 0;
 int show_usage = 0;
 extern int optind;
 extern int opterr;
 opterr = 0;
 while ((c = getopt_long(argc, argv, optstring, long_options, &longopt_index)) != -1) {
   switch (c) {
   case 1:
     break;
   case 'h':
     show_usage = 1;
     break;
   case 'b':
     ring_size = atoi(optarg);
     break;
   case 'i':
     split_names(optarg, in_names);
     break;
   case 'o':
     split_names(optarg, out_names);
     break;
   default:
     show_usage = 1;
     break;
   }
 }

 if (show_usage == 1 || argc - optind < 2) {
   MSG("%s", usage);
   exit(1);
 }
}

Straightforward argument handling.

main

int main(int argc, char** argv) {
 recap_process_info_t info;
 recap_io_info_t reader_info;
 recap_io_info_t writer_info;
 recap_state_t state;
 memset(&info, 0, sizeof(info));
 memset(&reader_info, 0, sizeof(reader_info));
 memset(&writer_info, 0, sizeof(writer_info));
 memset(&state, 0, sizeof(state));
 info.reader_info = &reader_info;
 info.writer_info = &writer_info;
 info.state = &state;
 reader_info.state = &state;
 writer_info.state = &state;
 state.can_play = 0;
 state.reading = IDLE;
 state.playing = IDLE;
 proc_info = &info;

Initialize info instances and touch their memory to prevent pagefaults.

char* in_port_names[MAX_PORTS];
 char* out_port_names[MAX_PORTS];
 parse_arguments(argc, argv, in_port_names, out_port_names);

 proc_info->reader_info->path = argv[optind];
 proc_info->writer_info->path = argv[++optind];

Port names and file paths.

channel_count_w = array_length(in_port_names);
 frame_size_w = channel_count_w * sample_size;

Writer thread channel count and frame size. Those for the reader thread are taken from the input file in setup_reader_thread().

DEBUG("%s\n", proc_info->reader_info->path);
 if ((client = jack_client_open("recapture", JackNullOption, NULL)) == 0) {
   ERR("jack server not running?\n");
   exit(1);
 }

 set_handlers(client, proc_info);

Connect to jack and set up jack and signal callbacks.

int status = 0;
 if (!(status = setup_writer_thread(proc_info->writer_info)) &&
     !(status = setup_reader_thread(proc_info->reader_info))) {
   if (jack_activate(client)) {
     ERR("cannot activate client\n");
     status = 1;
   } else {
     connect_ports(in_port_names, out_port_names);
     DEBUG("connected ports\n");
     status = run_client(client, proc_info);
     jack_client_close(client);
   }
 }

Provided the IO threads execute ok, run this client and then close it once run_client() returns.

io_free(proc_info->reader_info);
 io_free(proc_info->writer_info);
 return status;
}

Free remaining resources and return.

Running

First start jackd, then run recapture like this:

recapture -i system:in_1,system:in_2 -o system:out_1,system:out_2 at_least_two_channels_input.wav two_channels_out.wav