Ardour and Linux VST

I have Ardour 3 compiled and working. I don't have a realtime kernel, so I can't run Jack in realtime mode and there is some significant latency (~1/4-1/2 second, didn't time exactly). I looked up the interwebs to see what was involved in getting a realtime kernel set up and found this gem:

http://askubuntu.com/questions/72964/how-can-i-install-a-realtime-kernel 

I followed the instructions exactly on a VM running the same OS as my notebook and it worked without a hitch. I decided it may be better to use the same kernel version (3.2.0). The closest match with a corresponding RT patch was 3.2.52. It took about two hours to compile everything but so far it's working perfectly. Just one hitch, not sure it's totally related to the kernel update: Firefox and emacs look-and-feel reverted to some ugly state and Firefox stopped recognizing the printer. I did some ridiculous things like reinstalling gtk3 and cups. Eventually, I just rebooted and everything was back to normal (but I have and RT kernel now).

I tried running Ardour with Jack in RT mode, but it wouldn't open my project. Unsure if I needed to rebuild everything (looking back, I don't think I had to) I felt it better to do that anyway, since originally I was adding packages kind of willy-nilly as I was building the dependencies. The new build of Ardour still didn't work. After scanning the interwebs for people with similar problems, I found a solution that worked for me: Simply add my username to the audio group with sudo usermod -a -G audio ${username}. The latency is much more manageable now (~30-40ms).  Not good for real production stuff, but good enough as a sandbox for plugin development.

I tried compiling the examples that come with the VST3 SDK. The following define flags will get most to compile:

-DPTHREADS -DBYTEORDER=kLittleEndian -DFORMAT_INT64A="\"%ld\"" -DFORMAT_UINT64A="\"%lu\"" -DENDLINE_A="\"\\n\""

You will also need to define things like -DRELEASE/-DDEBUG/-DDEVELOPMENT and -DPLATFORM_64.

Even with all these, I still had to hack my way through to get some of the code to compile. I won't reproduce the code here but if you've been programming C/C++ for a while you should find the required mods to be pretty straightforward.

I skipped trying to compile the examples, especially because a bunch are in a subdirectory with a space in the name (pain in the butt!). I copied one of the examples into QtCreator, tinkered with the configuration a bit and got it to build a nice SO.

When Linux VST's are enabled in Ardour, it searches the standard shared object library path for a folder named lxvst (I think status quo is to put it in /usr/local/lib/lxvst). I opened the plugin manager by right clicking on the mixer pane. There wasn't anything obviously wrong with the SO itself, so I searched through the Ardour source code to find out where the load was failing. Turns out Ardour 3 isn't equipped to read Linux VSTs compiled according to VST version 3. It wasn't clear to me how to mod Ardour to get this to work. I also haven't been able to figure out how to get the example code to build in 2.x format.

Instead, I created a new dummy plugin called JustVisiting. It took a little bit of tinkering, but eventually I got Ardour to recognize it. Here's the skeleton code:

justvisiting.h:

#include <stdio>
#include "public.sdk/source/vst2.x/audioeffectx.h"

class JustVisiting : public AudioEffectX {
public:
    JustVisiting(audioMasterCallback iAudioMaster, 
        VstInt32 iNumPrograms, VstInt32 iNumParams);
    void processReplacing(float **inputs, float **outputs, VstInt32 sampleFrames);
    bool getVendorString(char *text);
    VstInt32 getVendorVersion();
    VstInt32 canDo(char *text);
    VstPlugCategory getPlugCategory();
};

justvisiting.cpp:

#include "justvisiting.h"

AudioEffect *createEffectInstance(audioMasterCallback audioMaster) {
    JustVisiting *effect = new JustVisiting(audioMaster,1,1);
    return effect;
}

JustVisiting::JustVisiting(audioMasterCallback iAudioMaster, 
    VstInt32 iNumPrograms, VstInt32 iNumParams)
    : AudioEffectX(iAudioMaster,iNumPrograms,iNumParams){
    // nothing to allocate
    this->setNumInputs(2);
    this->setNumOutputs(2);
    this->setBlockSize(1);
}

void JustVisiting::processReplacing(float **inputs, float **outputs, VstInt32 sampleFrames) {
    // TODO: do something
}

bool JustVisiting::getVendorString(char *text) {
    sprintf(text,"CamelNote");
}

VstInt32 JustVisiting::getVendorVersion() {
    return 0;
}

VstInt32 JustVisiting::canDo(char *text) {
    if ( strcasecmp(text,"sendVstEvents") )
        return -1;
    else if ( strcasecmp(text,"sendVstMidiEvent") )
        return -1;
    else if ( strcasecmp(text,"receiveVstEvents") )
        return -1;
    else if ( strcasecmp(text,"receiveVstMidiEvent") )
        return -1;
    else if ( strcasecmp(text,"receiveVstTimeInfo") )
        return 0;
    else if ( strcasecmp(text,"offline") )
        return -1;
    else if ( strcasecmp(text,"midiProgramNames") )
        return -1;
    else if ( strcasecmp(text,"bypass") )
        return -1;
    return 0;
}

VstPlugCategory JustVisiting::getPlugCategory() {
    return kPlugCategEffect;
}

Be sure to build this in with vstplugmain.cpp from the VST SDK so that the VSTPluginMain symbol gets compiled in. Ardour uses this as the entry point for the Linux VST (It actually looks for 'main' first, and if it can't find it, then looks for VSTPluginMain. When if finds that one, it will complain that the version is too new and may not be compatible with Ardour). At the bare minimum, you'll need to define processReplacing, but there are a number of other member functions you should override for safety. I haven't gone through them all yet.