McDemigod.com

PUISNE 🪟❗

PUISNE: Portable UnINtruSive Executor

What is it?

A tool to package an executable & its resources quickly and easily. Think MacOS Application Bundles that work anywhere, even across operating systems! They’re exceedingly simple to work with and to share.

It’s mostly intended for research teams who frequently collaborate using custom programs with code and/or data dependencies. Packaging all that into a single file allows rapid iteration & collaboration, with barely any technical overhead. You’re more likely to use it for something in ~/some/random/project than /bin/ or ~/.local (although the latter is indeed reasonable).

PUISNE is…

• simple:
Packages have the barest structural requirements, and no spec files.
• self-contained:
A single binary, no daemon or framework needed. It just works™.
• shareable:
Not just source-portable. PUISNE is implemented with Cosmopolitan Libc so it runs on Linux, Mac, BSD, and Windows.
• transmutable:
Changed something? Update the zip & share it back.
Wanna make one of your own? ctrl-c ctrl-v your way to a new puis.

It does not…

• get in the way:
It avoids intercepting inputs or arguments intended for the packaged executable. Drop it in-place & forget it’s there.
• need root privilege or any group shenanigans:
You do you, hun.
• isolate, sandbox, or virtualize;
bring your own interpreter.

How does it work?

It’s basically a glorified self-extracting zip archive. That’s the key to its ease-of-use: creating one doesn’t need any specialized software; just use Info-ZIP or what-have-you. Add your application directory to an empty PUISNE & ship it off.

When the package is run, it extracts the contents, then execs the main executable – with all the resources already in-place & as though you had done so directly. Furthermore, caching is available to prevent re-extracting unnecessarily (see the -u option); by default it only replaces out-of-date files.

Run an empty PUISNE for detailed usage instructions.

Building or modifying a package

PUISNE needs a truly zip-compliant archiver. Info-zip is the most thoroughly tested most & works seamlessly; it’s basically the standard available & everywhere. Examples in the help text use that. NB to maintain Windows compatibility using zip’s -g flag is recommended. If you ever forget your package will still work in *nix, but to recover Windows functionality, re-make your package by extracting its contents and adding them to a fresh, empty PUISNE.

GUI tools

Since “simple” is the whole point & to avoid scaring-off the CLI disinclined, it would be nice to support GUI-based zip utilities. Admit it: you love drag-and-drop. Unfortunately, not many GUIs tested work – usually because they cut corners. Ah well.

Works-ish

1. WiZ

   This is a front-end for Info-ZIP. It’s only available for Windows, and the UI isn’t really more user friendly than the CLI. That said, it can work; make sure to set the reference directory properly with the “Set Ref” button. I haven’t found how to set the -g switch, so ironically this won’t build Windows compatible packages. So it goes…

2. WinRar

   Yes, I actually tested this. It worked well enough but might drop Windows compatibility. Then again, if this is what you’re using, you probably only care about Windows in the first place & in that case you can just make an SFX with WinRar already…

Works to extract content but not to add

3. Windows “Compressed Folder Tools”

   To view a package with Windows Explorer, change the file extension to .zip. Supposedly older versions of Windows might work to modify as well as extract, but I have not confirmed this. Please let me know if you succeed.

4. 7Zip

   Need to specifically open as a .zip, otherwise it will parse as an ELF. Attempting to modify content yields an error.

Don’t work at all

5. Peazip,

6. xArchiver

   These seem only able to parse the executable content, not the zip object store. Boo!

7. File Roller + p7zip-full

   Fails altogether.

Additional Linux sexiness

In Linux it uses a few more tricks to remain out-of-sight. After it extracts its contents to the target directory, it mounts it over or under the working environment.

Mounting over the working environment is more “app-like” – it will “hide away” any modifications to the target directory & preserve the state of the working environment. In contrast, mounting under is “notebook-like” – Not only can it modify the working environment, it “prefers” local files to its own, if any already exist at the specified path.

An example makes this clearer:

Starting from scratch; there's just a script & some "data" in an app folder, and a PUISNE binary.

...which is indeed empty.

The data just specify their source: the package.

The script just prints the executable name, then prints the data, then logs user input.

We make the package...

after which we don't need the source.

Overlay

For “app-like” behavior, mount over the working environment. The package will still be able to read from the working environment, but will not write to it. This is useful, for example, to keep config or log files out of the way, on a per-app basis. To see any files written, look in the target directory. Furthermore, if there are files in the environment with paths that clash with any in the package, it will ignore those for its own.

Running it goes as expected.

No log cluttering up the place.

Oh, right...

Underlay

Mounting under the working environment is more “notebook-like”. Changes will persist in the working environment. This behavior is in-line with tools like RStudio / Jupyter notebooks, which create outputs like figures in the directory containing the notebook; this means users don’t need to look in eg. a .puisne subdirectory. Finally, if any files in the environment have the same path as those in the package, those are available; this way, users can more easily provide data on which to operate.

I'm just going to continue right here.

Provide some "local" data.

Remember the leading -- to talk to PUISNE.
See? It found the local data.

Now the environment contains our "output".

Which of course differs from last time.

Don't worry, our shame yet lingers.

Other questions

Why not just use X?

• Why not just use X?
  • If X works for you, use X! I’ve just seen a lot of times when X was overkill, or collaborators didn’t know how to use X, or X held up scientific progress, or…
    This just provides a middle-of-the-road alternative that might just hit that balance between ease & function. I will address the two I’ve seen most frequently, though:

Why not Git?

• Why not Git?
  • git’s great for developing code, not necessarily running it. People want your program, not your code; this is just that. If you’ve ever made a package you know how much nicer it is for your users, especially considering any make, install, or activate steps you might be able to circumvent.

Why not Docker?

• Why not Docker?
  • Again, great for what it does, but… does more than you might need. PUISNE interacts with the environment instead of creating it.
    

    “Invent the universe? I just want some damn pie.”

    Carl Sagan--

    That said, these aren’t mutually exclusive – you can always bake it into your Dockerfile with COPY my_puisne.com ..., or mount it via -v so you can tweak it without rebuilding, or… Bottom line: do you. But when your team is in the “running laps between the whiteboard & their keyboard” phase, do you really want to slow them down?

Windows compatibility?

• What’s the deal with Windows compatibility?
  • zip sometimes truncates the file; we don’t want it to do that. Try again with the -g option.
    
    It’s an issue with the PE executable format. If you look at a hexdump of the “empty” PUISNE binary, all those zeroes at the end are actually important. The PE header includes a section table that describes how the executable is organized (where each section begins & how large it is). If the file does not match what is contained there, Windows gets real mad. Unfortunately, zip by default tries to see if it can squeeze in whatever it’s adding in-place & if so, it does so then drops the padding. Therefore adding something small can potentially shrink the archive file; using the “grow” flag, -g, should avoid this regardless. This should be unnecessary for large additions, since that won’t be “shoved into a gap” & thus should be appended.
    If you encounter any issues, please let me know.

It makes stuff cross-platform?

• So this makes my stuff cross-platform?
  • No. PUISNE itself is cross-platform, but whatever it launches needs to be compatible with whatever environment in which it will run. For interpreted languages, this shouldn’t be a problem; the system interpreter should have you covered.

Can I include the interpreter?

• Can I include the interpreter?
  • Of course! But as described above, Doing so might limit where you can run. That said, if you include a statically-linked binary you potentially don’t need to rely on host libraries at all. In fact, this is exactly what Google does with “Hermetic Python” BUILD rules in Bazel. This is just easier & more flexible.

What have you got planned for this?

• What have you got planned for this?
  • I’d like to implement functionality similar to what is accomplished by overlayfs, even on systems that don’t support it. I think I may have worked out a version that basically covers what I want it to. I’d like to improve Windows use, too; bonus if that means finding a compliant GUI that works. Adding support for tar archives could improve compression, I guess (Zip doesn’t compress information redundant across files). I think I have worked out a solution that should take care of those last two, but would potentially change the way packages are made.
    Beyond that, it depends how people use this. You can always request a feature.