Abstract

mini-buildd is a custom build daemon for Debian-based distributions with all batteries included: I.e., it covers incoming, (distributed) building, installing, repository maintenance, repository delivery – and some extras.

Main notion is to make it easy to bootstrap and maintain an add-on (i.e., add onto an existing Debian distribution) Debian package infrastructure. So it’s, for example, suitable for sandboxing, personal package repositories or organizations maintaining custom or adapted Debian packages.

mini-buildd does not re-invent the wheel, it’s rather a sophisticated glue written in python3 to a number of standard (Debian) or off-the-shelf software components. As core components, it uses Debian package ‘twisted’ (HTTP server), Debian package ‘python-django’ (web application), Debian package ‘python-pyftpdlib’ (package uploads), Debian package ‘sbuild’ && Debian package ‘schroot’ (package building) and Debian package ‘reprepro’ (for the APT package archive).

Features

Core features:

  • Integrated HTTP server (webapp and repository delivery).

  • Integrated FTP server (incoming).

  • Easy bootstrap or maintenance via API call ‘setup’, fine-grained custom setup via Web-based configuration.

  • Distributed builders (with optional wake support for remotes).

  • Web-based or command line repository maintenance via API calls.

Some Arbitrary Extras:

  • Events Queue: Web or shell, check what’s going on.

  • User Management: Package subscriptions, GPG key management, upload authorization.

  • Package QA: Internal sanity checks, version enforcing, lintian, autopkgtest – we make package building hard again.

  • Package Tracking: Debian PTS-(sort-of)-like web based source package tracker.

  • No-Changes-Ports: Automates ports, for internal or external source packages.

  • Keyring Packages: Generated automatically.

  • Package Staging: unstable->testing->stable-style (manual) migration.

  • Package Rollbacks: Keeps N rollbacks for any distribution.

  • Repository Snapshots: Automated (time based) or manual.

Basic Mode Of Operation

mini-buildd is a Unix daemon written in python. When running, it provides a HTTP server (on port 8066 by default).

The HTTP server serves both, mini-buildd’s web application as well as the delivery of the package repositories.

The instance is being configured in the configuration section of the web application.

As soon as a mini-buildd instance has been configured to have an active ‘Daemon’, you may start the engine, running an FTP server (on port 8067 by default).

The FTP server acts on incoming *.changes files, both from developers and other mini-buildd instances (via special buildrequest and buildresult changes).

As soon as an instance of mini-buildd has active chroots configured, it acts as builder. Chroots are completely generic and interchangeable, and identified by codename and arch only; distribution-specific build configuration is all carried through the internal buildrequests. Thus, mini-buildd instances may be interconnected as so-called ‘Remotes’ to share builders.

This is a simplified example mini-buildd ‘network’ with three mini-buildd instances ernie, grover and bert:

digraph flow_simple { node [fontname=Arial fontsize=11 shape=diamond style=filled fillcolor=grey]; edge [fontname=Helvetica fontsize=8]; subgraph cluster_0 { style=filled; color=lightgrey; label="ernie"; "Ernie-Packager" [label="Packager"]; "Ernie-Builder" [label="Builder"]; "Ernie-Repositories" [label="Repositories" shape=folder]; } "Ernie-Developer" [shape=oval fillcolor=lightgrey]; "Ernie-Developer" -> "Ernie-Packager" [label="uploads"]; "Ernie-Packager" -> "Ernie-Repositories" [label="installs"]; "Ernie-Packager" -> {"Ernie-Builder" "Grover-Builder"} [dir=both label="builds"]; "Ernie-Manager" [shape=oval fillcolor=lightgrey]; "Ernie-Manager" -> "Ernie-Repositories" [label="manages"]; "Ernie-User" [shape=oval fillcolor=lightgrey]; "Ernie-Repositories" -> "Ernie-User" [label="apt"]; subgraph cluster_1 { style=filled; color=lightgrey; label="grover"; "Grover-Builder" [label="Builder"]; } subgraph cluster_2 { style=filled; color=lightgrey; label="bert"; "Bert-Packager" [label="Packager"]; "Bert-Repositories" [label="Repositories" shape=folder]; } "Bert-Developer" [shape=oval fillcolor=lightgrey]; "Bert-Developer" -> "Bert-Packager" [label="uploads"]; "Bert-Packager" -> "Bert-Repositories" [label="installs"]; "Bert-Packager" -> {"Ernie-Builder" "Grover-Builder"} [dir=both label="builds"]; "Bert-Manager" [shape=oval fillcolor=lightgrey]; "Bert-Manager" -> "Bert-Repositories" [label="manages"]; "Bert-User" [shape=oval fillcolor=lightgrey]; "Bert-Repositories" -> "Bert-User" [label="apt"]; }

  • ernie has repositories and chroots, and uses himself and grover as remote for building.

  • grover only has chroots, and is used by ernie and bert for building.

  • bert only has repositories, and uses ernie and grover as remotes for building.

About this Manual

This manual is neither complete nor the only source of documentation, especially for the nitty-gritty details:

Tip

Some links do not work in this manual?

If you are reading this manual stand-alone the resp. links to “In-App content” will not work.

Read this manual served by mini-buildd itself.

Conventions

Sections

Section hierarchy #*=-~:

  • ###: Document

  • ***: Section

  • ===: Subsection

  • ---: Subsubsection

  • ~~~: Subsubsubsection

Admonitions

We use RST admonitons error, attention, note, tip (so these can be summarized in Admonitions (FAQ, NOTE, TODO, BUG)).

Code

In code snippets, names written all-capital (FOO, HOST) are not meant literal but placeholders for customized values.

Code expected to run as any user:

mini-buildd-dput foo bar.dsc

Code expected to be run as mini-buildd:

mini-buildd-schroot-cleanup

Code expected to be run as root:

apt install foo