Changing a setting
A Flockfile is committed to the app's repository, and shep never writes to it. So a load has to answer one question before it can be safe to run against a flock that is already up: when the file and the running config disagree, who wins? The answer is the file, but only for settings nobody has spoken for yet.
A Flockfile is a template, not live config
A Flockfile lives in the repository of the app it starts, committed like anything else there, and shep never writes to it. That is what makes it safe to load one onto a flock that is already running without reading the diff first: shep start Flockfile.toml appends the settings nobody has established yet and overwrites nothing.
The alternative was letting shep start Flockfile.toml overwrite whatever it found. That reads fine on a laptop and is indefensible on a host: a Flockfile arrives through the app's own repository, so anybody who can merge a pull request could lower a memory ceiling an operator raised at 3am, and the operator would find out from the incident rather than from the diff.
So a load appends. It writes the settings nobody has established yet and leaves everything else exactly as it is, which means the worst a merged change can do to a running flock is nothing.
The corollary is the part worth remembering. Editing a line in a Flockfile and loading it again does not move a setting that is already established; shep start Flockfile.toml --reset=policy is how you say the file wins. The rest of this page is that whole story, including which settings change under a running process and which wait for the next spawn.
Naming a sheep reads no Flockfile at all. shep start web starts the sheep called web from the config the shepherd already holds, and cannot be turned into a load by anything sitting in the current directory. Three things are a load, and nothing else is: a path you name, a Flockfile discovered in the directory you ran the command from, and -, which reads Flockfile JSON on stdin.
Three things a load with no flag will not do
shep start carrying it registers and starts that app as it always has, because starting things is what you typed. What the merge cannot do is reach a host through config alone: the load half of a merged pull request can only touch sheep that are already there../a/Flockfile.toml and then ./b/Flockfile.toml would otherwise have the second wipe the first's flock.All three are about the load a merged pull request can cause, which is the one with no flag on it, and that is the bound worth knowing: an unreviewed template cannot cost you a process. --reset is a person asking for the template's shape, and a shape includes a count. Three of the four modes can kill, and it has a section of its own below.
A dog is refused outright, at every depth. A Flockfile that names metrics gets metrics is a dog, and a dog's config comes from `shep adopt` rather than from a Flockfile and nothing merges. A dog runs at the shepherd's own trust level, so a file that could set its script would be a way to swap the binary on the next restart.
What a load looks like
One app and two environment variables with no values in them. The empty strings are deliberate and there is a section on them below:
[[app]] name = "web" script = "./server.sh" [app.env] DB_HOST = "" DB_PASSWORD = ""
$ shep start Flockfile.toml ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 0 web online 96289 0 - - - 32.0K 0s - -
Nothing to report on a first load: the app was not in the flock, so shep start registered and started it, and every key in the file became one somebody has now spoken for. Add a max_memory and a cwd pointing somewhere else, and load the same file again:
$ shep start Flockfile.toml notice[start]: web: applied max_memory; cwd waiting on the next spawn (`shep reload web` promotes them) notice[start]: web is already online; `shep restart web` replaces it. ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 0 web online 96289 0 - !1 - 1.2M 4s - -
Two different outcomes in one line, which is the whole model in miniature. max_memory is read fresh every time the memory enforcer makes a decision, so writing it down is enough and it is applied. cwd is baked into the child at exec and there is no way to change it under a running process, so it is parked, and the report names the verb that promotes it rather than leaving you to guess.
The CFG column
!1 in CFG means one field is parked on this sheep. *1 means one field an operator has taken ownership of, whether or not the current Flockfile declares it, and - means neither. It is a count rather than a list because the cell is three characters wide; shep describe is where the names are:
$ shep describe web ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 0 web online 96289 0 - !1 - 1.2M 4s - - Pending for web, parked by a load; `shep reload web` promotes it: cwd
A parked field takes the cell when a sheep has both. It is the more urgent of the two: an override has been true since the last load, while a parked field is one command away from taking effect and the sheep is not on it yet.
shep reload and shep restart both promote. Reload is the one the report names because it is the one that keeps the flock up:
$ shep reload web ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 0 web:0 stopping 96289 0 - !1 - 1.2M 4s - - 1 web:0 starting 96368 0 - - - - 0s - - $ shep flock ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 1 web online 96368 0 - - - 1.2M 3s - -
Identity is the one thing a promotion re-resolves. Changing user or group sends shep back to the passwd database, because an operator who changed who a sheep runs as means it; every other promotion reuses the credentials resolved at the sheep's original start and costs no lookup.
A second load does not repeat the first
Once a key has been established, the file stops speaking for it. Move max_memory to 2G and load again and shep says nothing at all, because the key is one somebody has already set:
$ shep start Flockfile.toml notice[start]: web is already online; `shep restart web` replaces it. ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 1 web online 96368 0 - - - 1.2M 3s - - $ shep start Flockfile.toml --reset=policy notice[start]: web: applied max_memory notice[start]: web is already online; `shep restart web` replaces it. ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 1 web online 96368 0 - - - 1.2M 3s - -
--reset and --reset-all are gone, with no alias for either: --reset=<mode> replaces both, and the mode is required.--reset=<mode> is what puts the file back in charge, and the mode is required: shep asks you to name the destruction rather than guess it. There are two axes rather than one. env is either kept or put back; everything else is either left alone, put back to what the template declares, or put back to what a fresh start off that template would give even the keys it never mentions. Four combinations are worth having:
instances is held out of even this: a plain load never reshapes a flock.--reset=filePuts back what the template declares, env kept, and leaves every key the template is silent on alone. Nothing to put back for an undeclared key means nothing changes for it.--reset=policyPuts back every setting but env, declared or not, the instance count included, which is the one that can delete a process. A setting the file never mentions goes back too, to whatever a fresh start off that file would give it.--reset=envPuts back only env and touches nothing else, so a policy tuning survives it.--reset=allEverything goes back to the file, env and the instance count included, and the sheep's override record is dropped.dogs merges the same way: one field, established whole. A changed [app.dogs.<dog>] table is left alone by a plain load and by --reset=env. --reset=file, --reset=policy or --reset=all puts it back.
env is the axis that splits data from policy. env is the one field holding something an operator supplied rather than tuned, and a reset that took an app's database credentials away as a side effect of putting its restart budget back would be a different kind of event. Resetting policy is recoverable; resetting data is not. policy and env exist for the operator who wants exactly one of them; all is for the operator who means both.
Every mode is refused outright when the target supplies no template. Naming a sheep is one: --reset=policy needs a Flockfile to reset to; web names a sheep, not a file, exit 2, with the mode you typed echoed back. A bare script path is the other, for the same reason and with the same exit code, since the command line is not a template either. There is nothing to reset to in either case, and doing nothing quietly would be worse.
Three of the four modes reshape the flock, and that can kill
instances is the one setting that changes the shape of a flock rather than the value of a field, which is why a plain load holds it out of scope however unestablished it is. file, policy and all do not. Each puts the count back to what it resolves to under that mode, and putting a count back down deletes the instances above it, through the same path shep delete takes. Only env leaves it alone, the same as a plain load.
file and the other two resolve an undeclared count differently, and that difference is the whole reason file exists. A Flockfile that declares instances gets it applied under any of the three, the same as any other declared key.
instances still says something about it under policy or all: it says 1, because that is what a fresh start off that file would run. So an app you scaled to four with shep stock goes to one against a file that has no opinion about instances at all. --reset=file is the mode that survives exactly this case: it only puts back what the template declares, so an undeclared instances is left running at whatever count it already had. Write the count you want into the template and --reset=file applies it like any other declared field.$ shep flock ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 0 web:0 online 96556 0 - - - 1.2M 2s - - 1 web:1 online 96578 0 - - - 1.2M 2s - - 2 web:2 online 96579 0 - - - 1.2M 2s - - 3 web:3 online 96580 0 - - - 1.2M 2s - - $ shep start Flockfile.toml --reset=policy notice[start]: web: applied instances notice[start]: web are already running; `shep restart web` replaces them. ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 0 web:0 online 96556 0 - - - 1.2M 2s - - 3 web:3 online 96580 0 - - - 0B 2s - - $ shep flock ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 0 web online 96556 0 - - - 1.2M 5s - -
Nothing there is a bug and nothing was silent: the report names instances among what it applied. --reset=file against the same file would have left all four running, because the file says nothing to put back. Against a file that declares instances = 2, file would have settled at two, the same as policy or all would have.
The table start prints is a snapshot taken while the departures are still leaving, which is why it can still show a row on its way out with 0B against it. The shep flock underneath is the settled count.
Empty values are the point
env = { DB_HOST = "", DB_PASSWORD = "" } is doing what a committed .env.example does. It says which variables the app needs without putting a value for either one in a repository other people can read. The first load establishes both keys, and from that moment the file cannot write to them however often it is loaded:
$ shep start Flockfile.toml notice[start]: web is already online; `shep restart web` replaces it. ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 1 web online 96368 0 - - - 1.2M 3s - - $ shep start Flockfile.toml --reset=all notice[start]: web: env waiting on the next spawn (`shep reload web` promotes them) notice[start]: web is already online; `shep restart web` replaces it. ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 1 web online 96368 0 - !1 - 1.2M 3s - -
The first command there set DB_HOST to a real hostname in the Flockfile and loaded it. Nothing happened, and nothing was supposed to: the key was established, so the template no longer speaks for it. --reset=env or --reset=all are what hand env back to the file; env is the one thing --reset=file and --reset=policy never touch.
A blank string is the right pattern for a plain value, but a value that is actually a credential has a better one now: env = { DB_PASSWORD = "{{secret:DB_PASSWORD}}" } instead of an empty string. Secrets reads that reference at spawn time from its own store, so the Flockfile's env key is established once and never needs editing again to fill it in.
Registering before starting
shep add takes the targets shep start takes and runs the same load. The one difference is that nothing spawns: the app lands registered and stopped, with the keys its file declared already established, and shep start <name> is what brings it up.
$ shep add Flockfile.toml ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 0 web stopped - 0 - - - - 0s - - $ shep flock ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 0 web stopped - 0 - - - - 0s - - $ shep start web ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 0 web online 4745 1 - - - 32.0K 0s - -
The empty values above are why it exists. Starting that file spawns a process against an empty database URL: it crashes, autorestart spends the restart budget on it, and the app has to be stopped before anyone can configure it. Registering first puts the steps in the order they belong.
The 1 under RESTARTS is not something add did. shep start <name> brings a stopped sheep up through the same request shep restart uses, and has counted it that way since before this verb existed.
The middle step is an assignment on the start that brings it up, and it works for a key the file established as well as one nobody has spoken for:
$ shep add Flockfile.toml ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 0 web stopped - 0 - - - - 0s - - $ shep start DB_HOST=db.internal web ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT 0 web online 16628 1 - *1 - 32.0K 0s - -
--reset=env is the other direction, handing env back to whatever the file says and touching nothing else, not even instances. A key holding a {{secret:...}} reference needs neither: the Flockfile is written once and shep secret set carries the value.
Running shep add again changes nothing and prints nothing, the way a second load of an unchanged file does. Naming a sheep instead of a file says so out loud, because a name reads no Flockfile at all and leaves nothing to register: notice[add]: web is already registered; nothing to add.
Setting one from the command line
An assignment in front of a target is an override, written the way a shell writes one:
$ shep start KOJI_TOKEN=s3cret ./koji --name koji
ID NAME STATUS PID RESTARTS EXIT CFG CPU MEM UPTIME FOLD SMIT
0 koji online 9970 0 - *1 - 32.0K 0s - -
$ cat $SHEP_HOME/overrides.json
{
"version": 1,
"apps": {
"koji": {
"fields": {
"env": {
"KOJI_TOKEN": "s3cret"
}
},
"declared": [],
"declared_env": []
}
}
}The value lands twice and both halves matter. In the sheep's env, so the first spawn carries it and a later shep stop koji and shep start koji need no assignment. And in the store above, so a Flockfile that later declares KOJI_TOKEN is skipped rather than obeyed. Without that second half a template arriving through the app's own repository would win at the next spawn, which is the thing this page exists to stop.
The name follows the shell's rule: a letter or _, then letters, digits or _. That is what makes ./A=1 a path and 1A=1 a target rather than an assignment, so a file called A=1 is reached here the same way it is reached in a shell. Only the first = separates the two, so A=x=y is the value x=y, and a repeated name takes its last value.
One target, and it is a script path or a sheep the flock already has. A Flockfile or a fold can name several sheep and an assignment names none of them, so both refuse. So does a single quoted "A=1 ./koji": quoting is what stops a shell splitting, and splitting it a second time would need shep to invent its own quoting rules for a value holding a space.
ps for as long as that command runs and in this shell's history for good. It is not in ps for the sheep, which holds it in env rather than args. For a credential, write KOJI_TOKEN={{secret:KOJI_TOKEN}} and let secrets carry the value instead.A template may add, never overwrite
A Flockfile can introduce a setting nobody had. It cannot change one somebody has. A person at a terminal, holding a flag, can do both.
The awkward consequence is deletion. Removing a line from a Flockfile does not remove the setting, once the setting is established: the file has stopped mentioning it, and a load that acted on an absence would be overwriting by omission. --reset=policy or --reset=all puts a deleted setting back to shep's own default, since it is now a key the file is silent on; all does the same for a deleted env key.
Where it is kept
$SHEP_HOME/overrides.json, one entry per sheep name, mode 0600, written through the same read-modify-rename under an exclusive lock the KV store uses. Keys are in sorted order, so two writes of the same content produce byte-identical files and the store diffs like any other small config file:
$ cat $SHEP_HOME/overrides.json
{
"version": 1,
"apps": {
"web": {
"fields": {},
"declared": [
"cwd",
"max_memory",
"name",
"script"
],
"declared_env": [
"DB_HOST",
"DB_PASSWORD"
]
}
}
}declared and declared_env are not overrides. They are the record of what the Flockfile has established, kept so a later load can tell "the file used to say this and no longer does" apart from "the file never mentioned it". fields is the other half: settings an operator has taken ownership of, whether or not the current file declares them, which is what a *N in CFG counts.
Two doors write fields. The first is the config pane: e on a sheep in shep lookout, behind --allow-control. One key at a time, and the value is recorded as yours rather than spent, so the key keeps its * and a later shep start of the app's Flockfile leaves it alone. The second is a leading NAME=VALUE on shep start or shep add, which records the same way and reaches env alone.
Neither is a flag, and there is still no flag on shep start that sets an arbitrary field: a load is a template arriving through a pull request, and an operator's own value is not a thing a template says. An assignment gets past that because it is the operator speaking rather than the file.
A pane edit is not a --reset=file at one key, which is what it used to send. That merge is a template load, so it moved the field correctly and then dropped the override for it, and the * the operator had just earned never appeared.
shep import env is a third door, beside a Flockfile load and a pane edit: it writes a sheep's env override directly, in one write, for every key the file held or the narrower set --only names. A key already holding the same value is reported unchanged rather than rewritten. The keys it writes survive a later template load the same way a pane edit's do. A key it did not mark --secret is stored in the clear, and is copied into flock.json and the handover blob with the rest of the sheep's config, same as any other override. See Secrets.
A plain value's own {{ and }} are doubled going in, so what lands in overrides.json and flock.json is not what the .env wrote: a line reading hello {{world}} is stored as hello {{{{world}}}}. Reading or hand-editing either file, expect the doubled form.
A store carrying a version number higher than this shep understands is refused rather than read or rewritten. It holds an operator's live edits with no copy anywhere else, and there is no undo for a downgrade that overwrites it.
When a load refuses
Per app, and the rest of the file still loads: one bad app in a file of eleven costing the other ten their load is the failure that would make people stop running this against anything real. What changes is the exit code, which goes non-zero the moment any app was refused, so shep start Flockfile.toml && deploy stops instead of carrying on with a config that did not land.
Every refusal is a sentence naming what happened, and several of them name the remedy. They share one exit code, 4, because the sentence is what travels on the wire and matching on its exact wording would break the first time anybody improved it.