A Homelabbing Experiment

After around 4 days of frustrating, yet rewarding work, I have set up a reasonably stable “homelab” server running the following stack:

  • WireGuard for all connectivity
  • Planka for personal project management and to-do lists
  • Synapse/Matrix to connect all of my messaging services together in one place
  • Nextcloud for a personal “Google Drive” alternative
  • Caddy to route requests to these different services via “reverse” proxy
  • dnsmasq to allow me to access services by subdomains (e.g. planka.home.arpa vs. nextcloud.home.arpa)
  • Docker Compose to rule them all

I made many mistakes along the way due in part to the particular technological limitations of each solution, including some limitations which are entirely undocumented. With this writeup I hope to save fellow first-time homelabbers a bit of pain.

The WireGuard Server

WireGuard is an extremely lightweight traffic tunneling protocol that provides encryption and private networking with a beautifully low attack surface. Most people would probably call it a VPN.

The problem with WireGuard is that it’s so lightweight and elegant that troubleshooting is nearly impossible. Logging can be enabled, but only at the kernel level; most of my debugging consisted of installing tcpdump on my server and trying my hardest to find any traffic bound in for the right port at all.

Part of the problem was my choice in VPS. Initially I used Linode with a VPC (virtual private cloud) which provides private networking for a group of servers. I wanted to put WireGuard on one server, and individual servers on the others. That would have been an elegant solution - but Linode doesn’t allow arbitrary inbound traffic to the server in the VPC that you designate as its router. You’re allowed ports 22, 80, 443, and a few other well-knowns - which doesn’t help when deconflicting WireGuard with other services.

So I took it all down and put it on Vultr. Running on a single server. Every service works like that. And the reason it works so well is because I use Docker Compose - but I’m getting ahead of myself.

I recommend this WireGuard installation script. It sets up the initial server configuration and manages the installation for you afterwards. You can even modify the base configuration it generates by changing /etc/wireguard/params - which is important, because we’ll be hosting DNS on this server too (so change CLIENT_DNS_1 to the server’s IP address on the WireGuard interface).

Docker Compose

In theory, the only server software I have running outside of Docker is WireGuard. In practice, I confused myself while configuring dnsmasq (DNS server) and ran it on bare metal to troubleshoot, which ended with me resolving the core issue. Since I already had it running, I vowed to “come back to it later” and move it to a Docker container. That was about 5 months ago.

I set up each service (besides dnsmasq) in its own folder under /opt. Each service folder under /opt minimally contains a docker-compose.yml file. More often, they also contain the configuration files which the service uses (mounted by the Compose file as read-only in the container itself) and the caches/persistent storage needed by the container (excluding Nextcloud, which as of writing uses about 350 GB of storage and as such mounts via external hard drive).

Additionally I set up SystemD files for each Composified (Composited? Composed?) service that consisted of a rather simple set of commands:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[Unit]
Description=SERVICE_NAME
Requires=docker.service
After=docker.service

[Service]
Restart=always
User=root
Group=docker
WorkingDirectory=/opt/SERVICE_NAME
ExecStartPre=/usr/bin/docker compose -f docker-compose.yml down
ExecStart=/usr/bin/docker compose -f docker-compose.yml up
ExecStop=/usr/bin/docker compose -f docker-compose.yml down

[Install]
WantedBy=multi-user.target

I’m sure you could automate this. But I wasn’t going to do that with only 4 services running on Compose.

Let’s dive into each service.

Planka

We’re starting off easy with a configuration as simple as just a docker-compose.yml file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
services:
planka:
image: ghcr.io/plankanban/planka:1.17.4
restart: on-failure
volumes:
- user-avatars:/app/public/user-avatars
- project-background-images:/app/public/project-background-images
- attachments:/app/private/attachments
ports:
- 127.0.0.1:XXXX:1337
environment:
- BASE_URL=https://planka.home.arpa
- DATABASE_URL=postgresql://postgres@postgres/planka
- SECRET_KEY=nottellingu
- [email protected]
- DEFAULT_ADMIN_PASSWORD=notmypassword
- DEFAULT_ADMIN_NAME=Your Name
- DEFAULT_ADMIN_USERNAME=yours
depends_on:
postgres:
condition: service_healthy

postgres:
image: postgres:14-alpine
restart: on-failure
volumes:
- db-data:/var/lib/postgresql/data
environment:
- POSTGRES_DB=planka
- POSTGRES_HOST_AUTH_METHOD=trust
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -d planka"]
interval: 10s
timeout: 5s
retries: 5

volumes:
user-avatars:
project-background-images:
attachments:
db-data:

As described by their official docs, the SECRET_KEY is OpenSSL generated (openssl rand -hex 64).

I expose the host-side port (XXXX) arbitrarily (remember, 1337 is what the container believes its exposing). As long as this value is unique system-wide, it can be anything you’d like. Just avoid conflicts with common names. And, ideally, leave 80 and 443 open for caddy (set up later). Note also that I prefaced the ports with 127.0.0.1 to force binding to localhost as opposed to 0.0.0.0 (exposing to outside world).

Quite security sidenote on POSTGRES_HOST_AUTH_METHOD=trust: trust means “skip authentication.” Yes, really. I can use it here because this postgres container isn’t being exposed back to the host - it’s purely accessed by the planka server, and only has one database being used by that server, at which point it doesn’t really matter if a password is being used or not. If you wanted to go down a more computationally-conservative route and set up one postgres container for use by every service on your system… you really would want authentication for that. What I’ve done instead is set up one database container per service.

Nextcloud

This one is a bit more involved. Here’s my docker-compose.yml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
services:
db:
image: mariadb:11.3
restart: unless-stopped
command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW
volumes:
- ./db_data:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD
- MYSQL_PASSWORD
- MYSQL_DATABASE
- MYSQL_USER

app:
image: nextcloud:29
restart: unless-stopped
ports:
- 127.0.0.1:XXXX:80
links:
- db
volumes:
- /mnt/blockstorage/nextcloud/nextcloud_data:/var/www/html
environment:
- MYSQL_PASSWORD
- MYSQL_DATABASE
- MYSQL_USER
- MYSQL_HOST

cron:
image: nextcloud:29
restart: unless-stopped
volumes:
- /mnt/blockstorage/nextcloud/nextcloud_data:/var/www/html
entrypoint: /cron.sh
depends_on:
- app
- db

A nifty feature of Docker Compose (podman untested) is that you can provide names for required environment variables but not values (as I’ve done for every environment: block here) and it’ll hunt for alternative sources. In my case, it’s a .env file in the same directory (although if any of those variables were defined in my environment already, Docker wouldn’t bother with the .env file at all).

The Compose file specifies the database will store its persistent data in the local directory but everything else in the “Block Storage” option I lease from my VPS provider. It also creates two containers using the same image - the difference is in their entrypoint (Nextcloud requires a maintenance task scheduler, and conveniently the Nextcloud image provides a cron setup for that purpose).

While I do keep it under /opt/nextcloud for convenience, the config.php file that controls most Nextcloud settings is not mounted locally. I had some bizarre issues whenever I tried that, and - yet again - told myself I’d “come back to it later” 5 months ago (in hindsight it was perhaps that I mounted it as read-only when it must, in fact, be writable). Here’s that config.php:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
<?php
$CONFIG = array (
'htaccess.RewriteBase' => '/',
'memcache.local' => '\\OC\\Memcache\\APCu',
'apps_paths' =>
array (
0 =>
array (
'path' => '/var/www/html/apps',
'url' => '/apps',
'writable' => false,
),
1 =>
array (
'path' => '/var/www/html/custom_apps',
'url' => '/custom_apps',
'writable' => true,
),
),
'upgrade.disable-web' => true,
'instanceid' => 'redacted',
'passwordsalt' => 'hush',
'secret' => 'terces',
'trusted_proxies' =>
array (
0 => 'yourserverip',
),
'trusted_domains' =>
array (
0 => 'nextcloud.home.arpa',
),
'datadirectory' => '/var/www/html/data',
'dbtype' => 'mysql',
'version' => '29.0.1.1',
'overwrite.cli.url' => 'https://nextcloud.home.arpa',
'overwriteprotocol' => 'https',
'dbname' => 'nextcloud',
'dbhost' => 'db',
'dbport' => '',
'dbtableprefix' => 'oc_',
'mysql.utf8mb4' => true,
'dbuser' => 'nextcloud',
'dbpassword' => 'thatsasecret',
'maintenance_window_start' => 9,
'installed' => true,
'loglevel' => 2,
'maintenance' => false,
);

Many of the options here are designed to mitigate the problems that crop up when you reverse proxy through Docker and Caddy.

P.S. Whenever the config file doesn’t cut it, you can run the OCC utility via docker exec --user www-data nextcloud-app-1 php occ [rest of command]. If nextcloud-app-1 isn’t your container’s name, replace that part.

Synapse

I’m most familiar with Synapse out of any of these other services, so I made a more thouroughly customized setup. Here’s my compose file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
services:
synapse:
build:
context: .
dockerfile: synapse.Dockerfile
restart: unless-stopped
environment:
- SYNAPSE_CONFIG_PATH=/data/homeserver.yaml
volumes:
- ./files:/data
depends_on:
- db
ports:
- 8448:8448/tcp

db:
image: docker.io/postgres:12-alpine
environment:
- POSTGRES_USER=synapse
- POSTGRES_PASSWORD=nope
- POSTGRES_INITDB_ARGS=--encoding=UTF-8 --lc-collate=C --lc-ctype=C
volumes:
- ./schemas:/var/lib/postgresql/data

mautrix-discord:
image: dock.mau.dev/mautrix/discord:latest
restart: unless-stopped
volumes:
- ./mautrix-discord:/data
depends_on:
- synapse
- db

mautrix-telegram:
image: dock.mau.dev/mautrix/telegram:latest
restart: unless-stopped
volumes:
- ./mautrix-telegram:/data
depends_on:
- synapse
- db

mautrix-whatsapp:
image: dock.mau.dev/mautrix/whatsapp:latest
restart: unless-stopped
volumes:
- ./mautrix-whatsapp:/data
depends_on:
- synapse
- db

As you can see, I’ve got a lot going on here - like a custom-built Synapse image via Dockerfile. Here’s that synapse.Dockerfile:

1
2
3
FROM docker.io/matrixdotorg/synapse:latest

RUN pip install synapse-auto-accept-invite

Coupled with a config change to include the correct module, the image built from this Dockerfile automatically accepts invites to new chats (appropriate in my case where I am using this just to bridge to my other services, and not publicly exposing my Synapse instance).

You may see that I was rather uncreative with the ports in this one (8448:8448/tcp being the default). Unfortunately I underestimated the difficulty of changing those values after the server went online. Eventually I allowed Synapse to win that particular fight and left the ports alone, but if you change it before you set anything up I don’t see how it could fail.

I’ve also included a bunch of bridges in this Compose file - I won’t include their configurations here in the interest of space. There’s nothing really Docker-specific about them besides basic setup. That’s also true of the main Synapse homeserver.yaml config file.

Only other thing I should mention is that all the bridge registration files were copied under /opt/matrix/files, which is linked as /data in the container. That made things a lot easier, if slightly less automated (as opposed to setting up mounts to each bridge’s configuration folder, which ended in disaster when I first tried it).

Caddy

Caddy, similar to WireGuard, is just used to provide infrastructure. Think of it as a switch for the web traffic reaching your server - it proxies the correct service depending on the subdomain you access it with. Here’s the Compose file I use:

1
2
3
4
5
6
7
8
9
10
11
services:
caddy:
image: caddy:2-alpine
cap_add:
- NET_ADMIN
volumes:
- ./data:/data
- ./config:/config
- ./certificates:/var/certs
- ./Caddyfile:/etc/caddy/Caddyfile
network_mode: host

Providing the NET_ADMIN capability and network_mode: host mitigated every issue I experienced - without those two concessions, Caddy ended up being unable to access all the other ports exposed to the host system, so there was no reverse-proxying at all. It is unfortunately less containerized than the other services because of this.

Here’s my Caddyfile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
planka.home.arpa {
tls /var/certs/server.crt /var/certs/server.key

reverse_proxy localhost:XXXX
}

matrix.home.arpa {
tls /var/certs/server.crt /var/certs/server.key

reverse_proxy /_matrix/* localhost:8448
reverse_proxy /_synapse/client/* localhost:8448
}

nextcloud.home.arpa {
tls /var/certs/server.crt /var/certs/server.key

request_body {
max_size 1TB
}

reverse_proxy localhost:XXXX
}

You’ll notice that I’m manually specifying certificates… even though the main selling point of Caddy is that HTTPS is automatic. Unfortunately that doesn’t extend to self-signed certificates (required in the case of localhost/home.arpa domain usage). Here’s the script I used to generate those certificates:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#! /usr/bin/env bash

# This script is presented as an archived version of the commands which
# succeeded in setting up a self-signed CA with a wildcard certificate for
# the home.arpa domain.

# Assumes the existence of a correct openssl.cnf in the CWD.

set -e

openssl genrsa -out ca.key 4096
openssl genrsa -out server.key 2048
openssl req -new -nodes -key server.key -out server.csr -config openssl.cnf
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -out ca.pem -config openssl.cnf
openssl x509 -req -in server.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out server.crt -days 825 -sha256

And my openssl.cnf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = req_distinguished_name
req_extensions = req_ext
x509_extensions = v3_req

[ req_distinguished_name ]
C = US
ST = XX
L = Anytown
O = Your Name
CN = *.home.arpa

[ req_ext ]
subjectAltName = @alt_names

[ v3_req ]
subjectAltName = @alt_names

[ alt_names ]
IP.1 = X.X.X.X

Where IP.1 is the server’s IP address under the WireGuard interface.

dnsmasq

As mentioned before, I have dnsmasq running on bare metal. Here’s my configuration for that:

1
2
3
4
domain-needed
bogus-priv
log-queries
server=1.1.1.1

“But wait,” I hear you asking. “Shouldn’t it define records? Isn’t it just passing all traffic to 1.1.1.1 and logging it?”

dnsmasq actually reads /etc/hosts and creates records from that on the fly. That’s why I love dnsmasq: all of the domain name resolution, none of the zone files. (All my memories of bind9 are universally negative.)

My /etc/hosts reads as follows:

1
2
3
4
5
6
127.0.1.1 wireguard wireguard
127.0.0.1 localhost
X.X.X.X wg.home.arpa
X.X.X.X planka.home.arpa
X.X.X.X matrix.home.arpa
X.X.X.X nextcloud.home.arpa

Where X.X.X.X is your server’s IP address under the WireGuard interface. You just repeat it for every applicable service, and let Caddy do the actual work from there.

(server=1.1.1.1 is the fallback DNS server for when your clients try to access anything besides the homelab services.)

Summary

It’s been great running my own “homelab” server via VPS. I get to manage my own infrastructure, have reasonable faith in the software that handles my most personal information, and learn some new things along the way.

There’s also the great cost savings! I used to pay $22 per month combined for all these services (where they even had costs to begin with) and now that it’s on my VPS, I pay… $36 per month.

Admittedly cost efficiency didn’t factor into my decision to try “homelabbing.” And it will be significantly cheaper once I actually get to remove the scoff quotes from “homelabbing” because I plan to migrate to “on-prem” (once I move to a new place I’ll buy a PC and run it off that). Most of my expenses currently are towards storage ($18 per month for 400 gigabytes through my current VPS provider) which I can eliminate by plugging in a 1 terabyte hard drive into my real (planned) homelab setup.

But aside from that one hang up I do thoroughly enjoy this setup and I doubt I’ll move back to the old way anytime soon. Hopefully my documentation here can help more people set up their own “homelabs” (or real ones too!).


“WireGuard” and the “WireGuard” logo are registered trademarks of Jason A. Donenfeld.

Why Is (Almost) Nobody Talking About XZ Utils?

I learned about the XZ Utils backdoor around 2 hours after it was published on YCombinator.

Around 5 hours after it was published, I did a quick Google News search for terms related to it — “xz utils,” “Linux,” “cybersecurity” — and got nothing.

Only after around 8 hours did I actually see a semi-mainstream publication write about it: Ars Technica. As of publication time, they are the largest publication I’ve seen write about what may be the worst out-and-about software vulnerability we see this year, with startling implications for national cybersecurity.

  • It has a base CVE score of 10, the absolute maximum rating a vulnerability can receive;
  • It was slowly and steadily implanted into a widely-used library depended on by the most popular Linux init system (by far), using obfuscation techniques both in the code itself and in its distribution (such as being absent from the source and only existing in the tarballs used for building system packages) to execute what has been rightly described as “the best executed supply chain attack we’ve seen described in the open” and “a nightmare scenario”;
  • The library used to stage the attack only came into the possession of the attackers after what was described as a manufactured “hostile takeover” of the maintainership for that library. Before it was taken offline by GitHub, I looked at the repository for this incredibly widely-used tool: it had less than a hundred stars. One of the most widely used implementations of a favorite compression algorithm had less than a hundred GitHub stars, but major distributions pulled from that repository - again, as a dependency for a required system utility. It makes me wonder how many other major libraries have almost no reputation and oversight but are still getting pulled into major distributions.

This is the biggest supply chain attack since SolarWinds, but the largest publication to talk about it so far has been Ars Technica, even though this was discovered 2 days ago already.

This isn’t just some technical babble that the cybersecurity community is going on about. This is a very real and dangerous threat that (while we’re in the process of mitigating) could affect up to possibly 30% of extant Linux servers. Even more worrying than the immediate problem is the set of circumstances that allowed it to occur: the blind and naive trust given to a library with almost no oversight or reputation, the lack of testing done on systems before deployment (the backdoor in this case was only discovered after an ordinary user benchmarked a completely different tool, not because Linux distribution maintainers caught it), the ability for a compression library to inject code into a remote server administration tool… and so on.

At some point I’d like to write more about that last point, because of how plainly absurd it is for a compression tool to have this kind of effect at all. But for now what baffles me is that mainstream press is just not talking about it.

Is it because they don’t know how to? I doubt it — something among the lines of “hacker plants virus into widely-used server software” would do the trick pretty well for an entirely un-technical audience. Is it because they don’t find it newsworthy? Again, if the high-end estimate of compromised servers is 30%, that sounds pretty newsworthy to me! SolarWinds received tons of news coverage, which makes this even more surprising because depending on what happens in the next few months, this could be worse than SolarWinds! After all, SolarWinds’s customers were primarily government (which was why they were targeted), but Debian and Fedora’s customers are everyone. While the vulnerability is only confirmed present in the unstable versions of both, that doesn’t mean the others don’t have that vulnerability — actually, it makes it more likely. The malicious maintainer who uploaded these backdoors has had access for a long while. What if a different vulnerability is in the other versions, or what if the tarballs for those versions were retroactively infected by this malicious maintainer?

My guess is that mainstream outlets just don’t have the staff or bandwidth in their coverage for this kind of news. It seems to me like the same problem befalling educational institutions: however much these places pay their staff, it will never compare to how the industry itself pays the same kind of staff. I could become a high school Computer Science teacher or a college Cybersecurity professor making somewhere between $60-$80,000 a year, or I could work in a SOC for a 6-figure starting salary. I imagine the same is true for reporting.

This lack of security coverage leads directly to the mystification of the “hacker” as a kind of techno-magician that can take over large swaths of society’s computing power at will, and subsequently leads to a sort of learned helplessness among the general public. People believe that there’s nothing we can do to stop hackers; that everyone’s personal information will be inevitably leaked; and that it’s worthless to try and stop it. None of these things are true! But a person who believes in these falsehoods will inevitably not try to prevent their predictions from coming about, leading to a self-fulfilling prophecy. This could be changed if better security coverage existed to educate the general public about the true nature of cybersecurity and of cyber-attacks, but without competitive pay in the reporting industry I doubt that will happen anytime soon.

The Martyrdom Contagion

The rhetoric we have been seeing regarding the שמחת תורה (Simchat Torah) war about “ceasefire,” “genocide,” “apartheid” and the rest did not start when Israel invaded Gaza.

It started when Gaza invaded Israel.

The overwhelming popular response to the death of 1,200+ Jews was to celebrate our deaths, and then quickly pivot to condemning us for either existing or for responding to the attack. Sometimes they did not wait before performing this second part, combining celebration and protest into one.

Yes, the same people protesting the humanitarian crisis in Gaza today — and ascribing blame for it to the Jews — celebrated our deaths. Human rights for thee, not for me.

This made it even more baffling to me when former allies started joining the crew in droves. Don’t they see who they are aligning themselves with?

Maybe they do. But they also see the dead bodies of the forced martyrs in Gaza.

Yet this is not deception and these people are not deceived. It is 2023, where the only thing stopping our former friends from learning about Hamas’s human shielding tactics, from learning basic international law, from learning the definition of genocide — is a few button clicks.

No, these people are not deceived. They are choosing to know lies over truth, fictions over facts. Because only the fictions can give our new killers their license to hunt.

So the new — no, the old — story goes, “it is good to kill Jews because they are killing Gazans.” Or, better yet, “the October 7th pogrom was justified because of the oppression of Gazans.” Nevermind our disengagement and evacuation from the Strip almost a decade ago. Nevermind that the blockade was only imposed after a hostile takeover by a terrorist organization. Nevermind that Egypt enforced the blockade just as much as we did. No, those facts must be disregarded, lest they get in the way of killing Jews. Fiction provides the grounding for a new moral code. Except it is not new. It is old, very old.

“We need a ceasefire,” they say, “to provide aid to the civilians.” The Jew knows they mean aid to Hamas, because that is where all aid goes. The Jew knows they mean to allow more time for Hamas to stuff more children into munitions facilities, in command cells operating from hospitals, because then the martyrdom contagion can spread further. The Jew knows it — and the protestor knows it, too. They have it right at their fingertips. How could they not know it? What is more absurd? The suggestion that these pro-Palestinian (more accurately, pro-Hamas) supporters have no idea they are spewing libels against Jews, even though one Google Search confirms it? Or the idea that they do know it — and don’t care?

When was the last time any group chanted “Itbah al-Yahud?” When was the last time any group chanted “Death to Jews?” The new brownshirts do something much more effective: they call for other things that will kill Jews instead of asking for it directly. They cloak their death-talk in “social justice” or “human rights” language to legitimize the feelings of the outsider looking in: the closet antisemite yearning for the chance to be free. They spread their martyrdom contagion far and wide.

This contagion is a virus like no other. It is the only one where you must choose to be infected. It is an entirely voluntary procedure, undertaken by people who otherwise exercise basic critical thinking and research tactics — just not for Jews. We get nothing but regurgitated blood libel.

What did the old brownshirts protest? Jewish existence? Only in their hearts. To the rest of the world, they protested economic conditions, political dysfunction, Communism, and more. The German voter knew it was all a cover for the true agenda — Hitler had already published Mein Kampf, explicitly describing himself as an annihilationist antisemite! — yet they voted for the party anyway. Why? Haven’t I told you why?

Hitler’s journey, his Kampf, began with tolerance and good naturedness to the Jewish people, becoming antisemitic over time, eventually deciding on genocide as the final solution. Today’s new guard of brownshirts are individually retracing his steps.

What do the new brownshirts protest? Humanitarian conditions, perceived political dysfunction, Zionism, and more. They don’t need to say “Itbah al-Yahud” when they can say “from the River to the Sea.” We fill in the blanks.

The natural question is: is their variety of Jew killing brand new, or is it old? Ask them: they will tell you. It is not a coincidence that the new volk parrot the talking points of “neo”-Nazis; not a coincidence that when the Nazis come to their rallies and marches, they are disavowed after the fact and not during, because their speech and rhetoric is identical as to be completely indistinguishable from their own.

The American education system has failed. All of the Holocaust education was worth nothing. All of the anti-bigotry and multicultural awareness was meaningless. The whole point was to ask the young people, “if you were there, what would you have done?” and for them to answer “I’d defend the Jews!” like good students. A+ on your “No Jew Hating” quiz. Years later now, they were given the same quiz again.

Their new — no, their old, their ancient — answers?

“End Zionist Apartheid!”

“Khaybar, Khaybar, oh Jews!”

“Ceasefire Now! End Zionist Genocide!”

“Itbah al-Yahud!”

Death to the Jews!

We gave the education system the gravest task in its history — end the oldest hate in the world — and it failed. Students from my high school, having the greatest Holocaust education I have ever seen, have joined the legion of brownshirts. One of them reposted Norman Finklestein, the Holocaust denier. Should I be surprised?

The youngest Americans, in recent polling, are near-evenly divided on whether to support Hamas more than Israel. Should I be surprised? The oldest hate now dominates this country, and I have never felt this close to my father’s father, living in a Polish village as the jaws of his people’s executioner drew ever nearer, silently, as his fellow Yidden echoed the Nazi siren song, telling him he had nothing to fear. Should I be surprised?

I am not. I am not surprised that the same story we have been told for 3000 years is being told again. I am not surprised that former friends have joined the brownshirts to courageously lead the charge against my existence, from the comfort and safety of euphemisms and doublespeak. I am not surprised to be just as disposable now as I was in 1939.

The only ones who will be surprised are the ones who thought their assimilation would save them.


“And then, suddenly, one day in 1939, it all ceased to be.” - My grandfather Aryeh

UCF Finally Responded to ChatGPT

UCF’s Vice President for “Student Success and Well-Being” recently pushed a campus wide announcement, shown below:

“What Students Should Know About ChatGPT and Other AI

Chat GPT and other artificial intelligence (AI) tools can be helpful resources for research, brainstorming, and assistance. However, as a student at UCF, it is important for you to understand expectations around how these tools should be used.

How does ChatGPT work?

ChatGPT and other AI chatbots are trained to conduct thorough searches of information available across the Internet and use that information to build statistically-probable responses to prompts. This results in a seemingly correct answer, but it is important to double-check chatbot “facts.” These systems can invent facts, names, quotes, titles, and more, presenting them with confidence as truth. Chatbots are word predictors, not verified sources of accurate information.

When should I use ChatGPT? When should I avoid it?

UCF does not have an established policy on ChatGPT. Some of your faculty may allow you to incorporate AI in your assignments, while others may ban it. It is your responsibility as a student to understand each of your faculty members’ policies. When in doubt, ask your professor, or play it safe and assume that use of AI is banned.

If use of AI is permitted, you should still never submit a ChatGPT or other AI platform’s words as if they are your own. Attempting to pass off work created by another source as your own is plagiarism, and plagiarism puts your academic future at risk.

How would plagiarizing using AI be caught?

Detection to recognize AI writing is built in to UCF’s Turnitin tool directly within Webcourses. This detector produces an “AI score” the same way they get an “originality score.”

Can I use ChatGPT for help with online exams?

No. Faculty use a lockdown browser (which does not allow new browser tabs) and an electronic proctoring solution with online exams. The proctoring software records you via webcam and records what is on your screen.

Knights, you are here at UCF to earn a degree that demonstrates your readiness to enter the workforce. Tools like ChatGPT and AI can assist you on that journey, but they are not shortcuts to reading, studying, and learning that will help you be prepared for life after college.

Should you choose to use these tools, proceed with caution, and understand the expectations around academic integrity.

Charge On!”

In the wonderful work of science fiction Foundation by Issac Asimov, a diplomat is called upon by the fledgling Foundation to discuss protection in the face of a threatening, nuclear power developing nearby. The leaders converse with the diplomat and record the statements he makes, and subsequently subject them to logical analysis by translating the recordings into a logical language. They find that at all points where the diplomat makes a statement, he later retracts or refutes that statement (we will supply you with an army… assuming, of course, that the correct permissions are secured from General So-and-so) such that the translation amounts to nothing - literally a blank piece of paper - because nothing was actually said.

Allow me to subject the veep’s message to the same treatment:

“ChatGPT is very useful to students, but sometimes it lies. We will not be establishing a blanket policy on its use, but actually you should assume it is banned, even if nobody tells you outright that it’s banned. You shouldn’t submit ChatGPT’s work as your own, but as an institution we won’t prevent you from doing so. We allow Professors to use TurnItIn to check your work for AI influences, but we make no guarantees about the reliability of TurnItIn (and actually if you look at the research it is about as good at its job as snake oil is good at curing blindness). If you use ChatGPT in an online exam, and our proctoring software catches you, we consider it academic dishonesty, but we’re assuming our online exam monitoring tools will catch you in the first place.”

Now let’s apply some basic mathematics: cancel out every statement that has a “but” attached to it. We end up with:

“ “

Nothing. In this part of Foundation, the leaders of the Foundation become (rightly) terrified.

In that sense, it is not surprising that this veep took 9 months to make a statement about the technology which poses the most credible threat thus far to the hundred-year-old status quo of higher-ed. To craft a statement which mentions the existence of such a threat, and proceeds to say literally nothing about it, requires a lot of work. 9 months is almost early!

(It is also not surprising from another perspective: this veep of “Student Success and Well-Being” presides over an institution which is so notorious for not graduating its students on time that we call it “U Can’t Finish.” I wonder how this veep measures student success… I imagine not by percentage of students who graduate on time!)

Thankfully for UCF, relatively few students (somewhere among 30-40%) have told me that they regularly use ChatGPT to assist their studies. Not so thankfully, when I suggest to the remaining 60-70% that they should use ChatGPT (never to cheat; I am not unethical!) they respond with some variation of “Ahh! Now that’s a good idea!” So it seems that UCF’s time sticking their heads into the sand is running low.

Besides this update, I have nothing more to add regarding the impact of ChatGPT on higher-ed besides what I already wrote: http://milogilad.com/blog/2023/06/18/academics/how-will-schools-respond-to-reckoning/

(You can verify the digital signature of this blog post at this link.)

Trust in the Digital Age

A friend of mine was recently doxxed, but not for just any ordinary purpose. His identity was surreptitiously used to post vile comments to an elected official’s website, along with many other people’s stolen identities. Subsequently, this official released the comments in full, including the stolen personal information of my friend.

While this was by no means the official’s fault, and they were obviously upset at the contents being posted to their website, the subsequent attribution of these remarks to my friend has significantly derailed his life. He lost his job, and then found a Google search of his name now contains all of the comments attributed to him. He has entirely lost control of his online identity, which (in today’s age) means he has lost control of his identity in the real world, too.

My friend managed to convince the elected official that these were not his remarks and he had been doxxed. (I believe he showed her proof that his LinkedIn account had been hacked, making it the source of the personal information included in the form.) I imagine this is the least likely outcome for anyone targeted in this way. Most people attacked by celebrities, either justly or unjustly, do not subsequently gain audience with them.

These terrible effects aren’t the elected official’s fault. The guy who doxxed my friend could have just as easily posted the vile remarks elsewhere. And, for as long as the Internet has been around, trolls have been doing stuff exactly like this. The thing is, if this had happened 10 years ago, I doubt anyone would have believed my friend actually posted those comments. It’s not like they were in character for him! At the very least, people would have asked questions first. What changed?

Widespread Adoption of the Internet, Not Internet Culture

More people use the Internet now compared to 10 years ago for things beyond just web browsing. People have very widely adopted social media, as well as other forms of dynamic content generation. We have increased smartphone usage and even normalized it in situations where it was normally taboo. (Think of schools which adopted the use of Kahoot, companies which issue work smartphones, etc.) In a sense it feels like we’ve succumbed to our addiction of content consumption.

The problem is that the Internet is now increasingly integrated with the real world, without the vast majority of its users being “in” on Internet Culture. The inaccessibility of the Internet back then created a distinct set of norms and expectations among the people who actively engaged with it. It was never “OK” to impersonate someone to say terrible things, but it was something I would have expected from the Internet back then, and I would never have taken these statements at face value.

Most people who use the Internet now are lacking the mutual distrust which comes as part of Internet Culture. They take everything at face value. Including absolutely outlandish statements made by someone with no reason to state them, all because some barely-personal info (name and email address) are included. If that’s the standard for determining if someone really said something, then I could get every last person in my contacts list fired. Imagine how ridiculous it would be if banks allowed you to login using just your name and email address.

Anti-social behavior like this is so easily executed on the Internet that any mode of engaging with the Internet which doesn’t account for it is fundamentally flawed.

It Will Get Worse

The soon-to-be widespread use of deepfakes will make this problem worse. My friend was able to repudiate the statement attributed to him quite easily. If instead he had been deepfaked, would he be able to do the same?

There are telltale signs and AI detection tools which could potentially thwart a similar attack right now. Eventually, probably soon, these signs and tools will not work. Then, there will be nobody to vouch for the innocence of a deepfaked victim, and their exoneration (which is already quite rare) will not take place at all.

The reputational damage and subsequent life-ruining potential of online impersonation does not result from the impersonation itself. It results from the widespread acceptance of the impersonation, which is itself a consequence of blindly trusting what we see. But we should know better than this. Most people’s names and emails are publicly available. People’s social media accounts get hacked all the time. The existence of deepfake technology is widely known. Why do we continue to fall for these traps?

Solutions?

Security nuts (myself included) tend to sign our emails with PGP keys, and after seeing what happened to my friend, I considered extending that to all my written communication. But gpg is cumbersome and PGP keys are endlessly complicated. Subkeys and UIDs and signing vs. encrypting keys make the process rather unadoptable for people with better things to do. (That’s even without considering the fact that gpg is a CLI and there are quite a few GUIs to choose from, which could put people off.)

The best alternative to getting everyone into PGP keysharing is Keybase, which streamlines it for the modern Internet. They have a very easy-to-use signature verification tool which allows you to validate easily. Although their cool and less-bulky Saltpack format can’t be validated online, you can still use it for PGP signatures. And they make it pretty easy to generate PGP keys and signatures.

Are we all going to make Keybase accounts and start signing all of our online correspondence? Of course not. But if you’d like to protect yourself while waiting for people to realize that not everything online is real, you might consider it.

(You can verify the digital signature of this message at this link.)


Update on 9/18/2023: My friend’s story has gotten even wider reach, thankfully. He was featured on his local NBC affiliate: https://www.nbcmiami.com/news/local/i-felt-my-future-was-gone-a-college-students-fight-to-restore-his-reputation/3113321/

How will schools respond to the ChatGPT Reckoning?

I was disappointed that my school did not, as a matter of policy or unanimous agreement, abolish the take-home essay upon the release of ChatGPT. Leave it to a bureaucratic institution to be slow to respond to a situation that destroys a several hundred-year-old status quo. But my naive mind strongly believes they will respond to it, so I may as well speculate on how they will.

Before we begin, I think it’s important to understand how ChatGPT has so much disruptive potential. ChatGPT doesn’t prevent anyone from doing their assignments on their own. It just allows those who don’t want to do their homework to successfully avoid it. Previously, you’d need someone else to do it for you, which meant paying them and getting something of questionable quality. The cost of doing it yourself used to be less than the cost of someone else doing it. Now there’s not even a comparison. Your homework can be done in less than 5 seconds at the exact same quality you’d do it yourself and at no financial cost. The only benefit of doing your own homework is learning, so if you don’t view that as a substantial-enough benefit, you just won’t do it.

The truth is that, thanks to a combination of factors, the vast majority of aspiring white-collar workers view college as necessary and attend not out of a desire to learn but out of a desire to obtain a piece of paper. If shortcuts are available to someone in that situation, they will take them. Computer science majors don’t want to learn about philosophy and psychology (even though everyone should!), and the tools they have available to avoid doing do (sleeping in lecture, procrastination) now include the Get-out-of-Homework Free card. Not to mention the just as important factor that college coursework in technical subjects is deeply inferior in quality to professional certification and on-the-job training.

Many have meditated on the “Death of the Liberal Arts Education” although few I have seen connected it to the prevalence of technical majors in universities. Humanities students come in wanting to learn so I can see why humanities professors are woefully unprepared to deal with students who don’t want to learn. But non-humanities students lack the intrinsic motivation to learn. The solution is either to spark the intrinsic motivation by getting students interested in psychology/philosophy/English/history or provide worthwhile extrinsic motivation to make it feel less like a waste of time for them.

Of course, that’s not what schools are going to do. It would be too effective. Here’s what I think they’ll actually do:

Option 1: Everything in Class

In AP World History, we wrote our essays in class. Some of them were short-answer and some were document-based 3-page long treatises. But they were all done in class. This might be the easiest solution that schools/educators pursue in this post-chatapocalyptic world. If you can’t guarantee your students actually write their essays outside of class, you can sure guarantee they do in class.

Any in-class approach that also allows technology usage would miss the point pretty heavily. Students have been using their phones to cheat in class for years. I can remember finishing my AP Computer Science exam and getting up from my desk only to see around half of my classmates checking their answers under their desks. ChatGPT has a pretty mobile-friendly site.

As far as false solutions (those which address symptoms and not their root causes) go, I like this one. It acknowledges that homework is useless in this day and age and means I don’t have to do it. But while the big focus has been cheating on essays with ChatGPT, Chegg and Mathway have allowed students to cheat in math/physics classes for much longer than ChatGPT’s been around for. The sheer importance of practice in these classes means I doubt they can get away with doing everything in class.

Option 2: Document History

If I was a technically-savvy administrator who wanted to address the symptoms of this problem, I’d find a way around doing everything in class: track everything done out of class.

The student information system would create something like a Google Doc for every assignment, and would automatically submit that assignment at the deadline. No other items would be permitted for submission. The advantage here is that Google Docs track history pretty well - if you copy-pasted from ChatGPT, it would know. If you typed out ChatGPT’s output by hand, it could theoretically see you alt-tabbing way more than normal (yes, your browser does give that info away - which is most notably used by Canvas quizzes).

But this would probably flag source-referencing and research as suspicious activity. Also, alt-tabbing is very easily defeated by another device being present - so unless we want to implement ProctorHub for Essays (which also wouldn’t work for similar reasons) it’ll be very easy to get around this requirement. There’s also a certain absurdity to the idea of implementing document histories for every software which would be used for assignments - Photoshop? iMovie? Visual Studio? Vim?

Option 3: Abolish the Essay!

Down with the graphiarchy! Essays can be cheated too easily? Get rid of them! My ideal English class is a 3 hour Socratic seminar where everyone is forced to participate and graded on… some arbitrary criteria! Those who disagree get hemlock!

Writing teaches very important skills: collecting your thoughts into a coherent argument, understanding how other people will perceive your words (so you know how to properly express your ideas, and maybe even use rhetoric to achieve a desired effect on your reader), and finding research to support your ideas. Eschewing the essay altogether will deprive your students even more than ChatGPT does, because at least your students see the essays it writes. Something is better than nothing.

Socratic seminars are valuable in their own right. They don’t need to crowd out essays as well. It also doesn’t apply to the sciences (try doing a Socratic seminar on Gauss’s law).

(Side anecdote: in my 8th grade English class my classmates ran a fantasy Socratic Seminar league based on participation. I don’t think my teacher was prepared for just how engaged we became after that. The talking-over became progressively more difficult to moderate until the league was officially banned after what amounted to a screaming match to guarantee a win. No money was involved; we were just bored middle schoolers.)

How Long Will We Keep Doing This?

Knowing that technical majors in college are present not out of intrinsic motivation but extrinsic compulsion - if my internship programs didn’t require college enrollment I’d be an idiot to attend school - makes all this seem rather absurd. Doesn’t it spark a conversation in the administrative backrooms when more students cheat to pass than pass legitimately? Plenty of people have written about the concept of grade inflation, which stems from the same apathy towards education pushing universities to make things easier for the students who don’t want to be there. Even more people have written about the financial inflation of tuition and its causes, which further reduces the value of a liberal arts education (“Why are we spending $20,000 each semester on a History degree when your annual salary will be about the same amount?”) while breeding additional resentment amount the technical majors who can afford it… and consider it highway robbery.

It’s not that something has to change, it’s that something will and is changing - just not the colleges. Increasing numbers of employers are dropping college-education requirements. Tesla never had it to begin with (they politely declined my application a semester before I began college). The morons stuck in the college-admin backrooms aren’t going to change, and in that weakness there is wonderous opportunity for those willing to seek it. Traditional companies who partner with colleges to keep tech students trapped in the educational system in order to continually access their talent pools are going to get their asses kicked by companies who provide far more value to their employees by allowing them to press “Skip”. Younger corporate cohorts replicate the college lifestyle without the deadweight homework assignments (or more accurately the deadweight GPT-4 subscription).

The longer this goes on for, the more expensive it is going to get for students to be given the opportunity to use ChatGPT. If companies don’t right now realize this is ridiculous then they will once their smarter competitors undercut them. It will be a race to see how quickly corporations can abandon colleges and appeal directly to their new talent pool, no middleman University need apply. Hopefully then we can get back to the era of dirt-cheap liberal arts education.

On the National Cybersecurity Strategy

As part of my coursework last semester, I wrote a short dissection of the National Cybersecurity Strategy document published by the White House. Since my semester ended a month ago, I feel comfortable sharing my thoughts with the public. Please find the essay reproduced below.

(For added context: the prompt asked me to read the entire document, and inform my readers why the strategy would or would not be effective.)


The National Cybersecurity Strategy document’s welcome focus on economic incentives for cybersecurity investment is tempered by its deference to temporary technological trends (i.e., “Internet of Things”), focus on traditional regulation, and its refusal to elaborate on finer details. On the whole, the document barely proposes anything concrete; as such it can hardly be called “effective.” The author even acknowledges this near the end: “Realizing the strategic objectives outlined in this strategy will require a strong focus on implementation.” Yet what little is actually proposed leaves much to be desired. But some of its general principles bode well for the future should they be adopted by the industry (nobody in the industry will disagree with the statement “A single person’s momentary lapse in judgment… should not have national security consequences”); others, quite the opposite. My assessment is that this document will not be effective because it hardly makes any specific or actionable recommendations; it focuses on heavy-handed regulation instead of softer economic incentives; and it defers to an existing bureaucracy that cannot handle the challenge of strengthening America’s cybersecurity.

Despite the natural pitfalls of only spelling out principles without any indications for practice, many of the general ideas in the document deserve applause (in particular, the section regarding Strategic Objective (henceforth “SO”) 1.2 beautifully outlines the idea of distributed security and its advantages over a centralized response). Yet throughout the document little definitive action is proposed – and consequently little in the way of strategic value. To be singled out for particular ridicule are the requests in SO 1.2 for “enhanced coordination” (which I am certain the target audience shall interpret as a request for an additional email at the end of every fiscal year), in SO 1.3 for the “integration of Federal Cybersecurity Centers” (unless they are empowered similar to law enforcement agencies, I suspect this shall consist of an additional site visit or two), and the description of Internet technologies in the Introduction sectinn as “underlying structural dynamics” (which sounds like something inserted into an essay ChatGPT would write instead of a nominally serious paper about cybersecurity). These examples come from near the beginning of the document but many just like them are peppered throughout, forming the bulk of what is written, as will be noted below.

Regarding trends, both the document’s introduction and SO 3.2 embrace the current trend of putting a microchip in everything (called the Internet of Things). Subsequently the author discusses securing IoT devices instead of treating their existence at all as an inherent security risk. In addition to the inherent danger of connecting every household appliance to the Internet, (according to a rule of thumb, no system can ever be made perfectly secure [Chapple & Seidl, 193]), the massive control possessed by the adversarial People’s Republic of China over the IoT industry has already had several ramifications for national security across Western nations (Noone).

In terms of proposed government action to mitigate cyber risk, the document gives ground to regulatory measures in Pillar One as opposed to the promising financial incentives it mentioned earlier. The author rightly summarizes that “today’s marketplace insufficiently rewards… operators of critical infrastructure who invest in proactive measures.” But the proposed regulations fail to target the source of corporations’ risky behavior in cyberspace: data is treated as the property of its hosting provider, not of the subject it regards. To this day, no law has ever criminalized the grossly negligent mishandling of Americans’ most critical information at the hands of Equifax, Capital One, and far too many others to count (Klosowski). Rather than make companies treat the data they handle seriously and delicately, these proposed regulations rely on frameworks advising technical implementations which will be outdated by the time I give you this essay (“Cybersecurity Framework”). Of course, when the time comes for these regulations to be updated, we can trust that a regulatory board will oversee the review and approval process, alongside a lengthy public comment period, all of which will take so long that the approved regulations will be outdated as soon as they are implemented. While more promising incentives are discussed later in SO 3.3, namely the increase in liability for companies producing grossly insecure products and development of standards to shield companies following them from liability, the author quickly mentions that “this [standardization effort] will draw from current best practices for secure software development,” guaranteeing its irrelevance when put into practice.

Some concrete objectives are thankfully stated in SO 1.4, which focuses on the excessive difficulty in national cyber incident response created by byzantine reporting and coordination. It notes that the Cyber Incident Reporting for Critical Infrastructure Act and pending legislation to institute the Cyber Safety Review Board (CSRB) permanently will provide the private sector with simpler methods of reporting cybercrime (ideally, to coordinate responses faster) and to conduct postmortems on a national level. The author also rightly identifies the masking of aging infrastructure (in my experience, a hodgepodge of COBOL and discontinued IT systems) as a large risk to national cybersecurity and promises to develop a lifecycle plan to modernize architecture across the Federal behemoth. Sadly the details are left to existing agencies, and are not addressed in the document.

Pillar Two concerns counter-offensives and disruptions of malicious cyber-activity. An often-discussed technique in cybersecurity deterrence and counter-operations are corporate hack-backs, raising questions of legality and cost-effectiveness. When trying to address counter-ops in Pillar Two, the document sidesteps this issue while reiterating support for more conventional counter-operations against traditional forms of organized cybercrime (botnets, ransomware gangs) and meekly requesting that private sector organizations generously donate their “resources” to existing organizations and avoid paying ransoms if demanded (as opposed to suggesting concrete action, i.e., criminalizing ransomware payments). Again, some finer (and firmer) details would make this a more effective strategy, not to mention some justification for continuing what has been up until this point an ineffective approach to dealing with modern cybercrime.

Pillar Three, the section containing economic incentives to drive national cybersecurity, has plenty of ideas to celebrate. SO 3.1 vows that the White House will “impose robust, clear limits on the ability to collect, use, transfer, and maintain personal data and provide strong protections for sensitive data;” SO 3.3 (discussed previously) champions changes to liability law; and SO 3.5 would conscript the Federal procurement process (already quite lucrative for its participants) to further improve cybersecurity standards. Yet the complete lack of concrete details in this section leaves us to speculate on the implementation of these ideas. Uncharitable interpretations can lead us to view SO 3.1 as a proposal for an “American GDPR,” which has had disastrous consequences overseas (Fefer & Archick). The glaring flaw in SO 3.3 has already been discussed. And SO 3.5 is vulnerable to the same issues as SO 3.3 in that regulations move far slower than the technologies they regulate. Again, the author describes the “what” should be done while leaving the “how” for an unspecified later date.

The closest the document ever comes to elaborating on truly technical matters is in Pillar Four, where the author (in SO 4.1) mentions the Border Gateway Protocol, Domain Name System, and Internet Protocol by name. These technologies are correctly identified as the underpinning of the Internet and vital to its security. Similarly the notion of foreign influence on the governance of Internet technologies is pointed out. Sadly our respite from ambiguity ends here, as the author demands a “clean-up” of the vulnerabilities in these technologies without specifying any plan to do so. A touch of ambition can be seen in SO 4.3’s post-quantum aspirations, but the plan they mention consists of working groups, status reports, provisions of guidance, and other informational pieces (“National Security Memorandum on Promoting United States Leadership in Quantum Computing While Mitigating Risks to Vulnerable Cryptographic Systems”). Information without action is wasted opportunity. Additionally, what little has come out of these groups has been found cryptographically insecure (“NIST’s Quantum-Proof Algorithm Has a Bug, Analysts Say”). Similarly SO 4.5’s proposal for “strong, verifiable digital identity solutions” in the context of rampant identity fraud rings hollow when the most important credential Americans possess is a 9-digit number that instantly unlocks their (attackers’) access to credit, tax refunds, and government services. More welcome would be a gradual phase-out of the SSN and introduction of a rotational system similar to the one I propose in my blog (Gilad).

Pillar Five deals mostly with foreign policy objectives and their alignment with cybersecurity challenges in the United States. Unlike the other Pillars, this section is entirely non-technical; evaluating its general principles alone (coalition-building, expanding support for allies) shows only one missing element: previous notes about the threats posed by hostile nations are missing from this section, where they would appear most appropriate. Coalition-building will help strengthen our alliances, but without mentioning the threat these coalitions face we are left to wonder if these efforts will be too passive to make a substantial difference.

The concluding section of the document – incorrectly named “Implementation” – is a microcosm of the rest of the document. It designates other agencies and authorities to handle the dirty work of the document’s implementation while touting them as reliable and ready to handle the work. The most disheartening example here is their parading of the CSRB’s work regarding Log4j. They note that a review into the vulnerability (called Log4Shell) was completed in mid-2022, and stakeholders were provided with “clear, actionable recommendations based on what the review discovered.” There’s just one problem: Log4Shell was discovered in December of 2021 (Associated Press). That means it took the CSRB around 6 months to make recommendations about a vulnerability which by then had been patched, mitigated, and resolved by the private sector. That the author should appear almost proud that this occurred is a troubling sign for this document’s implementation. Not only can I not say that this document, which has hardly any will to say anything definite about America’s cybersecurity policy, will be “effective” – I can say that its feeding of an outdated and sluggish bureaucracy will actively harm America’s security.

Mass Transit Possibilities in South Florida

South Florida is again reasonably connected by train after a long stop since the late 60s/early 70s thanks to the Brightline, which opened a station very conveniently located to my family home in Boca Raton. It has all but solved my north/south axis travel woes along I-95, and when the Orlando station opens early next year (supposedly, although given their “track” record with delays in the past I am somewhat skeptical) it will have solved all of my long distance travel needs in the state of Florida.

Yes, the Greyhound bus existed before this train, although having ridden it in the past I can tell you that it (like the existing SoFlo rail option, Tri-Rail) leaves much to be desired. So does Brightline presently, which is currently struggling with a double-booking issue which has victimized me twice and generated plenty of bad press for them. Yet it remains the single best mass transit option I have ever used in the United States. It’s always on time (barring the times Florida Men, species Homo sapiens floridanus, believe themselves faster than it — Brightline shall surely outpace crystal meth and heroin as the apex predator for this creature), clean, and spacious, for train and station alike.

Which got me to thinking about how the idea behind Brightline could be used to solve my philosophical grudge against driving.

Why I Don’t Like Driving

It’s inefficient.

The traffic light ranks in my mind as one of the most pernicious inventions of mankind. Along one axis a group is allowed to proceed for a certain length of time, sometimes adjusted with the help of subsurface pressure plates measuring traffic. Then that axis is stopped, with cars along it building up in droves, thumbs twiddling uselessly, and the other axis goes. An endless cycle of wasted hours. Even a 4-way stop sign would be better, if not more accident-prone.

Roundabouts are a wonderful solution to the matter of traffic lights. But asking cities to abandon a system that “works” and replace it with a roundabout that costs money (although the associated expenses are typically one-time installations of islands in the center) does not have appeal. Especially for municipal and county budgets. It would be far more likely to happen if state funds went towards the issue.

But why should state funds go towards the issue when private funds can solve the same problem — in a slightly different way — at no taxpayer cost?

Private vs. Public Mass Transit

Brightline is today the only privately owned and operated intercity train in the entire United States. Although the company has received some small-ish amounts of federal money for improving track-crossing safety (a futile effort to protect the Homo sapiens floridanus from themselves) and large amounts of money from the City of Aventura because the city wanted a station on the line (Boca Raton, in contrast, paid only for a small fraction of the station which they built), Brightline cannot depend on their public-sector funding for bailouts, day-to-day operating cost coverage, or profitability. The same cannot be said for the only other train following a similar route: Amtrak, which has operated the same dilapidated trainsets since beginning service, makes no profit (even before the pandemic), and carries as many passengers per year as Brightline does in 5 months*. Although I can’t find data on the Brightline’s profitability, at least they have a profit incentive in the first place!

The route Brightline is following now (and plans to continue following by expanding into Tampa) was originally a public high-speed rail project that never broke ground. It appears to me a vindication of mass-transit privatization (and perhaps privatization in general) that what the public sector couldn’t or wouldn’t do, the private sector did and is doing for a much smaller bill to the taxpayer (although I would have preferred none at all). Members of the public who prefer driving, or simply never travel the route proposed by the public rail project would have been liable for at least some of the costs (perhaps the lion’s share, since public fares hardly cover the costs** of operating a railroad) of a public Brightline, which strikes me as fundamentally unfair. I would never ask someone to pay for something they have no intention of using (or even ability to use, if you live in a community far away from the nearest station).

In that moral regard the Brightline is an improvement, although not a complete solution. They kind-of paid for the Boca Raton station, but Aventura’s taxpayers are fully on the hook for their station, which will (hopefully) begrudge those residents of the city who will never use the train (let alone the moral implications of publicly funding a privately-operated service without guaranteeing discounted fares for the residents of that public polity). And I haven’t found any info on how they funded the construction of their original station trio (WPB->FTL->MIA). But the new track they laid (mostly for Orlando’s expansion) was their own; fares aren’t subsidized by public entities; and if nobody likes them they will either improve (at no cost to you) or die (whereas Amtrak and other public systems would be bailed out, or simply never fixed).

The West-East Axis

The Brightline and Amtrak both follow a North-South route in South Florida, parallel to I-95 (and physically not far from it). But plenty of communities in South Florida, connected to the coast by the Sawgrass Expressway, I-595, and I-75, exist far to the west (following the border of the Everglades). A few examples include Parkland, Coral Springs, Sunrise, Weston, Davie, Plantation, Miramar, Hialeah, Doral, and Sweetwater. There are plenty of transit options connecting the communities along the North-South coastline, but almost nothing connecting these Western communities by anything except car (at least according to Google Maps). This makes sense because the passenger rail options along the coast are built on top of existing freight railtracks, which follow a North-South route for shipping purposes. But if the success of the Brightline is any indication, a similar service going along the West-East axis (connecting communities there and eventually connecting to the coastline trains) would also be successful.

The instant difficulty is the question of where. The question of where along the North-South axis is easy because all the communities there are lined up like ducks in a row, forced into marching position by the Everglades. But the western communities could be connected to each other and the coast in a number of different ways. Here’s an image that crudely explains what I mean:

Edited map of South Florida from Deerfield Beach to Miami, with Brightline and Tri-Rail routes shown in yellow/blue respectively and theoretical routes, shown in red, between Deerfield/Lighthouse Point and Parkland, Cypress Creek and Sunrise/Tamarac, Ft. Lauderdale and Weston/Southwest Ranches, Hollywood and West Park/Miramar, Miami and Hialeah

These theoretical routes are not based around my knowledge of available land in the areas I’ve highlighted in red; they are just lines connected dots on a map together in ways I think make sense.

Plausibility

The issue is clearly that we’d need multiple rail lines to connect the Western communities when we only needed one to connect the North/South communities. The cost and time (and regulatory compliance issues) associated with laying segments of new track and building new track crossings is vast — a future-planned segment in Orlando is planned to cost $6 billion, and while I strongly suspect this planned cost might have something to do with the federal cash Brightline plans to solicit (if such a request is denied, I think this “plan” will be revised to 30% of the original cost), it is still a good indicator of what it will cost to build mostly just track.

Brightline’s current route from WPB to MIA runs on pre-existing track built by Florida East Coast, so they only bore the costs of station-building and track-crossing-upgrading. From WPB to Orlando, Brightline also ran on pre-existing track up until Cocoa — building new track from Cocoa to Orlando. So it wouldn’t surprise me if, despite the smaller distance, the costs of building up the West-East axis will probably exceed the costs of WPB->MIA and WPB->ORL (although probably not combined).

High entry costs may be the single biggest reason why we won’t see the Western communities connected by (private or public) commuter rail anytime soon, especially when Florida is known as a state where the single most common form of transportation is driving. With the exception of Miami and Fort Lauderdale, every city I’ve been to here has been very spread out, making the design of suitable bus lines difficult and strongly encouraging people to drive to nearly every location.

I think there’s a real appetitie among Florida residents to beat traffic, but with Brightline concluding major investments in the original Florida High-Speed Rail Corridor plan, and public entities reluctant to spend money on transit (Orlando voters just denied an ad valorem tax to raise funds for expanding their transit system), I have no hopes for an actual West-East rail connection anytime soon.


*Data on Brightline ridership from this article and on Amtrak from here and there. In both cases I take liberties with calculations: for Brightline I am probably undercounting ridership since I extrapolate average ridership numbers per month from their August 2022 numbers, when in reality since the December expansion into Boca and Aventura those numbers (based purely on Brightline’s ticketing system reporting few seats available) should probably be significantly higher; and for Amtrak I am certainly overcounting because the reported ridership includes all stops on the route, which goes from Florida to New York; I can’t find any data about FL ridership alone.

**The article here notes that there do exist public fare-collecting mass transit systems which recoup their operating costs: New York ferries, a few busses, and “van pooling”. Do what you will with that information.

How can UCF fix its outmatched administration? Make the colleges do it instead

The modern American university consists of a central bureaucracy overseeing its academic colleges, with administrative duties split up between the two. At UCF, the central office handles academic recordkeeping, financial aid, and admissions, while the colleges handle degree certification and academic advising (among other tasks I may be unaware of).

That system has been put under strain over the years, as state and federal budget restrictions make it impractical for the University to devote its increasingly-limited hiring powers to administrative duties as opposed to academic faculty. As enrollment grows, administrative capacity has not kept pace, resulting in situations where students may need to wait weeks for a response to an important email, or send in 8 separate copies of their college transcripts (at $8 per transcript, I think the University ought to reimburse me with a $64 scholarship). To remedy this, UCF subcontracted most (if not all) of its financial aid and undergraduate admissions (representing the vast majority of the University’s administrative duties) to third parties starting in Summer 2021. This is great for the average student, seeking admission in their last year of high school and requiring no special consideration. For everyone else, it’s a complete disaster, as communication between these subcontracted call centers and the skeleton crew comprising the “upper-level” of the admissions and financial aid departments is practically non-existent. The University, desiring to retain some level of decision-making autonomy in these departments, tightly restricts the authority of the call center staff, preventing them from making important decisions for students with special cases and from directly contacting managment on the UCF campus. A callback, which usually goes unhonored (I know from personal experience), is the best they can do.

Time-consuming bureaucracy is a part of the American lifestyle, but for students in desparate need of financial aid to pay their rent or attend class, it’s entirely unacceptable.

The easiest, and possibly the worst, solution to this problem is to pump more money into Millican Hall*. Not only would it involve spending more money on a function of the University which is tangential to its core educational mission, it would also likely result in a more convoluted and potentially corrupt upper managment. Some students will remember the last time the University had more money than it knew what to do with**.

The ideal solution, in my world, would have the following qualities:

  • Keeps pace with its clients in 65%+ of cases,
  • Handles time-sensitive situations within the required timeframe in 100% of cases,
  • Doesn’t make students feel like they ought to call the Ombudsman just to get an answer to what’s happening with their case, and
  • Doesn’t cost more than the current solution.

I want a bureaucracy that works efficiently, handles emergencies well, doesn’t feel like a bureaucracy, and doesn’t cost more than the current bureaucracy. A nice fairy tale, but with enough decentralization it might be made into a reality. At the start of this post I said that the central bureaucracy splits its duties roughly 60-40 with the colleges. What if this split more closely resembled 5-95, similar to an Oxford-style university?

The only role I can imagine the central bureaucracy fulfilling better than the colleges is academic recordkeeping, which requires the participation of multiple colleges to compile a complete academic record. Everything else - admissions, financial aid, degree certification, academic advising, and even tuition/account payment - seems to be well suited for handling on a college-level. In all administrative operations, the colleges would have a fraction of the administrative burden compared to Millican Hall, which could give admin-related funds to the colleges proportionally based on their average enrollment and the size of their application pools. They would be able to pay closer attention to each individual student, avoiding the salient issues raised by automated systems and toothless call centers.

One minor wrench that could be thrown into the gears of this new system is the federal nature of student financial aid. Would the colleges be authorized to make transactions directly with the federal government, circumventing the University altogether? Probably not, but Millican Hall could retain an automated system that processes financial aid transactions submitted by the colleges, as well as a small number of staff to talk to the colleges’ liasons in the event of a special case. This would add minor costs to that portion of the system but keep its overall efficiency more or less intact.

A more important issue: would this new system actually cost the same as the current one? In the short-term, I doubt it. The University would need to give extra funds to the colleges for one-time setup expenses: purchasing computer systems and initiating hiring processes for new staff. These costs aren’t factored into the long-run performance, but keeping pace with current enrollment at an exponentially-growing university inevitably means spending more money. While decentralization can help the University achieve administrative efficiency, we can’t expect it to achieve maximum efficiency in the absence of reasonable funding. If the University’s enrollment expands from its current 70,000+ to 100,000 in the next 5-10 years, and the funds allocated to the colleges for administration remain the same, even this new system will fail miserably to serve its students.

Of course, the greatest solution of them all would replace UCF with several smaller Universities that can actually seat their students, but considering that the very existence of UCF is a Florida law, I doubt this will come to pass. So I have to be a bit more pragmatic.


*A building at the southern end of the Orlando campus, containing the offices of all the constitutent departments of the University’s central managment. The University of Maryland, College Park should take note of this model instead of spreading their administrative offices around campus like confetti, making it rather difficult to interact with the administration.

**To be fair to former President Hitt and his staff, Colbourn Hall remains one of the nicest buildings on campus to this day.