AI Is a Tool. You Still Have to Think.

I recently saw someone and have often seen people saying the following about artificial intelligence with two words:

Fuck AI!

The most recent version of this involved someone in the middle of a disagreement. They asked a large language model to compose a response, sent what it produced without properly considering it, and made the situation worse.

I understand the reaction. Family disagreements are emotional, context-heavy, and full of history that no language model can completely know. A generated message can be technically polished while being entirely wrong for the people receiving it. It can sound cold, clinical, patronizing, manipulative, or strangely unlike the person supposedly sending it.

I do not think the lesson is “fuck AI.”

The lesson is: you still have to think.

If you copy an AI-generated response verbatim, send it into a sensitive human situation, and then blame the tool when it goes badly, you have misunderstood both the technology and your own responsibility.

The model did not decide to send the message.

You did.

Generated text is not judgment

A large language model can produce remarkably convincing language. That is one of its strengths, but it is also one of its greatest dangers.

Fluent writing creates an impression of thoughtfulness. A neatly organized response can look reasoned even when it is based on incomplete context, questionable assumptions, or a complete misunderstanding of the relationship involved.

The model does not know the look your brother gives when he is hurt but trying not to show it. It does not know which phrase your mother will interpret as condescending. It does not know that a technically accurate point has already been repeated six times and that repeating it once more will only deepen the argument.

It knows the information you supplied, the patterns learned during training, and the conversation currently in front of it.

That can be useful. It is not the same thing as wisdom.

AI can generate options. It can suggest a structure, identify a blind spot, remove unnecessarily hostile wording, or help us see how a message might be received. It cannot relieve us of the obligation to decide whether those words are true, appropriate, humane, and recognizably ours.

Copy and paste is not using AI well

I use large language models regularly. I use them professionally, for technical projects, for learning, for debugging, and sometimes for improving written communication.

I do not simply copy the result and press Send.

If I ask an LLM to help refine an email, I read what it produces. I compare it with what I intended to say. I retain useful phrasing, remove language that does not sound like me, restore context that was lost, and often merge its suggestions into my own draft.

The finished message remains my message.

That is not an insignificant distinction. Editing the result is where critical thought re-enters the process. I am asking:

  • Is this factually correct?
  • Does it express what I actually believe?
  • Is the tone appropriate for this person?
  • Has the model inferred something I never said?
  • Is it avoiding an uncomfortable point that needs to be addressed?
  • Is it escalating a situation while sounding superficially calm?
  • Would I be willing to say these words face to face?
  • Am I prepared to take responsibility for the result?

If the answer to that last question is no, I should not send it.

Blindly forwarding generated text is not thoughtful adoption of AI. It is outsourcing the appearance of thought.

Human-in-the-loop is not corporate jargon

“Human-in-the-loop” can sound like one of those phrases people place in a slide deck to make an AI project seem responsible.

It describes something essential.

The human provides the objective, context, constraints, judgment, verification, and final accountability. The system contributes speed, language generation, pattern recognition, analysis, or implementation assistance.

Remove the human judgment and the process becomes ungrounded. The model may still produce an answer—these days a very polished one—but nobody is confirming that the answer belongs in the real world.

In my arcade-cabinet work, agents have written code under my direction. They have helped analyze logs, build maintenance tools, normalize controller behaviour, audit metadata, and document complicated interactions among Linux devices and services.

That does not mean I type “make it work” and accept whatever appears.

I define the intended behaviour. I set boundaries. I ask the agent to inspect the existing system. I review the proposed change. We test it against the actual hardware. When the evidence contradicts an assumption, we correct the assumption. If a script could damage data, I require dry runs, backups, explicit apply modes, and verification afterward.

The code may be generated agentically. The responsibility is not.

That is human-in-the-loop.

AI magnifies the way we already work

AI can help a thoughtful person examine more possibilities, work through unfamiliar material, and accomplish more in a day. It can also help a careless person produce mistakes faster and with greater confidence.

In that sense, it is an amplifier.

Give it a carefully defined problem, relevant evidence, meaningful constraints, and an engaged human reviewer, and it can be extraordinary.

Give it one side of a family argument, ask it to prove that you are right, and paste the result into a group chat, and it can provide beautifully formatted gasoline.

This is not unique to AI. People have always misused tools, repeated claims they did not understand, forwarded articles they did not read, and borrowed other people’s words to avoid difficult conversations. AI makes that behaviour easier, faster, and more convincing.

That increased power deserves increased care.

You cannot outsource accountability

Suppose an LLM produces a message that damages a relationship. The model deserves scrutiny. Perhaps it responded too confidently, mirrored the user’s anger, validated a distorted premise, or generated language that was inappropriate for a vulnerable situation.

The organizations building these systems also have responsibilities. They should communicate limitations honestly, design safeguards thoughtfully, and stop presenting every generated answer as though confidence and correctness were the same thing.

But the person who chose to send the message is still part of the causal chain.

We cannot demand complete personal credit when AI helps us produce something successful and then claim complete helplessness when the same process produces harm.

If I publish generated code without testing it, the resulting bug is my responsibility.

If I submit invented citations without checking them, the false claims are my responsibility.

If I send a cruel or tone-deaf message without reading it carefully, the damaged relationship is my responsibility.

“The AI wrote it” is an explanation of process. It is not an exemption from accountability.

The problem is choosing not to think

I will admit that this is the part that frustrates me most.

We have been given access to tools that can help us explore ideas, learn new skills, analyze huge amounts of information, and build things that previously would have required far more time or specialized knowledge.

And some people use that power to avoid thinking.

They paste the first response. They do not check facts. They do not question assumptions. They do not notice that the tone is wrong. They do not ask whether the output reflects their own beliefs. Sometimes they do not even appear to read it closely before attaching their name to it.

It is tempting to say that people doing this are too stupid to think.

I think the more accurate criticism is harsher: in that moment, they are choosing not to think.

Intelligence is not the only issue. Patience, humility, curiosity, and personal responsibility all matter. Critical thinking requires us to slow down long enough to ask whether a convenient answer is actually a good one. AI does not remove that requirement. It makes it more important.

Use AI to strengthen your voice, not replace it

There are constructive ways to use an LLM during a disagreement.

You might ask:

  • What assumptions am I making about the other person’s motives?
  • Which parts of my draft are likely to sound accusatory?
  • Can you show me three interpretations of what they said?
  • What important context might be missing from my account?
  • Can you help me shorten this without removing the apology?
  • Where am I trying to win rather than resolve the issue?
  • What questions should I ask before responding?

Those prompts use AI as a tool for reflection rather than as an automated weapon in an argument.

Even then, the output requires judgment. A model can suggest empathy without feeling it. It can construct an apology without meaning it. It can imitate vulnerability without taking the emotional risk that vulnerability requires.

The human being has to supply those things.

Sometimes the right conclusion after consulting AI will be to write the message yourself. Sometimes it will be to call rather than text. Sometimes it will be to wait until you are calmer. Sometimes it will be to say only, “I don’t want this argument to become worse. Can we talk tomorrow?”

No model should make that decision for you.

Criticism of AI is necessary

None of this means that AI should be above criticism.

There are serious questions about accuracy, bias, privacy, labour, copyright, environmental cost, concentration of power, manipulation, and the effect these systems may have on education and employment. There is plenty to criticize, and “fuck AI” may express a completely understandable anger about some of those issues.

But if the specific complaint is that someone blindly copied generated language into a family dispute, then we should identify the failure accurately.

The system produced words without understanding the full human situation. That is a known limitation.

The user treated those words as finished judgment. That was a human decision.

Both facts can be true at once.

Keep your hands on the controls

I do not advocate surrendering our work, relationships, or critical faculties to artificial intelligence. I advocate learning how to use it well.

Use it to generate possibilities, not commandments.

Use it to challenge your thinking, not replace it.

Use it to improve a draft, not impersonate you.

Check its facts. Question its assumptions. Rewrite its language. Test its code. Protect private information. Consider the person who will receive the result. Be willing to reject the output entirely.

Most importantly, take responsibility for whatever leaves your hands with your name attached.

AI is a tool. It can be a remarkably powerful one.

But you still have to think.

Retropie Console – The Arcade Cabinet Owner’s Manual Nobody Thinks to Write

An arcade cabinet feels finished when the games launch, the controls respond, the marquee changes, and the buttons glow.

That is usually the moment when nobody thinks to write the manual.

The wiring is still fresh in memory. The unusual startup command is obvious because it was edited yesterday. We remember which controller must use X mode, why one group of RGB pins appears backwards, and which service creates the virtual Player 1 device. If something breaks, surely we will remember what we did.

Then six months pass.

The cabinet still looks like one machine, but it is actually a collection of systems that happen to start together: Raspberry Pi hardware, a Linux operating system, RetroPie, EmulationStation, emulator cores, an I-PAC, wireless controllers, virtual input devices, RGB lighting, a Pixelcade marquee, a Sinden Lightgun, launch hooks, metadata tools, ROM collections, BIOS files, configuration archives, and a four-terabyte SSD.

The hard part of maintaining it is rarely remembering that those components exist. It is remembering why they were connected in precisely this way.

That is why we wrote an owner and service manual.

A custom cabinet is an integration, not an appliance

A commercial appliance usually has a defined model number, a factory configuration, and a manual written by the organization that designed the whole product.

A custom arcade cabinet has manufacturers for its individual parts, but no manufacturer for the assembled system.

Ultimarc can document the I-PAC Ultimate I/O. Raspberry Pi can document the computer. RetroPie can document its setup scripts. Pixelcade can document its listener. Sinden can document the lightgun. None of them can describe what happens when all of those components share this cabinet’s USB buses, startup sequence, controller mappings, lighting rules, and recovery process.

Only the owner can document the integration.

Our manuals therefore serves three different purposes:

  • a quick operating guide;
  • a reference to the emulators used and different configurations;
  • a technical service reference; and
  • a disaster-recovery starting point.

Those purposes overlap, but they are not identical. The person turning off the cabinet needs a short, safe procedure. The person diagnosing a missing Player 2 controller needs device identities, service names, and expected mappings. The person rebuilding a failed drive needs file locations, dependencies, backups, and evidence of the last known-good state.

Trying to make one undifferentiated list satisfy all three audiences produces a document nobody can use. We organized ours around questions.

Start with the questions a future owner will ask

The manual’s navigation is based on practical needs:

QuestionInformation required
How do I start or shut down the cabinet safely?Power sequence, front-end procedure and expected startup behavior
What hardware is installed?Exact models, quantities, connections and intended roles
Why does this USB device have several names?Kernel nodes, stable identities, aliases and physical topology
Which controls should work for each player?Physical layout, raw inputs, logical mappings and known exceptions
Why are these RGB pins backwards?Wiring evidence and the required Player 2 BGR compensation
What starts automatically?Services, launch hooks, dependencies and expected status
Which emulator runs this title?System defaults, per-game overrides, BIOS requirements and test titles
What do I back up?Configuration payloads, ROM and BIOS boundaries, full-drive image and checksums
How do I know a repair worked?Commands, expected results and end-to-end tests

This changes the tone of the document. It is not a catalogue assembled for its own sake. Every detail should help someone operate, diagnose, change, or recover the machine.

Record what is observed, not what seems likely

The cabinet provided an early example of why assumptions are dangerous.

The Raspberry Pi 4 has a 64-bit processor. The running kernel reports aarch64. It would have been natural to describe the installation as a 64-bit operating system.

The live package architecture told a more precise story:

Hardware:   Raspberry Pi 4 with a 64-bit-capable CPU
Kernel:     aarch64
Userspace:  32-bit armhf
OS release: Raspbian GNU/Linux 10 (Buster)

That distinction explained why a 64-bit Pixelcade executable could exist on disk yet fail with the misleading message No such file or directory. The program required a loader that the 32-bit userspace did not provide. Replacing it with the correct ARMv7 build resolved the problem.

“Raspberry Pi 4” was true but insufficient. “64-bit kernel” was also true but insufficient. The combination of hardware, kernel, userspace, and actual executable format was the useful fact.

We use that standard throughout the manual:

  • live output is stronger than an assumption based on hardware;
  • a running process is stronger than an installer description;
  • an exact device property is stronger than a temporary /dev number;
  • a photographed lighting result is stronger than an attractive-looking XML file; and
  • a game tested from launch through exit is stronger than a configuration that merely parses.

Document reasons as carefully as settings

A configuration file can tell a future maintainer what value is present. It rarely explains why changing it would be a mistake.

Player 2’s RGB pin triplets are a good example. They descend where Player 1’s values ascend because that side of the panel is physically wired in BGR order. Tidying the numbers would swap red and blue across the right side of the cabinet.

The I-PAC provides another. It is kept in Dual DInput mode because the panel must appear as two game controllers while the cabinet’s real keyboard remains available to emulated computers. Returning it to keyboard mode might simplify one old tool while reintroducing conflicts throughout the rest of the system.

The controller merger exists because four physical gamepads should become two logical players:

I-PAC Player 1 + first 8BitDo  ──> Arcade Merged Player 1
I-PAC Player 2 + second 8BitDo ──> Arcade Merged Player 2

The trackball merger is activated only for listed games because globally taking control of mouse-like devices would interfere with classic computers, adventure games, and other pointer-driven software.

The Pixelcade listener uses a USB by-id path because ttyACM0 and ttyACM1 exchanged roles after reboot. The numbered nodes were observations from one boot, not hardware identities.

These explanations turn configuration trivia into engineering decisions. Without them, a future repair can easily undo the solution while appearing to simplify the system.

Map the whole path from hardware to game

Input problems are particularly hard when documentation stops at the physical panel or begins only inside an emulator.

Our manual follows the path through every layer:

Physical button or controller
        ↓
Linux input device and USB identity
        ↓
Translation or virtual-device service
        ↓
RetroArch player assignment
        ↓
Emulator or MAME input type
        ↓
Observed behavior in a game

That path revealed details which would otherwise be almost impossible to reconstruct from memory. Player 1 and Player 2 report Coin and Start in opposite raw button order. The virtual-controller service normalizes those inputs before RetroArch sees them. MAME uses the resulting stable Joy 1 and Joy 2 devices, with global input types named START1, START2, COIN1, and COIN2.

The manual also records known imperfections. At the time of testing, the fourth action-button column did not emit DInput events. That is not hidden simply because most games do not need those buttons. A service document should distinguish “working and verified,” “installed but untested,” and “known follow-up.”

Give every custom service an address

Custom integrations become much less mysterious once their active locations are recorded.

The manual and its companion technical reference identify components such as:

/usr/local/sbin/arcade-gamepad-merger
/usr/local/sbin/arcade-trackball-merger
/usr/local/sbin/arcade-trackball-game-hook
/etc/systemd/system/arcade-gamepad-merger.service
/etc/systemd/system/arcade-trackball-merger.service
/opt/retropie/configs/all/runcommand-onstart.sh
/opt/retropie/configs/all/runcommand-onend.sh
/usr/sbin/rgbcommander/rgbcmdd.xml
/opt/retropie/configs/all/autostart.sh

A useful service entry includes more than a pathname. It explains:

  • what starts it;
  • which hardware or files it depends on;
  • what other component consumes its output;
  • how to check whether it is running;
  • where its logs appear;
  • what normal success looks like; and
  • what else will fail if it is absent.

This allows troubleshooting to proceed in layers. We can ask whether Linux sees a physical controller, whether the merger reads it, whether the virtual device exists, whether RetroArch selects it, and whether MAME receives the expected input. “The controls do not work” becomes a series of answerable questions.

Use test titles as part of the documentation

An emulator inventory tells us what is installed. It does not tell us whether the surrounding integration works.

We record representative games as regression tests. Bubble Bobble helped verify two-player MAME controls. Demon Attack became the recovery and regression title for the custom IBM PCjr cartridge launcher. Marble Madness verifies the conditional trackball merger.

A named test title is valuable because it fixes several variables at once:

  • the ROM exists;
  • its launch path is known;
  • its emulator selection is known;
  • its controls have an expected behavior; and
  • a previous successful run provides a comparison.

“Test the PCjr” leaves too much room for interpretation. “Launch Demon Attack and confirm immediate joystick movement, firing, and a clean status-zero exit in the launcher log” is a service procedure.

Separate quick backups from complete recovery

Our RetroPie installation occupies a four-terabyte SSD. A raw drive image is the most complete recovery artifact because it captures the partition table, boot files, operating system, installed software, configuration, and data exactly as they exist.

It is also enormous and slow to create.

The manual therefore defines two complementary backup levels.

Configuration snapshots

Frequent snapshots preserve the files that embody our custom work:

  • RetroPie and emulator configuration;
  • virtual-controller and trackball scripts;
  • systemd services;
  • launch hooks;
  • RGB Commander configuration and animations;
  • controller profiles;
  • current gamelists;
  • metadata tools; and
  • selected recovery records.

The backup and restore scripts are kept payload-symmetric: every archive the backup creates has a corresponding restore path. Manifests, checksums, and README records are used to validate a snapshot rather than being copied blindly into the live filesystem.

Full-drive images

A full image is slower but protects against failure of the boot drive or the loss of installed package state. The manual records the SSD’s model, capacity, partition scheme, and observed macOS disk identifier—while warning that the identifier must be reconfirmed every time.

That warning is not decorative. Reversing the input and output of dd can destroy the wrong disk.

I keep one precious backup image, two current backup images on two NASes, and offsite backups with friends.

ROM and BIOS preservation is also stated explicitly. The configuration snapshot intentionally excludes BIOS files. The complete BIOS directory and ROM library require their own preservation strategy or the full-drive image. I have several copies of both across my two NASes and an additional hard drive.

A backup policy is only trustworthy when it records what is absent as clearly as what is included.

Add photographs and a change log

Some cabinet facts are easier to prove visually than verbally.

We retain photographs of the default illuminated panel – a ColecoVision calibration pattern, and the console at boot up. Those images establish physical button positions and show what a successful lighting profile looked like on the real hardware.

The change log then records when a behavior changed and how it was validated. Entries include XML checks, live service inspection, hardware tests, game launches, backup audits, and recovery dry runs.

This prevents the manual from pretending to be timeless. It is a record of a changing machine. A statement verified against the running cabinet has a date and a basis. When the system changes again, we know which conclusions require retesting.

Writing the manual with an agent

The manual itself was developed through agentic work under my direction.

That did not mean asking an AI to invent documentation from a list of products. We used the agent to inspect the real system, correlate configuration files, read service status and logs, compare documentation with live behavior, identify contradictions, organize the results, and maintain both focused references and the consolidated manual.

I built up trust in the agent. I started small with some small fixes, and built up to granting it full access to my Retropie console to make direct changes as we fixed issues and built tools. It preformed flawlessly and that built up a corpus of knowledge about the inner workings of the console. It completed months of work in a matter of a few days.

I supplied the intent, physical context, acceptance decisions, and hands-on validation. The agent supplied persistence across hundreds of small details and the ability to trace relationships that crossed several subsystems.

This division of work proved particularly useful during documentation audits – a human MUST be in the loop, and the agent provided a wholistic view of everything.

A claim in one guide could be compared with the running operating system, a service file, the contents of a backup script, and the behavior recorded in another troubleshooting session. When the evidence disagreed, the document was corrected rather than averaging the stories together.

Human-in-the-loop matters here for the same reason it mattered in the metadata project. A generated paragraph can sound authoritative while being wrong. The cabinet, its files, its logs, and its observed behavior remain authoritative. The agent helps turn that evidence into durable knowledge.

That’s what people miss about using artificial intelligence – a human is prescriptive, directs and provides clarity, and a human must be in the loop to review.

The manual is part of the machine

The final documentation is not one giant file trying to contain everything. The owner and service manual provides the map, while companion references hold detailed lighting profiles, controller integration, maintenance tools, and the system-and-emulator inventory. Markdown remains easy to search and revise; generated PDFs provide convenient archival and offline copies.

The documentation archive belongs beside the working cabinet and beside its backups. If the only copy of the recovery instructions lives on the drive that fails, that is not a recovery plan.

More importantly, the manual is treated as another maintained component. When we add a system, change a controller mode, update a service, alter the USB arrangement, or discover that an old assumption was wrong, the documentation changes with it.

That may be the most important idea in the entire project.

The cabinet’s most valuable knowledge is not merely which files exist. It is why the machine works, how we know it works, and how to recognize when it no longer does.

Nobody thinks to write that manual while the attract mode is running and everything feels finished.

We did—and the cabinet is more maintainable because of it.

Retropie Console – Two trackballs, one console

For this console, I have duplicate controllers. I have the arcade console that has two joysticks, buttons and a trackball. I have two 8BitDo Ultimate Controllers and a Logitech MX Ergo. Why? Sometimes I want to use the arcade panel, sometimes I want to sit on the couch, I might want multiple players too.

Marble Madness is one of those games that immediately explains why an arcade cabinet needs a trackball. I grew up playing this game with a joystick on a Commodore 64, and it is painful to play that way. This game needs a trackball, just as Duck Hunt needs a lightgun, just as the original arcade game was designed. The speed, momentum, diagonals, narrow paths, and frantic changes in direction all feel different when the control is a rolling ball rather than a joystick or D-pad.

Our cabinet has a trackball built into the centre of its control panel. We also use a Logitech MX Ergo, which is convenient when sitting farther back or when someone would rather use a handheld thumb-operated trackball.

Although, the MX was purchased for a different purpose – to help with Amiga emulation, and the added benefit is that it can be used for trackball games.

The requirement sounded uncomplicated:

  • launch Marble Madness;
  • roll either physical trackball; and
  • have the game respond as though there were one consistent Player 1 trackball.

As with many seemingly small arcade-cabinet requirements, the difficult part was not getting each device to work. The difficult part was making both devices behave predictably without changing how every other mouse-driven game and computer system behaved.

Two working devices do not automatically become one control

Linux can detect multiple mouse-like devices at the same time. The panel trackball and the MX Ergo can each produce pointer movement, and diagnostic tools can see their events.

That does not guarantee that an emulator will combine them in the way we want.

An application may select one mouse index, choose whichever device it discovers first, or remember a device assignment that changes after a reboot. A configuration that works while the panel trackball is Mouse 1 can fail when the wireless receiver appears first and changes the order.

This is the same category of problem we encountered with gamepads and USB serial devices elsewhere in the cabinet: a temporary device number is not a durable identity, and several physical devices do not automatically represent one logical player.

We could have tried to configure MAME to understand both physical devices directly. That would have pushed hardware-specific knowledge into the emulator, however, and it would have made the result dependent on device enumeration and MAME input details.

Instead, we gave the emulator one stable control.

Creating one virtual trackball

We built a small service called:

/usr/local/sbin/arcade-trackball-merger

Its job is to combine movement from the two approved physical sources into one virtual pointer:

Panel trackball ─────┐
                     ├── Virtual Player 1 trackball ──> MAME
Logitech MX Ergo ────┘

MAME no longer has to decide which physical trackball is Player 1. It receives one logical device. Movement from either source reaches the same destination.

This is an important separation of responsibilities:

  • Linux still detects the physical hardware;
  • the merger understands which devices are acceptable sources;
  • the virtual device represents the control the game needs; and
  • MAME maps that one stable logical device.

The idea is similar to the cabinet’s virtual gamepad arrangement, where an I-PAC panel side and an assigned wireless controller become alternate inputs for one player. The trackball implementation is narrower because mouse-like devices affect more than arcade games.

Why we did not merge the trackballs globally

A global merger would have been easy to describe: start it at boot and leave it active forever.

It would also have been too broad.

The cabinet runs classic computers, adventure-game engines, menus, ports, and other software that can use a mouse or trackball. Some of those programs should see the panel trackball normally. Others may need the Razer mouse, the MX Ergo, or their own device selection. Taking exclusive control of the physical devices throughout the entire session could make those systems harder to use.

Our actual requirement was much smaller: either trackball should control Marble Madness.

We therefore made activation game-specific.

The list of games that request merged-trackball behavior is stored in:

/opt/retropie/configs/all/arcade-trackball-games.txt

At the time of writing, it contains the known Marble Madness ROM basenames:

marble
marble2
marble3
marble4
marble5

Using ROM basenames rather than a display title matters. A front-end title may be reformatted, translated, or supplied by metadata, while the launch hook receives the actual file being started. The basename gives the automation a small and explicit allowlist.

The rule is effectively:

IF system is mame-libretro
AND ROM basename is in arcade-trackball-games.txt
THEN activate exclusive trackball merging
ELSE leave the physical pointer devices alone

This is a useful design principle for cabinet automation: apply unusual behavior only where it is needed.

Connecting the merger to a game launch

RetroPie provides launch hooks that run around an emulator session. Our start hook is:

/opt/retropie/configs/all/runcommand-onstart.sh

When a game begins, that hook performs three independent jobs:

  1. It notifies RGB Commander, with a two-second limit so a lighting problem cannot prevent the game from starting.
  2. It resolves the cabinet’s virtual Player 1 and Player 2 gamepads for Libretro.
  3. It asks the trackball game hook whether this launch requires the merged pointer.

The trackball-specific decision is handled by:

/usr/local/sbin/arcade-trackball-game-hook

Before evaluating the new launch, the hook clears stale state. It then checks both the system name and ROM basename. Only an approved mame-libretro title activates the exclusive merge.

When the emulator closes, RetroPie’s end hook runs:

/opt/retropie/configs/all/runcommand-onend.sh

That deactivates the trackball merge and releases the physical devices. The next program once again sees the ordinary mouse and trackball arrangement.

Cleaning up at both ends is deliberate. The end hook handles a normal exit, while clearing stale state at the next start prevents an earlier abnormal termination from permanently contaminating later launches.

The service and the policy are separate

The virtual trackball service is managed by systemd:

arcade-trackball-merger.service

The service provides the mechanism. The game list and launch hook provide the policy.

That distinction makes the setup easier to maintain. Adding another compatible trackball game does not require rewriting the merger. We can review the game, add its exact ROM basename to the allowlist, launch it, and test whether its MAME mapping behaves correctly.

The relevant diagnostic commands are:

systemctl status arcade-trackball-merger.service --no-pager
journalctl -u arcade-trackball-merger.service -b --no-pager

The service status tells us whether the mechanism is available. The journal tells us what happened during discovery and activation. If Marble Madness launches but only one source works, we can examine the physical input events, the service log, and the virtual output separately instead of treating the whole path as one opaque configuration.

Testing the behavior that matters

A successful service start is not the acceptance test. The user experience is.

We test the complete sequence:

  1. Start in EmulationStation with both physical trackballs available normally.
  2. Launch a listed Marble Madness ROM.
  3. Confirm that the panel trackball moves the marble in both axes.
  4. Confirm that the MX Ergo moves the same player without changing the MAME input assignment.
  5. Alternate between the two devices during the same session.
  6. Exit the game.
  7. Confirm that the physical pointer devices have been released.
  8. Launch a non-listed mouse or trackball application and confirm that it retains its ordinary behavior.
  9. Reboot and repeat the test so success does not depend on device discovery order.

That last test is important. A setup that works only until the next reboot is not a finished cabinet feature.

Building the solution with an agent

As with the cabinet’s controller and metadata tools, the supporting code was developed agentically under my direction.

I defined the behavior and the boundaries: both trackballs should operate Player 1 in Marble Madness, the physical devices should remain normal elsewhere, game selection should be explicit, a failed lighting action must not block a launch, and the result had to survive both game exit and reboot.

The agent helped inspect device behavior, connect the requirement to RetroPie’s launch hooks, implement the merger and allowlist, examine service logs, and refine cleanup behavior. The actual cabinet remained the source of truth. We tested physical movement and game behavior rather than accepting a syntactically correct service file as proof.

This is the kind of task for which agentic development is particularly useful. The individual pieces—Linux input events, virtual devices, systemd, shell hooks, and MAME configuration—are not enormous. The difficulty lies in following their interaction across an entire launch lifecycle and noticing where a broad solution would introduce a new problem.

The human contribution is deciding what “correct” means. Here, correct did not mean “Linux can see two trackballs.” It meant:

  • either trackball controls the same marble;
  • only selected games receive the merged behavior;
  • other pointer-driven systems are unaffected;
  • cleanup occurs after play; and
  • the design can be understood and recovered later.

The larger lesson

The final experience is exactly as simple as we wanted. Launch Marble Madness and use whichever trackball feels right.

That simplicity exists because the complexity has been placed at a deliberate boundary. The game sees one stable logical control. The merger understands the physical alternatives. The launch hooks decide when that behavior is appropriate. Everything else continues to work normally.

Good cabinet integration often looks like nothing happened. There is no controller-selection menu and no instruction to unplug one device before using another. The player rolls a ball, and the marble moves.

That is the entire point.

Retropie Console – Journeys in legacy software installation and USB conflicts

In this arcade project, I’ve chosen to use deprecated solutions often not out of choice, but because despite my best searches, other tools and options seemed not to be available at the time, and I haven’t had time or capacity to build my own.

Screenshot

I’m a big fan of reuse. Why reinvent the wheel when most of the work has been done already and all we need to do is a tweak?

I did this with RGB Commander, and I’ve somewhat done something similar with Pixelcade.

When I found Pixelcade, I attempted an initial install and failed with the V2 instructions. While trying to figure this out, I stumbled on a previous version of the installation page at https://pixelcade.org/pi-install/

Legacy Version – DO NOT USE

There is a particular kind of anxiety that comes from revisiting the installation guide for a working piece of software and finding a large warning at the top. Let’s give it a try, and if I fail, I’ll figure out how to clean this up.

Our cabinet is not a new installation. It’s been slowly curated, cleaned up, and augmented for several years.

It has a working AtGames Legends BitPixel display integrated with RetroPie, EmulationStation, a Raspberry Pi 4, and a Sinden Lightgun among other devices. It starts automatically, follows the system and game selection in the front end, displays artwork, and exposes its companion interface over the network.

I’m documenting this part of the journey for anyone who may be in a similar situation.

What the Pixelcade adds to the cabinet

The BitPixel acts as an active marquee. As we move between systems and games in EmulationStation, the display changes to matching artwork. When a game starts, Pixelcade can show its marquee, scroll “Now Playing” text, and—in supported MAME configurations—cycle high scores.

This behavior is not implemented by one isolated program. Pixelcade sits across several layers of the cabinet:

USB BitPixel hardware
        ↕
Linux device identity and udev rules
        ↕
Pixelcade listener and artwork
        ↕
RetroPie startup configuration
        ↕
EmulationStation event scripts
        ↕
System and game selection

Which architecture and how do we get USB consistent?

The original Java-based installation got the BitPixel running, but the Sinden Lightgun exposed a USB serial-port conflict. Solving that eventually led us to Pixelcade’s newer native listener, an architecture mismatch, a stable device path, and a startup configuration that survives USB numbering changes.

The Raspberry Pi 4 and the version of the OS, which I’m using for this project, has a 64-bit kernel and a 32-bit user space, which means we have to be careful about the software we install. Looking only at uname -m made the machine appear to need an ARM64 program. Programs installed through the operating system actually needed 32-bit ARM binaries.

To be clear, the 32-bit user space is a result of the old operating system I’m using behind RetroPie. We don’t specifically need a fully 64-bit operating system.

Rant: I find it frustrating that there isn’t an upgrade path other than wiping and starting new with Raspberry Pi OS. There’s got to be a better way of migrating easily. Especially given we could test on Raspberry Pi OS 12 Bookworm which supports both 32-bit and 64-bit executables on the Pi4, so we can gradually upgrade to 64-bit emulators which can then better leverage the Mesa and OpenGL drivers and Vulkan support. That, is for another day.

The legacy Pixelcade instructions did work. Their installer did much more than copy one program. It installed Java, created /home/pi/pixelcade, copied artwork, added a Pixelcade listener to RetroPie’s startup, installed fonts, and added EmulationStation event scripts.

This worked beautifully, although Pixelweb, which allows us to download updated artwork, upgrade the software and control the Bitpixel display independent of the console didn’t work.

Still, that gave us an important baseline: We were able to confirm the BitPixel hardware, its USB connection, the artwork, and the EmulationStation integration were all capable of working on the cabinet.

The Sinden Lightgun uses both a camera device and a USB serial interface. When we began configuring it, the gun appeared to be connected correctly, but its software could not claim the serial port it needed.

We checked which process owned the device:

sudo fuser -v /dev/ttyACM0
ps -ef | grep '[j]ava'

The result identified the conflict:

/dev/ttyACM0:  java
java -jar pixelweb.jar -b -s

Pixelcade’s documentation specifically calls out potential conflicts with devices such as Sinden lightguns and USB volume-control boards.

Pixelcade’s Java listener was scanning the available serial devices and had opened the Sinden’s port. The marquee was working, but it had accidentally claimed the lightgun.

Temporarily stopping pixelweb.jar released the port and let the Sinden software proceed. That gave us a useful diagnostic split: both devices worked independently, but automatic serial discovery allowed one to interfere with the other.

Device numbers were not identities

With both devices connected, Linux created paths such as:

/dev/ttyACM0
/dev/ttyACM1

At one point the Sinden was ttyACM0 and the Pixelcade IOIO controller was ttyACM1. After a reboot, they could swap.

Hard-coding – which I’m never a fan of – one of those numbers would therefore make the fix temporary. We needed to identify the hardware rather than its discovery order.

Linux already provided stable paths under /dev/serial/by-id/:

/dev/serial/by-id/usb-IOIO_Open-Source_Project_IOIO-if00
/dev/serial/by-id/usb-Unknown_SindenLightgun_HIDLG-if00

The first path identifies the Pixelcade’s IOIO interface. The second identifies the Sinden. The names continue to point to the right hardware even when the underlying ttyACM numbers change.

The Pixelcade udev rule may additionally create an alias named /dev/PIXELCADE0 or /dev/PIXELCADE1. We saw /dev/PIXELCADE1 point to the same ttyACM1 device as the IOIO by-id path during troubleshooting. We did not use either numbered Pixelcade alias in the final startup command; the IOIO by-id path is more explicit and remained correct when the raw device numbers swapped.

The different ways Linux can recognize a USB device

Linux exposes several views of the same USB hardware. They answer different questions, and not all of them are equally suitable for configuration.

IdentityExampleWhat it tells usStability
Kernel device node/dev/ttyACM0The device class and the order in which the kernel discovered itConvenient, but the number can change at reboot or reconnection
USB bus addressBus 001 Device 006 from lsusbWhere the device is during this bootTemporary; the device number changes
Vendor and product IDvvvv:pppp from lsusbThe manufacturer and product typeUsually stable for a model, but not unique when two identical devices are attached
Serial-based link/dev/serial/by-id/...Identity derived from USB descriptors, often including a serial numberUsually the best application path when the device supplies a useful unique identity
Physical-path link/dev/serial/by-path/...The USB port and hub route through which the device is connectedStable if the cabling and ports stay fixed; changes when the device is moved
udev propertiesID_SERIAL, ID_VENDOR_ID, ID_MODEL_ID, ID_PATHNormalized metadata that rules and services can matchAs stable as the underlying property
Sysfs path and attributes/sys/bus/usb/devices/...The kernel’s complete device tree, interfaces, driver and raw attributesAuthoritative for inspection; parts of the path reflect USB topology
Custom udev alias/dev/pixelcadeA human-readable link created from chosen matching propertiesStable when the rule uniquely identifies the intended hardware

Other device classes have equivalent link directories. Game controllers may appear under /dev/input/by-id/ and /dev/input/by-path/; cameras can have /dev/v4l/by-id/; storage devices add identities such as /dev/disk/by-id/, /dev/disk/by-label/, and /dev/disk/by-uuid/. A filesystem UUID is excellent for mounting a drive, but it does not identify a serial controller or camera.

There are two important caveats. A vendor/product pair identifies a type of device rather than one physical unit. Also, some hardware does not provide a unique serial number. In those cases, by-id may not distinguish two identical units, and by-path or a rule tied to a known physical USB port may be the better choice.

A single USB product can also be a composite device: one physical cable can expose a serial interface, camera, audio device, keyboard or several of these at once. The interface number—often visible as if00, if01, or the ID_USB_INTERFACE_NUM property—can therefore matter just as much as the vendor, product and serial values.

These commands show the device from several angles:

lsusb
lsusb -t
ls -l /dev/serial/by-id/ /dev/serial/by-path/
readlink -f /dev/serial/by-id/usb-IOIO_Open-Source_Project_IOIO-if00
udevadm info --query=property --name=/dev/ttyACM0
udevadm info --attribute-walk --name=/dev/ttyACM0

lsusb is useful for discovery. lsusb -t shows the hub topology and active drivers. udevadm info connects a /dev node to the properties and parent attributes from which a durable rule can be built.

A short primer on udev

The kernel detects the hardware and emits an event. The userspace device manager, udev, receives that event, evaluates its rules, sets permissions and creates the useful symbolic links under /dev. In other words, /dev/serial/by-id/... is not a second serial device—it is another name pointing to the same underlying ttyACM node.

Distribution and package rules normally live under /lib/udev/rules.d/ on this Buster system. Local rules belong under /etc/udev/rules.d/, which keeps cabinet-specific changes separate from files managed by installed packages. Rule files are processed in lexical order, so a descriptive late filename such as 99-arcade-serial.rules is common.

A rule consists of conditions followed by assignments. Match operators such as == ask whether the event belongs to the desired device. Assignment operators set a value, while += adds another value without replacing an existing list.

Common match fields include:

  • ACTION — whether the device is being added, removed or changed;
  • SUBSYSTEM — such as tty, input, video4linux or block;
  • KERNEL — the kernel name pattern, such as ttyACM*;
  • ATTR{...} — an attribute on the current device;
  • ATTRS{...} — an attribute found while walking through a parent device; and
  • ENV{...} — a property such as ID_SERIAL produced during rule processing.

Useful assignments include SYMLINK+= for an additional friendly path, GROUP= and MODE= for access, and TAG+="systemd" when the device should participate in a systemd-managed workflow.

For example, after confirming the exact properties reported for the IOIO interface, a local rule could take this shape:

SUBSYSTEM=="tty", KERNEL=="ttyACM*", \
  ENV{ID_SERIAL}=="IOIO_Open-Source_Project_IOIO", \
  ENV{ID_USB_INTERFACE_NUM}=="00", \
  SYMLINK+="pixelcade"

The values above must be checked against udevadm info on the actual cabinet before installing the rule. A good rule uses enough properties to select exactly one intended interface without depending on its current ttyACM number. If two devices report the same serial metadata, a physical ID_PATH match may be necessary instead.

After saving a rule, reload the rules and reconnect the device:

sudo udevadm control --reload-rules
# Unplug and reconnect the device, then:
udevadm settle
ls -l /dev/pixelcade

For live diagnosis, this shows the kernel event followed by the processed udev event and its properties:

sudo udevadm monitor --kernel --udev --property

There are a few practical boundaries worth preserving. A udev rule should identify the device and establish access; it should not run a long-lived application directly with RUN+=. A systemd service is a better home for a listener such as Pixelcade. Likewise, broadly setting a serial device to world-writable mode is usually unnecessary: giving the service account membership in the appropriate group, commonly dialout, is the cleaner permission model.

Finally, identity and ownership are separate questions. udevadm tells us which device this is. Tools such as fuser or lsof tell us which process currently has it open. We needed both views to solve the Pixelcade/Sinden conflict.

The rule became simple:

Pixelcade → usb-IOIO_Open-Source_Project_IOIO-if00
Sinden    → usb-Unknown_SindenLightgun_HIDLG-if00

Java to Native Application

When testing the Java version of Pixelweb, we found that despite indicating the serial/by-id device, the ttyACM ports would still be held.

The native application provides an explicit -d option for selecting the Pixelcade device by-id. This is the style documented by the current V2 instructions for systems that also have devices such as a Sinden Lightgun.

The native listener offered a cleaner boundary:

./pixelweb \
  -d /dev/serial/by-id/usb-IOIO_Open-Source_Project_IOIO-if00 \
  -image system/retropie.png \
  -startup &

As I alluded to earlier, there was just one problem: Pixelweb would not run.

A misleading “No such file” error

The file was plainly present and executable:

ls -l /home/pi/pixelcade/pixelweb

Yet running it produced:

-bash: ./pixelweb: No such file or directory

That message does not always mean the requested file is missing. Linux also reports it when the executable requires a loader that does not exist on the system.

The installed pixelweb binary was built for the wrong architecture. The Raspberry Pi reported an aarch64 kernel, which suggested 64-bit ARM, but its Buster userspace was 32-bit armhf. A 64-bit Pixelcade binary cannot be loaded by the 32-bit userspace.

The checks that mattered were:

uname -m
dpkg --print-architecture
file /home/pi/pixelcade/pixelweb

The combination was:

Kernel:     aarch64
Userspace:  armhf
Required:   32-bit ARM executable

We replaced the incompatible listener with Pixelcade’s ARMv7 Raspberry Pi build from its official build repository. The working file identified as:

ELF 32-bit LSB executable, ARM, EABI5
interpreter /lib/ld-linux-armhf.so.3

The misleading missing-file error disappeared, and the native listener could finally be started with an explicit device.

The working startup configuration

The functional Pixelcade line in autostart.sh became:

cd /home/pi/pixelcade && \
  ./pixelweb \
  -d /dev/serial/by-id/usb-IOIO_Open-Source_Project_IOIO-if00 \
  -image system/retropie.png \
  -startup &

EmulationStation starts afterward:

emulationstation #auto

This arrangement does four useful things:

  • starts Pixelcade automatically with RetroPie;
  • uses the native listener instead of pixelweb.jar;
  • assigns Pixelcade to its exact IOIO hardware identity; and
  • allows EmulationStation to start while Pixelcade remains in the background.

The old Java line was removed so that both listeners could not start together.

We also corrected the physical USB arrangement so the Pixelcade was no longer buried behind the cascaded external hub. The Sinden remained directly connected. Stable software identities solved the numbering problem, while the cleaner physical layout reduced another source of uncertainty.

The working legacy startup logic does not assume that Pixelcade will always appear at one generic serial path. It looks for:

Proving it survived a reboot

Testing before reboot was not sufficient. The entire problem involved discovery order, so a cold start was part of the acceptance test.

After reboot, the raw numbers did in fact change. The IOIO interface became ttyACM0 and the Sinden became ttyACM1. The stable by-id paths continued to point to the correct devices.

The running Pixelcade process showed the intended command:

./pixelweb -d /dev/serial/by-id/usb-IOIO_Open-Source_Project_IOIO-if00 \
  -image system/retropie.png -startup

Port ownership also showed the desired separation:

Pixelcade pixelweb → IOIO serial device
Sinden Mono app    → Sinden serial device

We then checked the behavior that actually mattered:

  1. Pixelcade started with the cabinet.
  2. The BitPixel responded while browsing EmulationStation.
  3. System and game artwork continued to display.
  4. The Sinden software could claim its own serial interface.
  5. The two devices remained separate when ttyACM0 and ttyACM1 swapped.

That was the point at which the setup was working.

Legacy does not mean worthless

The installation began with the legacy V1 procedure because the V2 installer did not work on this cabinet. Thinking about it, I wonder if it may have had to do with the mixed 64-bit/32-bit nature of the current setup.

Preservation matters. We used a legacy installer to establish a working baseline, allowed the software to evolve, adopted the newer native listener when it solved a real limitation, selected the build that matched the actual userspace, and pinned it to the hardware identity

The Pixelcade folks keeping the legacy instructions and having an upgrade path to the newer version is awesome. To me, this is how software should be developed with their upgrade paths. I can’t express how much I appreciated this.

The warning on Pixelcade’s old page is useful. It tells new users that there is a supported path they should use instead. It also tells owners of existing installations that the assumptions behind their systems are aging and deserve attention.

The BitPixel now behaves like part of the cabinet. It starts automatically, follows the front end, displays its artwork, and leaves the Sinden alone.

This is, yet, another reason why I believe in preservation, archiving old releases and making those releases available for reuse. Two examples stand out as part of this project – RGB Commander which I found via the Internet Wayback Machine, and now Pixelcade.