← Pharos Data Ark

PHAROS DATA ARK · 2026-10-29-01024f5b59

Handbook contents

Configuration reference

How to read this reference

This reference describes the deployed Ark and the reproducible configuration sources. Reviewed for obsolete instructions on October 28, 2026; machine-specific observations still require verification on a replacement device. Installed helper copies are not automatically updated by editing the repository. Site-specific secrets belong in the root-only configuration capture, not this page.

Do not paste all commands into a running appliance. Network, mount and firewall changes can sever access or interrupt real work. For a new build, use spare media and a local recovery console until the bootstrap process has been validated.

Base system and prerequisites

The private snapshot records the installed package list and Python freeze. A clean-image dependency recipe and fixed container digest still need validation; the running Kiwix deployment uses a latest image reference, which is not an immutable software version.

Observed hardware and boot settings

The reviewed storage device identifies as MZVLB1T0HALR (about 954 GiB), behind a JMicron USB storage bridge; the SD capacity is about 116.5 GiB. A Netgear A6150/Realtek RTL8812BU USB Wi-Fi adapter supplies the additional radio. Multiple USB hubs are present. The storage bridge appeared in the USB 2.0 bus tree during this review; do not assume the USB 3-capable bridge is actually using a SuperSpeed path. Port placement, hub power and actual I/O performance need verification, especially when readers stall during writes.

The board has 2 GB installed RAM, with roughly 1.8 GiB available to the operating system and zram swap present. Power-supply rating, enclosure cooling, optional RTC/UPS hardware and spare-media restore results are not established by this inventory. Record them after physical inspection instead of inventing a hardware bill of materials.

Observed boot configuration includes arm_64bit=1, auto_initramfs=1, vc4-kms-v3d, arm_boost=1, audio/camera/display auto-detection and display defaults. Conditional cm4/cm5 entries are image defaults, not Pi 4 requirements. The observed config is saved as a review reference, not a portable prescription for all boards.

Hostname is dataark; timezone is America/New_York and NTP was synchronized at review. An offline appliance still needs a clock strategy; previous online synchronization does not establish accurate time after every powerless boot. Both 99-ip-forward.conf and 99-ipforward.conf exist under /etc/sysctl.d; inspect effective settings and duplicates before consolidating them.

Filesystem and ownership

The content SSD is mounted at /mnt/storage as exFAT, volume label OfflineRepo. The observed fstab entry uses content UUID 22CB-1D5C and options defaults,uid=1000,gid=1000,umask=000 with dump/fsck fields 0 0. This UUID is specific to this drive. On another disk, discover its UUID and verify the intended device; do not reuse it or format a disk automatically.

The reviewed admin account maps to UID/GID 1000. exFAT presents permissions from its mount options rather than ordinary per-file Unix ownership semantics. Public read-only access therefore depends on service configuration and, for FTP, a read-only mount namespace. A writable local SSD does not imply anonymous users may write to it.

The SD root uses ext4 with noatime, and /boot/firmware uses vfat. Their PARTUUID values are installation-specific. Boot firmware config and kernel command-line settings are captured privately for comparison; do not copy SD partition identifiers to new media.

NetworkManager hotspot and upstream

The DataArk hotspot uses AP mode, autoconnect, ipv4.method shared and 192.168.4.1/24. The built-in radio (normally wlan0) is the default hotspot; the USB radio (normally wlan1) is the default upstream client. Either AP-capable radio can be assigned to the hotspot. Read the current roles from /run/dataark-wifi-roles.json; preferred hardware identity is in /etc/dataark/wifi-roles.json. Use Wi-Fi adapter assignment, not a hard-coded interface name, when configuring or diagnosing a running Ark. Credentials remain site-specific and private.

Keep the AP subnet distinct from the upstream network. The interface names are part of today's deployment, not a portable hardware guarantee. A bootstrap must verify AP capability and select the correct physical interfaces before writing profiles. The kernel's observed net.ipv4.ip_forward setting is 1.

The hostname dataark is useful where local resolution works. The portal explicitly supplies a local dataark answer on the hotspot, but clients can have different search/mDNS policies. The fixed hotspot address remains the manual fallback.

Apache and API

The observed service and virtual-host templates are tracked under deploy/reference for reconstruction review. They are not installed automatically. Editing Python requires API restart; publishing static files does not restart the API.

The web deployment script renders the dashboard, builds docs in a fresh temporary directory, and rsyncs static output. Documentation deployment explicitly uses --chmod=D755,F644; missing search permissions previously caused Help to return access errors. Help links directly to /docs/SUMMARY.html.

Captive portal and pass-through

The portal installer writes /etc/dataark/portal-dns.conf, portal.nft, Apache portal configuration and dataark-portal-dns.service. Dedicated dnsmasq listens on 1053 on the AP, forwards ordinary questions to 192.168.4.1:53, has a cache of 150 and local TTL 30, and supplies dataark at 192.168.4.1. It deliberately does not replace known OS discovery hosts with private addresses indiscriminately.

The ip dataark_portal table redirects AP-to-Ark TCP/UDP DNS requests from 53 to 1053. The service validates dnsmasq configuration, installs its owned table, and removes that table on stop. Do not flush the entire nftables ruleset: Docker and NetworkManager have separate requirements.

The pass-through installer owns inet dataark_forward. Timed MAC and IPv4 sets expire after six hours. Unapproved AP clients' nonlocal HTTP is directed to the separate 8081 welcome listener; local Pi addresses remain local. Forwarding rules preserve the Docker-DNAT Kiwix path and reject unauthorized AP forwarding, including IPv6. An approved client is checked against the per-device allowlists; permission is not a global switch and does not persist across reboot.

Apache overwrites X-DataArk-Peer from the connection peer and removes client-supplied server identity. Port 8081 only accepts AP-subnet callers and redirects to http://192.168.4.1/?portal=1. Portal mutation requires the expected local Origin and verified AP peer. Direct access to the backend would undermine those assumptions; keep it loopback-only.

Installers: deploy/install_portal.py and deploy/install_pass_through.py. Verify discovery and manual access from a real client and test offline separately; a running DNS daemon does not establish that all phone pop-ups behave identically.

Kiwix container

The existing compose source is /home/admin/kiwix-server/docker-compose.yml. The container name is kiwix-serve, published on host port 8080, restart policy unless-stopped, startup command glob *.zim and /mnt/storage mounted at /data. The observed bind is read-write; do not describe the container mount as read-only without changing and testing it. The activation tool checks the reviewed entrypoint, command and content mount before switching archives.

The collection catalogue and article-serving path need separate checks. Startup after a restart can take time to read archives. A Docker process in running state, HTTP homepage or catalogue is not sufficient evidence that article/image requests work.

Content updater and manifest

dataark.yaml controls repository visibility, registered asset paths, source series, hashes, update policy, map coverage, resource indexes and household job configuration. hide_missing is true. Removed files retain records marked removed with manual/never updates; reinstatement re-enables a supported source. Source details and published runtime state can differ from the last Git commit, so capture the live manifest too.

dataark-update.timer uses OnBootSec=2min, OnUnitInactiveSec=5min and AccuracySec=30s. Per-asset refresh intervals decide which publisher checks are due. Its root oneshot has ConditionPathIsMountPoint=/mnt/storage, Nice=10, idle I/O scheduling, unlimited start duration, 45-second stop allowance and control-group termination.

Worker state is /var/lib/dataark/updates; downloads and transaction files are in /mnt/storage/.dataark-updates. Locks, publisher size/hash checks, snapshot-change detection, reader catalogue checks and rollback journals are required parts of the lifecycle. Do not delete them to clear a failed status. Deploy/install_update_worker.py creates the unit/timer.

Library/network administration authentication

/etc/dataark/network-admin.json contains a salted PBKDF2-SHA256 hash, not the original shared password. The current implementation uses 300000 iterations. The directory is root:admin 0750 and the credential file root:admin 0640 so the API account can verify it. Do not publish its contents.

Root-owned helpers live under /usr/local/libexec. Reviewed sudoers files allow the network, library and pass-through helpers with no command-line arguments. Library operations accept only fixed operations and registered identifiers; callers do not supply shell commands or arbitrary deletion/download paths.

Deploy/install_network_admin.py preserves the existing credential unless explicitly reset. Deploy/install_library.py updates its installed helper but does not initiate a collection download. The shared caretaker prompt optionally caches authorization for a chosen fixed 1–120 minute duration in tab-scoped session storage. Reload and same-address navigation retain that cache; expiry, explicit locking or 401 clears it. Removal still requires confirmation.

This application uses local HTTP, not encrypted management transport. The caretaker password is not an SSH or Samba account. Keep credential use within the intended local deployment boundary and document any future transport/authentication changes.

Samba

/etc/samba/smb.conf defines Offline-Archives at /mnt/storage, read-only, with guest access offered subject to client policy. Bugout-Staging is a separate read-only legacy area at /mnt/storage/bugout-drop. The dataark development share points to /opt/dataark and is writable according to its account policy. Do not change the development share into an anonymous public source server merely to provide source references.

Deploy/install_content_access.py validates candidates with testparm and Apache configtest, preserves configuration backups and maintains public content exclusions. SMB account state is separate from configuration text; the private capture includes Samba private state so a maintainer can review recovery needs. Do not copy that state into the public library.

Anonymous FTP

Tracked configuration: deploy/ftp/vsftpd.conf and deploy/ftp/dataark.conf. Install the Debian vsftpd package, then use deploy/install_ftp.py. The installer backs up existing configuration, installs a systemd drop-in and enables/restarts the server.

Anonymous login is allowed without a password; local account FTP login and all writes are disabled. FTP control is port 21, passive data ports are 30000–30009, active mode is disabled, max clients is 12 and max connections per IP is 4. Idle timeout is 300 seconds and connection/accept timeouts are 60 seconds. Only world-readable files can be downloaded anonymously.

anon_root is /srv/dataark-ftp. The service privately binds /mnt/storage read-only at /srv/dataark-ftp/library, requires the SSD mount, protects home/system paths and masks private/staging/system-metadata directories. hide_file and deny_file provide additional exclusions. Clients enter /library. File additions/removals appear through the live directory bind; individual archives must not be pinned with per-file bind mounts that retain deleted storage.

The exact drop-in masks .dataark-updates, .bugout-staging, bugout-drop, System Volume Information and $RECYCLE.BIN. Verify anonymous listing, small download, resume, rejected writes and denied directory entry after changes. Logs are normal vsftpd transfer/service logs, not the library management event stream.

SSH, SFTP and rsync

Observed SSH policy: public-key authentication enabled, password authentication enabled, root login without-password, and the SFTP subsystem at /usr/lib/openssh/sftp-server. Preserve the intended account/key policy rather than resetting it to match a convenience password. SSH listens on 22. SFTP and rsync over SSH use that account's permissions. No anonymous rsync daemon or WebDAV server is configured.

SSH host keys, authorized keys and account private keys have distinct roles. The private configuration capture includes host configuration/identity and selected authorized-key files; account private keys need their own secure custody. A rebuild must decide explicitly whether to preserve the old host identity or introduce a new one and verify fingerprints.

Maps and household backups

The maps block of dataark.yaml enables monthly checks (refresh_days 30), reserve_gib 10, max_run_gib 8 and configured flood/parcels/topographic sources. Coverage is Massachusetts, Worcester and Middlesex counties, with Northern Worcester as the descriptive focus. Geographic scope is not a promise of complete data or an interactive basemap. See Local reference maps for downloader/coverage limits and service setup.

Household configuration is a separate bugout block with source-specific credentials and normal maintenance/bugout policies. The current configured manuals source uses daily prefetch; do not publish its actual credentials or assume an integration token exists merely because example HA YAML is present. See Household backups for state, outbox, encryption and cancellation semantics.

Diagnostics, permissions and configuration capture

Debug is represented by /var/lib/ark/debug_status. The common structured event stream is /var/lib/ark/events.log and exposed through /api/events. Library basic outcomes are recorded with Debug off; extra stages and coarse progress require Debug. Authentication headers and raw request bodies must not be logged.

dataark-portal-monitor.service runs a bounded metadata capture only while Debug is on. Its root-only log directory is /var/log/dataark-portal-monitor. The service limits CPU to 5 percent, memory to 64M, uses Nice=10 and restricts writable paths and capabilities. Four rotated 1 MiB files bound its local capture. The packet monitor is not the library application log.

Run tools/capture_config.py as root before significant changes. Its root-only archive contains sensitive settings, SSH host identity and Samba state; never commit or publish it. The capture now includes FTP, root helper copies, boot configuration, live manifest, package/dependency inventory, container inspection and repository revision/dirty status as well as the earlier network/web/service settings. Content libraries and source/private keys outside the listed scope still require separate backups.

Known reconstruction gaps

Existing installers cover feature-specific configuration but do not fully provision a clean Pi image. The base image, API unit/vhost creation, Docker/image pinning, physical interface mapping, user creation, storage provisioning and full rebuild validation need integration into the bootstrap described in Bootstrap plan. The settings above are an observed reference, not a fully tested reinstall recipe.

Temporarily disable upstream Wi-Fi

Open /config/wifi.html through the DataArk hotspot at http://192.168.4.1.

The enable/disable slider switch requires caretaker authorization.

Disable disconnects only the assigned upstream Wi-Fi interface using NetworkManager’s per-device disconnect operation;

it preserves saved SSIDs, passwords and profile autoconnect settings. The assigned hotspot radio

remains running. Enable reconnects the upstream radio to a suitable saved network. A restart can

also restore upstream Wi-Fi; this is an offline-test control, not a persistent radio block.

Never use nmcli radio wifi off for this purpose: that disables both wireless adapters.

For a complete offline test, disable upstream Wi-Fi and Ethernet through their pop-ups,

and disconnect other internet routes on the client (Ethernet or cellular data).

Keep the client on DataArk Wi-Fi. Disabling through the upstream address can end

that browser connection; use 192.168.4.1 so the controls remain reachable.

The dashboard’s Wi-Fi and Ethernet pop-ups each have an enable slider switch. Each

change requires caretaker authorization, using the optional cache when valid. Ethernet uses a fixed eth0 device

disconnect/connect operation and retains its saved profile. Both controls are

temporary until re-enabled or restarted. Use the hotspot address 192.168.4.1

before disabling either upstream route; disabling both simulates isolation

without unplugging cables. Client-side Ethernet/cellular routes must also be off.

Protect the current access path

Wi-Fi, Ethernet and Hotspot pop-ups provide password-protected enable slider switches.

The hotspot uses the saved DataArk profile on the currently assigned hotspot radio. Disabling or changing

the current access path is refused by the privileged helper, using Apache's

overwritten client address, local kernel HTTP socket records and the return

route. On a shared upstream subnet both receiving and return interfaces may be

protected. Missing or unrecognized connection information fails closed.

Hotspot clients cannot disable or rename the hotspot even using a LAN URL.

Use a different interface before changing the current connection. The Pi's

local keyboard/KVM remains an independent recovery path.

Caretaker actions use an on-demand password prompt. Remembering is optional: choose 1–120 minutes (default 10). The fixed expiry does not extend with use. The cache survives navigation and reload in the same browser tab and Ark address using tab-scoped session storage; it is not shared across addresses or independent tabs. Expiry, a rejected credential, Lock now, or the floating red unlocked padlock clears it. The padlock shows remaining minutes; the inline indicator gives the same countdown. Without remembering, only that action is authorized. A rejected password explicitly says Password not accepted. The server verifies every protected request.

Reboot diagnostics

deploy/install_reliability_logging.py installs /etc/systemd/journald.conf.d/dataark-reliability.conf: persistent local journal, SystemMaxUse=64M, RuntimeMaxUse=16M, SystemKeepFree=512M, MaxRetentionSec=7day, SyncIntervalSec=30s. Logs remain on the system disk; they are not a public library asset. The stock-Pi coordinator includes this installer. Use journalctl --list-boots and sudo journalctl -b -1 -k after another incident. A sudden power loss or kernel lockup can still prevent the last records from being written. This does not change watchdog or network settings and does not by itself establish or fix a reboot cause.

Reference publisher checks and pending source changes

deploy/reference/reference-checks.json configures a weekly check of installed

non-Kiwix publisher sources. dataark-reference-checks.timer wakes the checker

hourly; the per-source interval determines which URLs are due. Sources shared

by many assets (such as U.S. Code volumes) are checked once, not once per file.

Removed content is excluded. Generated print packs are labelled as generated;

the derived callsign database displays checks against its FCC source package.

Manage Library shows the interval, last attempted check, next due time and

result for each asset. The Source changes and check failures section lists

changed sources separately from failed HTTP/validation checks. These are review

items, not queued downloads. An HTTP 403/404 or invalid response never removes

the working offline copy. Pending change flags survive a subsequent failed check

or HTTP 304. A later response matching the original baseline clears the flag.

PDF checks validate the PDF structure before comparing hashes. Word packages

must contain document XML and no VBA project. HTML comparisons ignore script

and style contents and normalize whitespace. First-time HTML and package checks

establish a publisher baseline; they do not certify freshness of the saved

snapshot. ZIP/PMTiles packages use HEAD metadata to avoid repeatedly transferring

large archives. Sources without version metadata are labelled accordingly.

Checks follow the recorded URL by default. Reviewed release-family rules in

deploy/reference/edition-discovery.json also discover dated publisher folders,

filenames and links on explicit publisher indexes. Initial rules cover annual

IRS forms and Part 97; a validated candidate is reported for review and prior

editions remain installed. Exam-pool and other dataset releases still need

their collection-specific checks. See Edition discovery.

Kiwix release discovery and automatic activation continue through

dataark-update.timer. Specialist Massachusetts

GIS collection refresh remains controlled by maps.refresh_days (currently 30)

and dataark-maps.timer; the large street-basemap extract is a separate pinned

snapshot, not that GIS collection.

Reference checks deliberately do not overwrite files. A changed source must be

reviewed and reacquired with the appropriate collection installer or Capture

workflow so edition labels, transformations, companion files and provenance stay

consistent. Replacing a PDF alone is insufficient for an extracted code package,

FCC database, bundled reader or transformed HTML snapshot. There is no automatic

promotion of a detected change into the installed library.

State: /var/lib/dataark/reference-checks/state.json (public-source fingerprints,

ETag, Last-Modified, last attempt/success and result; no user documents). Inspect:

journalctl -u dataark-reference-checks.service. Run due checks now:

sudo systemctl start dataark-reference-checks.service. Configuration travels in

the content profile and recovery kit; check history starts fresh on a rebuilt

system. The stock-Pi bootstrap installs the timer with automatic content updates.

The service runs as admin with the repository and content SSD read-only, bounded

responses (64 MiB), a 45-second network timeout and 256 MiB memory limit.

To retry failures once without repeating completed checks, stop the timer's

current service, then run as admin:

/opt/dataark/venv/bin/python /opt/dataark/tools/check_reference_sources.py --retry-errors.

A worker lock prevents overlapping passes. Honda checks use the recorded

intermediate certificate with system roots and hostname verification intact.

Package hosts that refuse HEAD (HTTP 403/405/501) get a one-byte Range GET fallback.

The response is closed without reading the package body, and Content-Range's

full size is used rather than the one-byte response length. This permits FCC

metadata checks without downloading the roughly 200 MB callbook on each check.

Dashboard popup switches and refresh interval

The six on/off status controls (Ethernet, Wi-Fi, Hotspot, Internet pass-through,

Auto refresh and Debug) use accessible slider switches at the upper right of

their pop-ups. Opening a pop-up does not change state. Network authorization and

current-access-path protections are unchanged. The optional remember switch in the

caretaker prompt is off by default, with a selectable 1–120 minute duration (default 10).

Auto refresh accepts whole-number intervals from 5 through 300 seconds. Choose

Apply interval (or press Enter) to apply and save the choice in this browser's

local storage (dataark.dashboard.refreshIntervalSeconds). The default is five

seconds. Invalid/stale stored values fall back to the default. Other devices and

browser profiles keep their own choices. Changing the interval while paused does

not start polling. Resuming uses the selected interval. A page reload starts

Auto refresh again and retains its saved interval. The System pop-up's live

library statistics follow dashboard refreshes; initial/opening requests remain

available even when periodic refresh is paused. No server update/download

schedule is changed by this display preference.

Household content interests

The optional personalization section in dataark.yaml records vehicles, preferred destinations, languages and jurisdictions, with validated links to registered content. It is preserved in portable content exports. See Personalization for its schema and sharing considerations.

Configuration editor

Open Settings → System → Configuration… and choose Load configuration.

The editor opens the small ark.yaml; choose collection definitions, publication

catalogues or read-only version snapshots with Configuration file. See

Modular public configuration for the layout and CLI tools.

Reading, downloading, viewing previous public versions and validating a draft

require no caretaker password. Private household settings are excluded from these

views. Saving and publishing changes, restoring a version by publishing it, and

changing Community visibility require caretaker authorization, using the shared

optional timed remember switch. Loading a version into the editor does not apply it.

Community visibility

The slide switch controls this content setting:


community:
  enabled: true

false hides the Community card and its landing-page header link. It does not

delete messages or disable direct access to /community.html. Older configurations

without the section default to visible. This option travels in the public content

manifest and its portable configuration mirror.

Editing YAML

The editor opens public modular configuration files; the combined dataark.yaml

compatibility snapshot is an advanced option. Private household and authentication

settings are excluded. The folder picker, section selector and search sit directly

above the editor. The active filename and an unsaved-changes indicator identify the

current draft. Switching files clears the editor immediately while loading and

resets the section choices; a sole section is selected automatically and disabled.

How configuration files work and History are expandable disclosures. YAML

download, validation and a card summary remain available.

Storage/service paths and the schema version cannot be changed in this editor.

Use spaces for indentation and keep boolean values as true or false.

1. Load the current configuration and optionally download a copy.

2. Edit the YAML. Validate & preview checks syntax, duplicate keys, supported

sections, manifest references and values. The preview is a configuration

summary; actual visibility also depends on missing-content settings.

3. Save & publish checks the loaded revision again. A conflicting update is

rejected instead of overwriting it. The publisher also shares the updater lock;

wait and retry if an update is running.

4. Wait for Configuration saved and published. Publication runs independently

of the browser. Downloaded content is not removed and no collection downloads are started.

Private settings are merged from the current appliance configuration. The raw

public YAML is saved to config/content/public-manifest.yaml, retaining editor

comments there; the operational dataark.yaml is serialized after merging. The

desired restoration recipe is preserved rather than recreated from the current

disk inventory. Its public manifest is mirrored to DATA-ARK-CONFIG when the

content drive is mounted.

This first editor covers the public manifest. Acquisition catalogues under

deploy/reference and the restoration recipe in config/content/content.yaml

remain separate configuration files. It does not complete the planned migration

of every collection-specific installer decision into configuration.

Previous versions and recovery

Every save keeps the previous public configuration under History.

Load a version into the editor, inspect it, validate it and publish to restore it.

The list shows the latest 20 versions. A full private pre-change manifest is also

kept root-only under /var/backups/dataark-configuration for console recovery.

These copies are on the system drive; keep exported YAML on another device too.

On a publication failure the worker restores the previous files and attempts to

republish them. Any recovery failure is reported rather than silently marked

successful. If interrupted, reload to inspect the current configuration and use a

saved version when needed. A stale publication state is cleared when the service

is no longer running. Progress/history live in /var/lib/dataark/configuration.

The fixed service is dataark-configuration-publish.service; install it with

deploy/install_configuration.py (also called by the network-administration

installer). It is invoked through the existing fixed-scope administration helper,

not a general shell interface. No new general-purpose sudo permission is added.

YAML syntax coloring

The configuration editor colors keys, quoted strings, numbers, booleans, comments

and anchors using bundled offline JavaScript and CSS. It retains the native text

area for selection, keyboard editing, undo, search and downloading unchanged YAML.

Only visible lines are rendered in the coloring layer, keeping large manifests

responsive. Color is a reading aid; Validate & preview checks the actual YAML

and manifest rules. High-contrast system modes retain readable native text.

Fresh authentication for credential changes

Changing the caretaker password must always require the current password again, regardless of a cached caretaker session. Browser implementations must call CaretakerAuth.freshAuthorization(purpose) (or authorization(purpose, {forceFresh:true})), rather than the normal cached helper. This clears the previous cache, displays a fresh password prompt with remembering disabled, and cannot fall back to cached authorization if cancelled. Password-change success must leave the session locked. The current browser UI does not provide a caretaker-password change form; the helper is the required interface when that form is added. Server-side password rotation must verify the current credential independently.