<# boot.ps1 -- UNIVERSAL Windows remote-access enrollment (Tailscale + standalone OpenSSH) ======================================================================================== THIS IS THE CANONICAL SCRIPT, served at https://boot.mad.sg. Run it on any Windows box with: irm boot.mad.sg | iex It is the successor to machines/bettina-office/enroll.ps1 (the "golden" script): it keeps everything that script learned and adds STATE-ADAPTIVE behaviour. Whatever state the machine is in, boot.ps1 ends with a WORKING SSH CHANNEL for the operator (or says loudly and precisely why it could not, and what a human must do). REPORT-ONLY DESIGN (2026-07-11) -- THIS SCRIPT IS NEVER DESTRUCTIVE ------------------------------------------------------------------- boot.ps1 is a SAFE boot loader: detect state, install Tailscale + standalone OpenSSH + every backup (self-heal tasks, crash recovery, persistence), detect and NAME any third-party AV and rate its risk (Get-AvAdvice), spin off the decoupled Zoho installer, REPORT it all - then STOP. It NEVER removes an AV, NEVER reboots, and NEVER touches boot configuration / Safe Mode. WHY: the previous design auto-ran an integrated McAfee headless removal in PATH B. On 2026-07-11 that auto-removal bricked a machine's reachability - it set up a Safe Mode boot cycle that got stuck, before anyone could reach the console. AV removal is now PHASE 2: a SEPARATE, operator-directed step OUTSIDE this script. Preferred: uninstall via the Zoho remote-GUI once Zoho is installed, or at the console. The standalone remove-mcafee.ps1 / remove-mcafee-safemode.ps1 in this same dir remain available as OPERATOR-RUN tools - boot.ps1 never invokes them. THE THREE STATES IT HANDLES (same command, no parameters needed) ---------------------------------------------------------------- (a) BRAND-NEW box: installs Tailscale + standalone OpenSSH, enrolls (prompting for a tagged pre-auth key only if actually needed), permanent reboot-proof sshd service. PATH A. (b) HALF-ENROLLED box: idempotent - every step detects what is already done, repairs what is broken (ACLs, host keys, firewall scoping, unattended registry), and re-verifies. Safe to re-run any number of times. (c) UP BUT FIGHTING A THIRD-PARTY AV (McAfee - the real situation this script was born from): guarantees access via an interactive-session sshd that the AV tolerates, NAMES the AV + rates its risk, and REPORTS - removal is a separate operator-directed step. The permanent service is pre-staged so a re-run after the AV is removed flips cleanly. PATH B. LESSONS BAKED IN (every one verified the hard way on the Fujitsu ESPRIMO P558, Win10 22H2 + OEM McAfee LiveSafe, enrollment nights 2026-07-10/11) ------------------------------------------------------------------ 1. ExecutionPolicy Restricted under `irm | iex` blocks child .ps1 scripts. -> Set-ExecutionPolicy Process Bypass at the very top, and ALL child .ps1 files are invoked via `powershell.exe -ExecutionPolicy Bypass -File ...`, NEVER via `&` (which inherits a possibly-Restricted machine policy). 2. Fresh OpenSSH has no host keys -> sshd "no hostkeys available", service dead. -> explicit `ssh-keygen.exe -A` + verify C:\ProgramData\ssh\ssh_host_*_key exist BEFORE any start attempt. 3. Move-Item from %TEMP% drags user-profile ACLs into Program Files\OpenSSH, so LocalSystem / NT SERVICE\EventLog cannot read the files (dead ETW provider = sshd failures INVISIBLE in Event Viewer). -> `icacls "C:\Program Files\OpenSSH" /reset /t /c` on every run; host private keys locked to SYSTEM:(F) + Administrators:(R), inheritance off. 4. A firewall rule can EXIST yet be profile-scoped away from the Tailscale adapter -> SSH times out (not refused). -> rule recreated every run with -Profile Any AND -RemoteAddress 100.64.0.0/10 (Tailscale CGNAT range only, NOT wide open), then VERIFIED by reading it back. 5. Third-party AV (McAfee) lets an INTERACTIVE sshd run but KILLS the sshd SERVICE (WIN32_EXIT_CODE 1067 "terminated unexpectedly"; invisible in Event Viewer because of lesson 3). Self-heal/crash-recovery cannot fix this: the AV just re-kills sshd and the restart churn destabilises live connections. -> EARLY detection via root/SecurityCenter2 AntiVirusProduct + McAfee service/process probes, then PATH B (below). 6. Tailscale must reconnect at boot with NOBODY logged in. The CLI --unattended flag does NOT set this on Windows; only the registry value HKLM\SOFTWARE\Tailscale IPN\UnattendedMode = 'always' does. Set AND verified. 7. McAfee CANNOT be removed headlessly on current builds (PROVEN 2026-07-10): - mccleanup.exe (current MCPR, ~12.6 MB) has a PARENT-PROCESS GUARD: it logs "ValidateParentProcess failed" and aborts (exit 2) unless launched by McAfee's own signed GUI front-end, McClnUI.exe. It will NOT run silently over SSH or from SYSTEM. - SELF-PROTECTION: the mfevtp validation service + the mfehidk kernel driver make every SYSTEM-level Stop-Service / Set-Service Disabled / process kill return Access Denied. Do NOT try to force-stop it - on 2026-07-10 that backfired into a temporary lockout. - The sanctioned uninstaller (registry UninstallString -> mcuihost.exe ... /id:uninstall) is GUI-only too. -> REMOVAL REQUIRES A HUMAN AT THE CONSOLE (physically or via remote desktop): Settings > Apps > McAfee LiveSafe > Uninstall, or run MCPR / McClnUI interactively and click through. Windows Defender re-activates automatically afterwards. This script performs NO removal attempt of any kind (see the REPORT-ONLY DESIGN note above). -> A headless-removal path (self-protection drop via McAfee's own mc-sec-unprotector.exe + a pre-guard mccleanup.exe build) exists in the STANDALONE, OPERATOR-RUN tools remove-mcafee.ps1 / remove-mcafee-safemode.ps1 in this same dir. It used to be integrated here and auto-run by PATH B; that was REMOVED on 2026-07-11 after it broke a machine's reachability (see the REPORT-ONLY DESIGN note). 8. McAfee TOLERATES an sshd launched in an INTERACTIVE session (session id >= 1) but kills any session-0 background/service sshd. A DETACHED, hidden `sshd.exe -D` started with Start-Process SURVIVES the console window being closed and keeps SSH alive - the exact trick that restored access on 2026-07-10. It does NOT survive reboot or logoff. 9. REBOOT SAFETY GATE: while a third-party AV is present, the box is NEVER reboot-safe. After a reboot with nobody logged in, only the session-0 service exists, the AV kills it, and the machine is LOCKED OUT until a human reaches the console. boot.ps1 refuses to present the box as reboot-safe while the AV is present. 10. AUTO-LOGIN FALLBACK (documented, NOT auto-configured): because the AV tolerates interactive-session sshd, configuring Windows auto-login for the admin account plus a logon-triggered `sshd -D` task WOULD survive reboots even with McAfee installed. Tradeoff: the desktop auto-unlocks at boot (a security downgrade), and configuring it needs the account password - so it is left as a deliberate human decision, not automated here. 11. Tailscale SSH is NOT a Windows escape hatch: its server component runs on Linux and macOS only (verified against tailscale.com/kb/1193, 2026-07-11). Windows is client-only, so a hostile AV cannot be routed around that way. THE TWO PATHS (clearly labelled in the RESULT block) ---------------------------------------------------- PATH A - CLEAN (no third-party AV): permanent Automatic sshd service + crash-recovery + self-heal tasks; reboot-survivable; full PASS table. PATH B - AV-PRESENT, REPORT-ONLY (lifeline + report, then STOP). In an INTERACTIVE session it (1) guarantees a channel via a detached interactive-session sshd the AV tolerates (the lifeline), then (2) REPORTS the detected AV (name + risk via Get-AvAdvice) and STOPS. It does NOT remove the AV, does NOT reboot, does NOT touch boot configuration. AV removal is a SEPARATE, operator-directed step (Zoho remote-GUI once installed, or the console); the RESULT states loudly that the box is NOT reboot-safe until the AV is removed and boot.ps1 is re-run. In a HEADLESS session 0 run it never stops/kills any sshd (it may be serving this very connection) - it just reports. The permanent service is pre-staged on both paths so the post-removal re-run flips cleanly to PATH A. RUN MODES (both supported) -------------------------- A) Piped, no params (the canonical way): irm boot.mad.sg | iex Already-enrolled machines never need an auth key; it is only prompted for when Tailscale is genuinely down/logged-out (Enter skips). The iex path NEVER hard-fails on missing parameters. B) File with params: powershell -ExecutionPolicy Bypass -File boot.ps1 -AuthKey "tskey-auth-XXXX" -NodeName "my-node" -ExpectedTag "tag:bettina" TAILSCALE AUTH KEY REQUIREMENT ------------------------------ The auth key should be a PRE-AUTH KEY created WITH A TAG (tag:bettina, tag:aunt, ...) so the node enrolls tagged under the one-way ACL model. An untagged node can reach back into the tailnet - that is a security gap; the script warns. Idempotent: safe to re-run any number of times, in any machine state. Output is PURE ASCII on purpose (operator consoles mangle anything else). #> param( [string]$AuthKey = '', # tagged Tailscale pre-auth key [string]$NodeName = $env:COMPUTERNAME, # tailnet hostname [string]$ExpectedTag = '' # e.g. 'tag:bettina' - verified if given ) # ============================================================================ # 0. PROCESS SETUP (must be first -- fixes lesson 1 for everything below) # ============================================================================ Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass -Force $ErrorActionPreference = 'Stop' try { [Net.ServicePointManager]::SecurityProtocol = ` [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12 } catch { } $BOOT_URL = 'https://boot.mad.sg' # where this script lives; used for self-elevation + re-run guidance # Bootloader version: a short content hash injected at DEPLOY time by # deploy-boot-worker.py (it replaces 31a34d08 with sha1(boot.ps1)[:8]). # Stamped into the run header + the summary so we can tell WHICH version ran and # whether it worked. If you see the literal placeholder, the script was served raw. $BOOT_VERSION = '31a34d08' # ---------------- remote progress log (best-effort, NEVER blocks) ---------------- # Detailed, streamed progress relay: batches plain-text lines and POSTs them as # JSON to the mad1 collector (boot-log.madcats.sg -> Cloudflare -> Traefik -> # /srv/boot-logs/boot.log), read over SSH. Plain HTTPS, needs nothing installed # on the machine (same transport as the script download). A SEPARATE, concise # end-of-run summary goes to $BOOT_URL/summary -> the mad-messages Telegram bot. # HARD RULES: fire-and-forget (try/catch everything, no throw, short timeout) - # logging must NEVER break or block the enrol. NEVER log secrets (auth keys / # tokens are never passed through the output helpers below). $script:RemoteLogUrl = 'https://boot-log.madcats.sg/ingest' # detailed streamed log -> mad1 $script:SummaryUrl = $BOOT_URL + '/summary' # concise end-of-run summary -> mad-messages $script:RunId = $env:COMPUTERNAME + '-' + (Get-Date -Format 'HHmmss') + '-' + (Get-Random -Maximum 9999) $script:LogBuf = @() $script:LogLastFlush = Get-Date # LOCAL install-log file (operator request 2026-07-15): the streamed log is ALSO # written to disk so it can be referred back to until the install is confirmed # successful - streaming alone was "forgotten" once the Telegram scroll moved on. # Timestamped per run, readable later via SSH/atexec at C:\ProgramData\enroll-log\. $script:LocalLogDir = 'C:\ProgramData\enroll-log' $script:LocalLog = Join-Path $script:LocalLogDir ('boot-' + $script:RunId + '.log') # Init early so a top-level abort BEFORE the path decision (STEP 10) reports the # path honestly as '?' instead of misreporting a failed run as "PATH A - CLEAN". # (Fable review 2026-07-30, finding 10.) $script:PathTaken = '?' try { if (-not (Test-Path $script:LocalLogDir)) { New-Item -ItemType Directory -Path $script:LocalLogDir -Force | Out-Null } } catch { } # Prune old per-run logs (one file is written per run; under the boot-retry loop # this is otherwise unbounded). Keep the newest 30. (Fable review 2026-07-30, finding 8.) try { Get-ChildItem -Path $script:LocalLogDir -Filter 'boot-*.log' -ErrorAction SilentlyContinue | Sort-Object LastWriteTime -Descending | Select-Object -Skip 30 | Remove-Item -Force -ErrorAction SilentlyContinue } catch { } function Flush-RemoteLog { if ($script:LogBuf.Count -eq 0) { return } try { $payload = @{ run = $script:RunId; host = $env:COMPUTERNAME; lines = @($script:LogBuf) } | ConvertTo-Json Invoke-RestMethod -Uri $script:RemoteLogUrl -Method Post -Body $payload ` -ContentType 'application/json' -TimeoutSec 5 | Out-Null } catch { } # best-effort: swallow everything (offline, DNS, Worker down...) $script:LogBuf = @() $script:LogLastFlush = Get-Date } function Send-RemoteLog([string]$line) { try { $script:LogBuf += $line # Persist to the local install-log IMMEDIATELY (per-line, so the on-disk log is # complete even if the run aborts). Fire-and-forget; must never break the enrol. try { Add-Content -Path $script:LocalLog -Value ((Get-Date -Format 'HH:mm:ss') + ' ' + $line) -Encoding utf8 -ErrorAction SilentlyContinue } catch { } $age = ((Get-Date) - $script:LogLastFlush).TotalSeconds if (($age -ge 5) -or ($script:LogBuf.Count -ge 25)) { Flush-RemoteLog } } catch { } # never let logging break the run } # ---------------- output helpers (ASCII only; mirrored to the remote log) ---------------- function Banner([string]$t) { Write-Host '' Write-Host ('=' * 70) -ForegroundColor White Write-Host ('== ' + $t) -ForegroundColor White Write-Host ('=' * 70) -ForegroundColor White Send-RemoteLog ('== ' + $t) } function Info([string]$m) { Write-Host (' [..] ' + $m) -ForegroundColor Cyan; Send-RemoteLog ('[..] ' + $m) } function Ok ([string]$m) { Write-Host (' [OK] ' + $m) -ForegroundColor Green; Send-RemoteLog ('[OK] ' + $m) } function Warn2([string]$m){ Write-Host (' [!!] ' + $m) -ForegroundColor Yellow; Send-RemoteLog ('[!!] ' + $m) } function Bad ([string]$m) { Write-Host (' [XX] ' + $m) -ForegroundColor Red; Send-RemoteLog ('[XX] ' + $m) } $script:Results = [ordered]@{} function Set-Result([string]$name, [string]$status, [string]$detail) { $script:Results[$name] = [pscustomobject]@{ Status = $status; Detail = $detail } } # Run a native exe with stderr captured and EAP relaxed (PS 5.1 + EAP Stop + # stderr redirection is a known landmine). function Invoke-Native([string]$exe, [string[]]$argv) { $prev = $ErrorActionPreference $ErrorActionPreference = 'Continue' try { & $exe @argv 2>&1 } finally { $ErrorActionPreference = $prev } } function Test-Port22Listening { return [bool](Get-NetTCPConnection -LocalPort 22 -State Listen -ErrorAction SilentlyContinue) } # --------------------------------------------------------------------------- # SSHD FAILURE DIAGNOSTICS (added 2026-07-16 per operator's "closed-loop" model) # When sshd registers but will not START/LISTEN, the RESULT table alone is not # enough to know WHY (too-open perms vs unreadable-by-service-account vs bad # config). This gathers the full picture and ships it down the SAME /log path # as everything else (operator forwards it; also lands in the local enroll-log, # and in boot.mad.sg/lastlog once its KV binding is enabled). STRICTLY READ-ONLY # - no service starts, no ACL edits - so the data reflects the true failed state. # It exists so the maintaining session can adjust the FIX next retry cycle. # --------------------------------------------------------------------------- function Emit-SshdDiag([string]$context) { try { Send-RemoteLog ('== SSHD-DIAG BEGIN (' + $context + ') ==') try { Send-RemoteLog ('DIAG whoami: ' + ((whoami) 2>&1)) } catch { } $svc = Get-CimInstance Win32_Service -Filter "Name='sshd'" -ErrorAction SilentlyContinue if ($svc) { Send-RemoteLog ('DIAG sshd svc: StartName=' + $svc.StartName + ' State=' + $svc.State + ' StartMode=' + $svc.StartMode) Send-RemoteLog ('DIAG sshd PathName=' + $svc.PathName) } else { Send-RemoteLog 'DIAG sshd svc: NOT FOUND' } $ag = Get-CimInstance Win32_Service -Filter "Name='ssh-agent'" -ErrorAction SilentlyContinue if ($ag) { Send-RemoteLog ('DIAG ssh-agent svc: StartName=' + $ag.StartName + ' State=' + $ag.State) } # NOTE: PowerShell 5.1 does NOT allow try/catch as an EXPRESSION (e.g. # `(try {..} catch {..})`) - it parses `try` as a command and dies with # "'try' wurde nicht als ... erkannt". Keep all try/catch in STATEMENT # position (its own line), pre-computing values into a variable. (Bug that # truncated the first live dump, 2026-07-16.) $sshDataDiag = "$env:ProgramData\ssh" $folderOwner = 'err' try { $folderOwner = (Get-Acl $sshDataDiag).Owner } catch { } Send-RemoteLog ('DIAG folder ' + $sshDataDiag + ' owner=' + $folderOwner) Get-ChildItem "$sshDataDiag\ssh_host_*_key" -ErrorAction SilentlyContinue | ForEach-Object { $kf = $_.FullName; $kn = $_.Name try { $acl = Get-Acl $kf Send-RemoteLog ('DIAG KEY ' + $kn + ' owner=' + $acl.Owner) foreach ($ace in $acl.Access) { Send-RemoteLog ('DIAG ACE ' + $ace.IdentityReference + ' | ' + $ace.FileSystemRights + ' | ' + $ace.AccessControlType + ' | inherited=' + $ace.IsInherited) } } catch { Send-RemoteLog ('DIAG KEY ' + $kn + ' Get-Acl failed: ' + $_.Exception.Message) } # Raw icacls too (ground truth of the on-disk ACEs; catches a silent icacls no-op) $ic = Invoke-Native 'icacls.exe' @($kf) foreach ($l in @($ic)) { if ("$l".Trim()) { Send-RemoteLog ('DIAG ICACLS ' + ("$l").Trim()) } } } # Real sshd -t output from BOTH binaries (the exact rejection text) $sshdBins = [ordered]@{ 'builtin' = "$env:SystemRoot\System32\OpenSSH\sshd.exe" 'standalone' = "$env:ProgramFiles\OpenSSH\sshd.exe" } foreach ($tag in $sshdBins.Keys) { $exe = $sshdBins[$tag] if (Test-Path $exe) { $o = Invoke-Native $exe @('-t') Send-RemoteLog ('DIAG ' + $tag + ' sshd -t exit=' + $LASTEXITCODE) foreach ($l in @($o)) { if ("$l".Trim()) { Send-RemoteLog ('DIAG ' + ("$l").Trim()) } } } else { Send-RemoteLog ('DIAG ' + $tag + ' sshd.exe absent: ' + $exe) } } # OpenSSH operational event log (the real "no hostkeys available -- exiting" etc.) try { Get-WinEvent -LogName 'OpenSSH/Operational' -MaxEvents 12 -ErrorAction Stop | Sort-Object TimeCreated | ForEach-Object { Send-RemoteLog ('DIAG EVT ' + $_.TimeCreated.ToString('HH:mm:ss') + ' [' + $_.LevelDisplayName + '] ' + (($_.Message) -replace "`r?`n",' | ')) } } catch { Send-RemoteLog ('DIAG OpenSSH/Operational read failed: ' + $_.Exception.Message) } # Service Control Manager errors referencing sshd (7000/7009/7011/7031/7034) try { Get-WinEvent -FilterHashtable @{LogName='System'; ProviderName='Service Control Manager'} -MaxEvents 40 -ErrorAction Stop | Where-Object { $_.Message -match 'sshd|OpenSSH' } | Select-Object -First 6 | ForEach-Object { Send-RemoteLog ('DIAG SCM ' + $_.TimeCreated.ToString('HH:mm:ss') + ' id=' + $_.Id + ' ' + (($_.Message) -replace "`r?`n",' | ')) } } catch { } Send-RemoteLog '== SSHD-DIAG END ==' Flush-RemoteLog } catch { try { Send-RemoteLog ('SSHD-DIAG fatal: ' + $_.Exception.Message); Flush-RemoteLog } catch { } } } # --------------------------------------------------------------------------- # Set-SshAclExact - rebuild a DACL so ONLY SYSTEM + Administrators remain. # ROOT-CAUSE FIX (2026-07-16, BETTINA4AB9): the old icacls repair could NOT # remove a stray EXPLICIT ACE for a normal user (e.g. BETTINA4AB9\BHAGEDORN:(M), # left when ssh-keygen first ran in that user's logon context). `icacls # /inheritance:r` only strips INHERITED ACEs; `/grant:r` only rewrites the NAMED # principals - NEITHER deletes another user's explicit ACE. sshd then rejects the # host key ("Bad permissions ... this private key will be ignored" -> "sshd: no # hostkeys available -- exiting", service dies on start). This rebuilds the DACL # from scratch via .NET: protected (no inheritance), owner=Administrators, and # EXACTLY {SYSTEM:F, Administrators:F} - any stray user ACE is gone by construction. # --------------------------------------------------------------------------- $script:SidAdmins = New-Object System.Security.Principal.SecurityIdentifier('S-1-5-32-544') $script:SidSystem = New-Object System.Security.Principal.SecurityIdentifier('S-1-5-18') function Set-SshAclExact([string]$p, [bool]$container) { $sd = Get-Acl -LiteralPath $p $sd.SetAccessRuleProtection($true, $false) # protected: no inheritance, DROP inherited ACEs @($sd.Access) | ForEach-Object { [void]$sd.RemoveAccessRule($_) } # strip EVERY existing ACE (incl the stray user) $inh = if ($container) { 'ContainerInherit,ObjectInherit' } else { 'None' } $sd.AddAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule($script:SidSystem,'FullControl',$inh,'None','Allow'))) $sd.AddAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule($script:SidAdmins,'FullControl',$inh,'None','Allow'))) try { $sd.SetOwner($script:SidAdmins) } catch { } Set-Acl -LiteralPath $p -AclObject $sd } # Third-party AV probe (lesson 5). SecurityCenter2 + belt-and-braces McAfee # service/process probes (SecurityCenter2 can lag reality). DETECTION ONLY - # nothing in this script removes what this finds (report-only design). function Get-ThirdPartyAV { $found = @() try { $avProducts = Get-CimInstance -Namespace root/SecurityCenter2 -ClassName AntiVirusProduct -ErrorAction Stop foreach ($p in $avProducts) { if ($p.displayName -notmatch 'Windows Defender|Microsoft Defender') { $found += $p.displayName } } } catch { } $mcSvc = Get-Service -DisplayName '*McAfee*' -ErrorAction SilentlyContinue $mcProc = Get-Process -Name 'mcshield','MMSSHOST','ModuleCoreService','McCSPServiceHost','McAPExe' -ErrorAction SilentlyContinue # NOTE: -notmatch on an array returns the NON-matching ELEMENTS (an array), # not a boolean - so use -not (array -match ...) to test "no element matches". if (($mcSvc -or $mcProc) -and (-not ($found -match 'McAfee'))) { $found += 'McAfee (service/process detected)' } return @($found | Select-Object -Unique) } # ---------------------------------------------------------------------------- # AV ADVISOR (data source: AV-BLOCKERS.md in this repo dir - keep in sync). # Maps a detected SecurityCenter2 displayName to a per-vendor advice record: # Product / Risk (blocks-sshd-service) / Headless (YES|NO|CONDITIONAL) / # Action (exact manual step) / Lockout (removal needs Safe Mode, CAPTCHA, # GUI tamper-toggle or forced reboot = remote-lockout risk, warn LOUDLY). # POLICY (2026-07-11): EVERY vendor is ADVISE-ONLY. boot.ps1 never removes # any AV - removal is a separate, operator-directed step (Zoho remote-GUI or # the console). Removers' Safe-Mode / CAPTCHA / GUI-tamper / forced-reboot # failure modes can lock a remote box out, so a HUMAN drives them, always. # ---------------------------------------------------------------------------- function Get-AvAdvice([string]$displayName) { $rec = switch -Regex ($displayName) { 'McAfee' { @{ Product='McAfee (LiveSafe / Total Protection)' Risk='HIGH - CONFIRMED: kills the background sshd SERVICE (exit 1067); tolerates interactive sshd' Headless='NO (operator-directed only; bespoke standalone tools exist)'; Lockout=$true Action='NOT removed by this script (report-only). Operator-directed: uninstall via the Zoho remote-GUI once installed, or at the console (Settings > Apps > McAfee LiveSafe > Uninstall, or MCPR/McClnUI interactively). Standalone operator tools: remove-mcafee.ps1 / remove-mcafee-safemode.ps1 (same dir; never auto-run).' } break } 'Norton|Symantec' { @{ Product='Norton / Symantec (Norton 360 / Norton Security)' Risk='MEDIUM-HIGH - SONAR behavioural + tamper protection; aggressive on unsigned listening services' Headless='NO'; Lockout=$true Action='NRnR (Norton Remove & Reinstall) is GUI-ONLY and FORCES A RESTART - no silent switch exists. Human at console/RDP: run NRnR, Agree > Remove > Restart.' } break } 'Avast' { @{ Product='Avast (Free / Premium Security)' Risk='MEDIUM - Behavior Shield + Self-Defense (aswbIDSAgent); known to interfere with inbound SSH' Headless='CONDITIONAL'; Lockout=$true Action='instup.exe /instop:uninstall /silent works ONLY after SilentUninstallEnabled=1 in stats.ini AND Self-Defense disabled (a GUI step). avastclear.exe needs SAFE MODE (2 reboots) - NEVER trigger that remotely.' } break } 'AVG' { @{ Product='AVG (Free / Internet Security)' Risk='MEDIUM - same engine as Avast (Behavior Shield + self-defense)' Headless='CONDITIONAL'; Lockout=$true Action='Business avgremover.exe /silent (run as SYSTEM) works headless; consumer avgclear.exe needs SAFE MODE (2 reboots) - NEVER trigger that remotely.' } break } 'Bitdefender' { @{ Product='Bitdefender (Total Security / Antivirus Plus)' Risk='MEDIUM - Advanced Threat Defense + Network Attack Defense; documented to block inbound SSH' Headless='CONDITIONAL'; Lockout=$true Action='Consumer Bitdefender_Uninstall_Tool.exe is GUI + reboots. Business BEST uninstallTool.exe /bdparams is silent but password-gated + reboots. Human at console/RDP for consumer boxes.' } break } 'Kaspersky' { @{ Product='Kaspersky (Standard / Plus / Premium)' Risk='MEDIUM - System Watcher + HIPS/Application Control can restrict unknown always-on services' Headless='NO'; Lockout=$true Action='kavremover CANNOT run silent - it REQUIRES A CAPTCHA every run (automation impossible by design). Human at console/RDP runs kavremover interactively.' } break } 'Webroot' { @{ Product='Webroot SecureAnywhere' Risk='MEDIUM-HIGH - journals + can ROLL BACK actions of unknown executables (an unsigned sshd.exe is exactly that)' Headless='CONDITIONAL'; Lockout=$true Action='WRSA.exe -uninstall can work headlessly but self-protection may block the scripted stop, and a REBOOT finalises removal (+ registry residue). Prefer a human at console/RDP.' } break } 'ESET' { @{ Product='ESET (NOD32 / Internet Security / Smart Security)' Risk='MEDIUM - HIPS + self-defense; rule-based but can block' Headless='NO'; Lockout=$true Action='ESETUninstaller.exe /force REQUIRES SAFE MODE (or GUI self-defense disable + reboot first). Rebooting into Safe Mode over SSH = guaranteed lockout. Human at console only.' } break } 'Avira' { @{ Product='Avira (Free Security / Antivirus Pro)' Risk='LOW-MEDIUM - lighter behavioural footprint than McAfee/Webroot' Headless='CONDITIONAL'; Lockout=$false Action='Installer/Launcher silent switch /remsilentnoreboot is headless-ish (installer-dependent; self-protection may need toggling). Avira Registry Cleaner is GUI post-cleanup only.' } break } 'Trend Micro' { @{ Product='Trend Micro (Maximum / Internet Security)' Risk='MEDIUM - Behaviour Monitoring + self-protection (TmListen/Amsp)' Headless='NO'; Lockout=$true Action='Consumer removal is GUI-ONLY (Diagnostic Toolkit SupportTool.exe > Uninstall tab) + reboot. Human at console/RDP.' } break } 'Malwarebytes' { @{ Product='Malwarebytes (Premium, standalone)' Risk='LOW-MEDIUM - usually coexists; less likely to kill a listening sshd' Headless='CONDITIONAL'; Lockout=$false Action='mb-clean.exe /y /cleanup /noreboot /nopr /deepclean is headless IFF Tamper Protection is OFF (commonly off on consumer standalone). NOT auto-run - surface-only for now (see AV-BLOCKERS.md).' } break } 'Sophos' { @{ Product='Sophos Home (consumer)' Risk='MEDIUM-HIGH - Intercept X Lockdown DOES terminate processes it deems malicious; strong tamper protection' Headless='NO (in practice)'; Lockout=$true Action='SophosUninstall.exe --quiet / SophosZap.exe --confirm exist BUT require Tamper Protection OFF first - a GUI/cloud-dashboard step - then a reboot to finalise. Human required.' } break } default { @{ Product=$displayName Risk='UNKNOWN - not in AV-BLOCKERS.md; treat as hostile to a background sshd service until proven otherwise' Headless='NO (unknown vendor)'; Lockout=$true Action='Unknown vendor - assume GUI-only removal with a reboot. Human at console/RDP: Settings > Apps > uninstall it, then re-run boot.ps1. Add the vendor to AV-BLOCKERS.md.' } } } return [pscustomobject]$rec } Banner 'WINDOWS REMOTE-ACCESS ENROLLMENT (Tailscale + OpenSSH) - boot.mad.sg canonical script' Write-Host (' Host: ' + $env:COMPUTERNAME + ' User: ' + $env:USERNAME + ' Time: ' + (Get-Date -Format 'yyyy-MM-dd HH:mm:ss') + ' bootloader v' + $BOOT_VERSION) Send-RemoteLog ('Host: ' + $env:COMPUTERNAME + ' User: ' + $env:USERNAME + ' Time: ' + (Get-Date -Format 'yyyy-MM-dd HH:mm:ss') + ' run=' + $script:RunId + ' bootloader-version=' + $BOOT_VERSION) # ============================================================================ # 1. ADMIN CHECK (graceful under iex -- no ugly #Requires abort; self-elevates) # ============================================================================ Banner 'STEP 1/11: Administrator check' $isAdmin = ([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent() ).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator) if (-not $isAdmin) { Bad 'This script must run ELEVATED (as Administrator).' if ($PSCommandPath) { # File mode: relaunch this file elevated with the same parameters. Warn2 'Attempting to relaunch elevated (accept the UAC prompt)...' # Quote the path: Start-Process -ArgumentList joins the array with spaces and # does NOT quote, so an unquoted spaced path (e.g. C:\Users\John Smith\...) is # split by the child powershell and -File fails. (Fable review 2026-07-30.) $argList = @('-NoProfile','-ExecutionPolicy','Bypass','-File',('"' + $PSCommandPath + '"')) if ($AuthKey) { $argList += @('-AuthKey', $AuthKey) } if ($NodeName) { $argList += @('-NodeName', $NodeName) } if ($ExpectedTag) { $argList += @('-ExpectedTag', $ExpectedTag) } try { Start-Process powershell.exe -Verb RunAs -ArgumentList $argList Warn2 'Elevated copy launched in a new window. This window is done.' } catch { Bad 'Elevation was refused. Re-run from an ADMIN PowerShell:' Write-Host (' powershell -ExecutionPolicy Bypass -File "' + $PSCommandPath + '"') } } else { # iex mode: we know the canonical URL, so we can normally self-elevate by # re-fetching ourselves in an elevated shell - but ONLY when there is an # interactive desktop to show the UAC prompt. Over SSH (session 0) the UAC # prompt is invisible and the "elevated copy" never actually runs, so do # NOT pretend a relaunch worked. $curSession = (Get-Process -Id $PID).SessionId if ($curSession -lt 1) { Bad 'Headless (SSH / session 0) and NOT elevated: self-elevation is IMPOSSIBLE here.' Bad 'There is no desktop to show the UAC prompt, so an elevated relaunch cannot work.' Bad 'Fix: reconnect over SSH as a user in the Administrators group (the admin' Bad 'keystore administrators_authorized_keys grants a full admin token over SSH),' Bad 'or get to the console/RDP as an admin, then re-run:' Write-Host (' irm ' + $BOOT_URL.Replace('https://','') + ' | iex') -ForegroundColor Yellow } else { Warn2 ('Attempting to relaunch elevated via ' + $BOOT_URL + ' (accept the UAC prompt)...') try { Start-Process powershell.exe -Verb RunAs -ArgumentList @( '-NoProfile','-ExecutionPolicy','Bypass','-NoExit', '-Command', ('irm ' + $BOOT_URL + ' | iex')) Warn2 'Elevated copy launched in a new window. This window is done.' } catch { Bad 'Elevation was refused. Open an ADMIN PowerShell (right-click Start >' Bad '"Windows PowerShell (Admin)", or type "powershell" in Start and press' Bad 'Ctrl+Shift+Enter), then paste the SAME command:' Write-Host (' irm ' + $BOOT_URL.Replace('https://','') + ' | iex') -ForegroundColor Yellow } } } Flush-RemoteLog # early exit - do not lose the buffered progress lines return } Ok 'Running elevated.' # ---------------- constants ---------------- $ts = 'C:\Program Files\Tailscale\tailscale.exe' $osshDir = 'C:\Program Files\OpenSSH' $sshData = 'C:\ProgramData\ssh' $fwName = 'OpenSSH-SSH-In (Tailscale)' # Trusted SSH public keys (Mike's Macs) -- public material, safe to embed. $PUBKEYS = @( 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMVR9A1sze7qf7XURw/9Dz0BzZ7ED1cpJo8QPsU+Ds+P mtyldesley@Mike-MacBook-Pro-M3.local' ) # ---- TOP-LEVEL RESILIENCE WRAPPER (operator ask 2026-07-13): guarantee the RESULT # report + Flush-RemoteLog ALWAYS emit, even if a step throws an UNGUARDED terminating # error. Per-step try/catch already handles most failures + continues; this catches the # rare escape so the server/operator always get the full picture. (Enclosed code is # intentionally NOT re-indented - PowerShell is brace-scoped, not indent-scoped.) ---- try { # ============================================================================ # 2. STATE DETECTION: third-party AV (lesson 5) + which session we are in # (lesson 8: session id decides whether the interactive-sshd trick is # available). Do this EARLY and LOUDLY. # ============================================================================ Banner 'STEP 2/11: State detection - antivirus + session' $mySession = (Get-Process -Id $PID).SessionId if ($mySession -ge 1) { Ok ('Running in INTERACTIVE session ' + $mySession + ' (console/RDP) - the AV-tolerated detached-sshd trick is available if needed.') } else { Info 'Running in session 0 (SSH / service / scheduled task) - headless. Any sshd serving this connection is a LIFELINE and will not be touched.' } try { $avProducts = Get-CimInstance -Namespace root/SecurityCenter2 -ClassName AntiVirusProduct -ErrorAction Stop foreach ($p in $avProducts) { Info ('Registered AV product: ' + $p.displayName) } } catch { Warn2 ('Could not query SecurityCenter2 (' + $_.Exception.Message + ') - relying on service/process probes.') } $thirdPartyAV = Get-ThirdPartyAV $avPresent = ($thirdPartyAV.Count -gt 0) if ($avPresent) { # -------- AV ADVISOR: REPORT-ONLY. This script NEVER removes an AV. -------- Write-Host '' Write-Host ('*' * 70) -ForegroundColor Red Write-Host '**** THIRD-PARTY ANTIVIRUS DETECTED ****' -ForegroundColor Red Write-Host ('*' * 70) -ForegroundColor Red Bad ('Third-party AV detected: ' + ($thirdPartyAV -join ', ')) Warn2 'KNOWN PROBLEM CLASS: such an AV can let an INTERACTIVE sshd run but KILL' Warn2 'the background sshd SERVICE (McAfee: exit 1067, "terminated unexpectedly").' Warn2 'Self-heal tasks CANNOT fix that - the AV just re-kills sshd every restart.' Warn2 'This script takes PATH B: LIFELINE FIRST, then REPORT, then STOP.' foreach ($avName in $thirdPartyAV) { $adv = Get-AvAdvice $avName Write-Host '' Warn2 ('ADVISOR: ' + $adv.Product + ' (detected as: "' + $avName + '")') Info (' sshd-service risk : ' + $adv.Risk) Info (' Headless removal : ' + $adv.Headless) Info (' Removal path : ' + $adv.Action) if ($adv.Lockout) { Bad ' *** REMOTE-LOCKOUT RISK: removing this AV can involve Safe Mode /' Bad ' *** CAPTCHA / a GUI tamper-protection toggle / a reboot. DO NOT' Bad ' *** trigger its removal remotely-unattended - the operator must' Bad ' *** direct it (Zoho remote-GUI / console) with eyes on the box.' } } Write-Host '' Warn2 'THIS SCRIPT DOES NOT REMOVE ANY AV (report-only by design, 2026-07-11).' Warn2 'AV removal is a SEPARATE, operator-directed step - recommended: uninstall' Warn2 'via the Zoho remote-GUI once it is installed, or at the console. The' Warn2 'interactive-session sshd LIFELINE (PATH B) is kept UP so the operator' Warn2 'retains access. This box is NOT reboot-safe until the AV is removed and' Warn2 ('boot.ps1 is re-run: irm ' + $BOOT_URL.Replace('https://','') + ' | iex') Write-Host ('*' * 70) -ForegroundColor Red Write-Host '' Send-RemoteLog ('[!!] Third-party AV detected: ' + ($thirdPartyAV -join ', ') + ' - PATH B (lifeline + report only; removal is a separate operator-directed step)') Set-Result 'AV status' 'WARN' ('Third-party AV present: ' + ($thirdPartyAV -join ', ') + ' - PATH B (lifeline + report; NOT removed by design)') } else { Ok 'No third-party AV found (Windows Defender only) - taking PATH A (permanent service).' Set-Result 'AV status' 'PASS' 'Windows Defender only - PATH A' } # ============================================================================ # 3. TAILSCALE (install if needed, unattended registry, up/reuse) # ============================================================================ Banner 'STEP 3/11: Tailscale' $tsip = $null $tsUp = $false try { if (-not (Test-Path $ts)) { Info 'Tailscale not installed - downloading MSI...' $msi = Join-Path $env:TEMP 'tailscale.msi' Invoke-WebRequest 'https://pkgs.tailscale.com/stable/tailscale-setup-latest-amd64.msi' -OutFile $msi -UseBasicParsing -TimeoutSec 300 Info 'Installing Tailscale (quiet)...' $mp = Start-Process msiexec.exe -ArgumentList "/i `"$msi`" /quiet /norestart" -PassThru if (-not $mp.WaitForExit(600000)) { try { $mp.Kill() } catch {}; throw 'Tailscale msiexec did not finish within 10 min - killed (step FAILs, run continues)' } Start-Sleep 5 if (-not (Test-Path $ts)) { throw 'tailscale.exe missing after MSI install' } Ok 'Tailscale installed.' } else { Ok 'Tailscale already installed.' } # Unattended-boot registry (lesson 6): the CLI --unattended flag does NOT set # this on Windows; only this registry value makes tailscaled reconnect at boot # with nobody logged in. This is what the GUI "Run unattended" checkbox writes. $regPath = 'HKLM:\SOFTWARE\Tailscale IPN' $curMode = $null try { $curMode = (Get-ItemProperty -Path $regPath -Name 'UnattendedMode' -ErrorAction Stop).UnattendedMode } catch { } if ($curMode -ne 'always') { Info 'Setting UnattendedMode=always in HKLM\SOFTWARE\Tailscale IPN ...' New-Item -Path $regPath -Force | Out-Null Set-ItemProperty -Path $regPath -Name 'UnattendedMode' -Value 'always' # RESTART GATE: restarting tailscaled severs every live Tailscale # connection - possibly including the very SSH session driving this script # over the Tailscale IP. Only restart when Tailscale is NOT already up # (first-time enroll / logged-out): there the restart cannot cut us off # and IS needed - it releases a logged-in-user GUI lock on the daemon # (otherwise `tailscale up` from an elevated shell can hit 401 "Tailscale # already in use by ") and makes unattended mode apply immediately. $preIp = $null try { $preIp = ((Invoke-Native $ts @('ip','-4')) | Where-Object { $_ -match '^100\.' } | Select-Object -First 1) } catch { } if ($preIp) { Warn2 ('Tailscale is ALREADY UP (' + $preIp + ') - NOT restarting tailscaled: this session may be riding that IP and a restart would sever it.') Warn2 'Registry value is set; unattended mode takes effect at the next tailscaled restart / reboot.' Ok 'Unattended mode set (daemon restart deferred - Tailscale already up).' } else { Warn2 'RESTARTING tailscaled to apply unattended mode + release any GUI lock (Tailscale is not up, so no live tailnet session can be severed).' Stop-Process -Name 'tailscale-ipn' -Force -ErrorAction SilentlyContinue Restart-Service 'Tailscale' -Force -ErrorAction SilentlyContinue Start-Sleep 5 Ok 'Unattended mode set (tailscaled restarted to apply).' } } else { Ok 'Unattended mode already = always.' } # VERIFY it stuck $verMode = (Get-ItemProperty -Path $regPath -Name 'UnattendedMode' -ErrorAction SilentlyContinue).UnattendedMode if ($verMode -eq 'always') { Set-Result 'Unattended registry' 'PASS' 'HKLM\SOFTWARE\Tailscale IPN\UnattendedMode = always' } else { Set-Result 'Unattended registry' 'FAIL' ('UnattendedMode = "' + $verMode + '" (expected "always")') } # Already enrolled? (reuse case: irm|iex on a machine already on the tailnet). # If so, skip re-auth entirely - no auth key needed, no prompt. $ipOut = Invoke-Native $ts @('ip','-4') $tsip = ($ipOut | Where-Object { $_ -match '^100\.' } | Select-Object -First 1) if ($tsip) { $tsUp = $true Ok ('Tailscale is ALREADY UP with IP ' + $tsip + ' - skipping re-auth (no auth key needed).') } else { Info 'Tailscale is not up (not enrolled or logged out).' if (-not $AuthKey) { # iex path: no params possible. Prompt only when actually needed; never hard-fail. Warn2 'No -AuthKey was provided and this node is not enrolled yet.' Warn2 'A TAGGED pre-auth key (tag:bettina, tag:aunt, ...) is required - an untagged node is a security gap under the one-way ACL model.' # Only prompt when genuinely interactive: over `ssh host "irm ... | iex"` stdin is # redirected-but-empty, and a bare Read-Host would BLOCK THE WHOLE RUN forever. if ([Console]::IsInputRedirected) { Warn2 'Non-interactive session (input redirected) - skipping the auth-key prompt. Pass -AuthKey to enrol Tailscale from here.' $AuthKey = '' } else { try { $AuthKey = (Read-Host 'Paste Tailscale auth key (tskey-auth-...) or press Enter to skip Tailscale') } catch { $AuthKey = '' } } $AuthKey = ($AuthKey | ForEach-Object { $_.Trim() }) } if ($AuthKey) { Info ('Bringing Tailscale up as "' + $NodeName + '" (tagged via auth key)...') Invoke-Native $ts @('up','--authkey', $AuthKey, '--hostname', $NodeName, '--timeout=60s') | ForEach-Object { Write-Host (' ' + $_) } Start-Sleep 3 $ipOut = Invoke-Native $ts @('ip','-4') $tsip = ($ipOut | Where-Object { $_ -match '^100\.' } | Select-Object -First 1) if ($tsip) { $tsUp = $true; Ok ('Tailscale is UP: ' + $tsip) } else { Bad 'Tailscale did not come up after `tailscale up`. Run "tailscale status" to diagnose.' } } else { Bad 'Skipping Tailscale login (no key). SSH will only be allowed from the Tailscale range - i.e. NOT reachable until Tailscale is up.' } } if ($tsUp) { Set-Result 'Tailscale up + IP' 'PASS' $tsip # Tag check: nodes should carry a tag under the one-way ACL model. try { $stJson = (Invoke-Native $ts @('status','--json')) -join "`n" | ConvertFrom-Json $tags = @($stJson.Self.Tags) if ($ExpectedTag -and ($tags -contains $ExpectedTag)) { Ok ('Node is tagged as expected: ' + ($tags -join ', ')) } elseif ($ExpectedTag -and $tags.Count -gt 0) { Warn2 ('Node tags are ' + ($tags -join ', ') + ' - expected ' + $ExpectedTag + '. Check the ACL.') } elseif ($tags.Count -gt 0) { Ok ('Node is tagged: ' + ($tags -join ', ')) } else { Warn2 'Node is UNTAGGED - this is a security gap under the one-way ACL model. Re-enroll with a tagged pre-auth key.' } } catch { Warn2 ('Could not read node tags: ' + $_.Exception.Message) } } else { Set-Result 'Tailscale up + IP' 'FAIL' 'not up - provide a tagged auth key and re-run' } } catch { Bad ('Tailscale step failed: ' + $_.Exception.Message) if (-not $script:Results.Contains('Tailscale up + IP')) { Set-Result 'Tailscale up + IP' 'FAIL' $_.Exception.Message } if (-not $script:Results.Contains('Unattended registry')) { Set-Result 'Unattended registry' 'FAIL' 'not reached' } } # ============================================================================ # 4. OPENSSH INSTALL (standalone, Program Files) + ACL RESET (lessons 1+3) # ============================================================================ Banner 'STEP 4/11: OpenSSH (standalone) install + ACL repair' try { if (-not (Test-Path "$osshDir\sshd.exe")) { Info 'Downloading standalone OpenSSH-Win64...' $zip = Join-Path $env:TEMP 'OpenSSH-Win64.zip' Invoke-WebRequest 'https://github.com/PowerShell/Win32-OpenSSH/releases/latest/download/OpenSSH-Win64.zip' -OutFile $zip -UseBasicParsing -TimeoutSec 300 $tmp = Join-Path $env:TEMP 'OpenSSH-Win64' if (Test-Path $tmp) { Remove-Item $tmp -Recurse -Force } Expand-Archive $zip -DestinationPath $env:TEMP -Force Move-Item $tmp $osshDir -Force Ok 'OpenSSH files placed in C:\Program Files\OpenSSH.' } else { Ok 'OpenSSH already present in C:\Program Files\OpenSSH.' } # ALWAYS reset ACLs (lesson 3): Move-Item from %TEMP% drags user-profile ACLs # along, which breaks LocalSystem / NT SERVICE\EventLog reads (dead ETW event # provider -> sshd failures become invisible in Event Viewer). Info 'Resetting ACLs on C:\Program Files\OpenSSH (icacls /reset /t /c)...' Invoke-Native 'icacls.exe' @($osshDir, '/reset', '/t', '/c') | Out-Null $inh = (Get-Acl "$osshDir\sshd.exe").Access | Where-Object { $_.IsInherited } if ($inh) { Ok 'ACLs reset - files now inherit standard Program Files permissions.' Set-Result 'OpenSSH ACLs' 'PASS' 'Program Files inheritance restored (icacls /reset /t /c)' } else { Warn2 'sshd.exe still has no inherited ACEs after reset - check icacls output manually.' Set-Result 'OpenSSH ACLs' 'FAIL' 'no inherited ACEs on sshd.exe after /reset' } # Register the sshd service. FLAVOR-ADAPTIVE (B4 FIX 2026-07-16, supersedes B3). # B3 removed the Windows BUILT-IN OpenSSH.Server (sc stop/delete + capability # removal) and then registered the standalone. PROVEN FAILURE (BETTINA4AB9, # Parallels Win11 VM): that removal leaves the 'sshd' (and often 'ssh-agent') # service name MARKED FOR DELETION until reboot, so install-sshd.ps1's # New-Service dies with error 1072 BEFORE it ever creates sshd -> no service at # all, and every re-run repeats it identically. Worse, the installer ran with a # hidden window and its output DISCARDED, so the real error was invisible. # NEW POLICY: NEVER remove the built-in. Both flavors share C:\ProgramData\ssh # (sshd_config, host keys, administrators_authorized_keys) and the DefaultShell # registry value, so everything else in this script is flavor-agnostic: # 1. Existing healthy sshd service -> USE IT, whichever flavor it is. # 2. No sshd service -> pre-clear ssh-agent (a running built-in ssh-agent # keeps a deleted name pending forever), try the standalone installer with # OUTPUT CAPTURED + relayed, and on failure (incl. delete-pending 1072) # fall back to Add-WindowsCapability OpenSSH.Server -- CBS owns that # registration, so it can reclaim the very name SCM refuses to hand out. # Installer invoked via powershell.exe -ExecutionPolicy Bypass -File (lesson 1) # -- NEVER via `&`, which inherits a possibly-Restricted machine policy. $script:SshdFlavor = $null $existing = Get-Service sshd -ErrorAction SilentlyContinue if ($existing) { $img = (Get-CimInstance Win32_Service -Filter "Name='sshd'" -ErrorAction SilentlyContinue).PathName if ($img -and ($img -match 'System32')) { $script:SshdFlavor = 'builtin' Ok 'sshd service already registered (Windows BUILT-IN, System32) - KEEPING and using it (removing it is what caused the delete-pending limbo).' } else { $script:SshdFlavor = 'standalone' Ok ('sshd service already registered (' + $(if ($img -and ($img -match [regex]::Escape($osshDir))) { 'our standalone build' } else { 'ImagePath: ' + $img }) + ').') } try { $existing.Close(); $existing.Dispose() } catch { } # drop OUR handle so nothing we hold can pend a delete } if (-not $script:SshdFlavor) { # Pre-clear ssh-agent: install-sshd.ps1 deletes + recreates it, and a RUNNING # agent process keeps the deleted name pending (1072) -> the installer aborts # before it ever creates sshd. Deliberately does NOT touch sshd PROCESSES: # one may be the lifeline serving this very session. Stop-Service ssh-agent -Force -ErrorAction SilentlyContinue Get-Process ssh-agent -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue Info 'Registering the STANDALONE sshd service (install-sshd.ps1, output CAPTURED)...' $insOut = Join-Path $env:TEMP 'install-sshd.out.txt' $insErr = Join-Path $env:TEMP 'install-sshd.err.txt' Remove-Item $insOut, $insErr -Force -ErrorAction SilentlyContinue $p = Start-Process powershell.exe -PassThru -WindowStyle Hidden ` -RedirectStandardOutput $insOut -RedirectStandardError $insErr -ArgumentList @( '-NoProfile','-ExecutionPolicy','Bypass','-File', ('"' + "$osshDir\install-sshd.ps1" + '"')) if (-not $p.WaitForExit(120000)) { try { $p.Kill() } catch {}; Warn2 'install-sshd.ps1 did not finish within 2 min - killed.' } $insText = '' foreach ($f in @($insOut, $insErr)) { try { if (Test-Path $f) { $insText += ((Get-Content $f -Raw -ErrorAction SilentlyContinue) + "`n") } } catch { } } foreach ($ln in @($insText -split "`r?`n" | Where-Object { $_.Trim() } | Select-Object -First 12)) { Info ('install-sshd: ' + $ln) } try { if ($p.ExitCode -ne 0) { Warn2 ('install-sshd.ps1 exit code ' + $p.ExitCode) } } catch { } if (Get-Service sshd -ErrorAction SilentlyContinue) { $script:SshdFlavor = 'standalone' Ok 'Standalone sshd service registered.' } else { # 'marked for deletion' matched in EN + DE ("zum Loeschen markiert") via the # ASCII-safe tail 'schen markiert'; 1072 = ERROR_SERVICE_MARKED_FOR_DELETE. $pendHint = [bool]($insText -match '1072|marked for deletion|schen markiert') if ($pendHint) { Warn2 'Service name is DELETE-PENDING (error 1072) - standalone registration cannot win before a reboot. Falling back to the built-in capability, which CAN reclaim the name.' } else { Warn2 'Standalone registration failed (install-sshd output above) - falling back to the Windows BUILT-IN OpenSSH.Server capability.' } $aj = Start-Job -ScriptBlock { Add-WindowsCapability -Online -Name 'OpenSSH.Server~~~~0.0.1.0' } if (Wait-Job $aj -Timeout 600) { Receive-Job $aj -ErrorAction SilentlyContinue | Out-Null } else { Stop-Job $aj -ErrorAction SilentlyContinue; Warn2 'Add-WindowsCapability did not finish within 10 min (Windows servicing busy).' } Remove-Job $aj -Force -ErrorAction SilentlyContinue Start-Sleep 3 if (Get-Service sshd -ErrorAction SilentlyContinue) { $script:SshdFlavor = 'builtin' Ok 'BUILT-IN OpenSSH.Server capability registered a working sshd service (System32). Using it.' } elseif ($pendHint) { throw 'sshd service name is DELETE-PENDING and neither flavor could register it - a REBOOT clears it; the boot-retry task then converges on its own.' } else { throw ('sshd service could not be registered by EITHER flavor. install-sshd said: ' + ($insText.Trim() -replace '\s+', ' ')) } } } Set-Result 'sshd service' 'PASS' ('registered, flavor=' + $script:SshdFlavor) # Default SSH shell = PowerShell (OpenSSH defaults to cmd.exe; PowerShell makes # `;`-chained remote commands work like the other family machines). New-Item -Path 'HKLM:\SOFTWARE\OpenSSH' -Force | Out-Null Set-ItemProperty -Path 'HKLM:\SOFTWARE\OpenSSH' -Name 'DefaultShell' -Value 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe' Ok 'Default SSH shell set to PowerShell.' # sshd_config hardening (idempotent): the firewall scopes port 22 to the # tailnet, but password auth must STILL be off - key-only, defense in depth. # Assert PasswordAuthentication no + PubkeyAuthentication yes; only rewrite # the file when the effective values differ. Directives are inserted BEFORE # any Match block (a directive after Match is scoped to that block, and the # stock sshd_config_default ends with "Match Group administrators"). New-Item -ItemType Directory -Path $sshData -Force | Out-Null $sshdConfig = Join-Path $sshData 'sshd_config' if (-not (Test-Path $sshdConfig)) { $defCfg = Join-Path $osshDir 'sshd_config_default' if (Test-Path $defCfg) { Copy-Item $defCfg $sshdConfig -Force; Info 'sshd_config created from sshd_config_default.' } } if (Test-Path $sshdConfig) { $cfgLines = @(Get-Content $sshdConfig) $cfgDirty = $false foreach ($pair in @(@('PasswordAuthentication','no'), @('PubkeyAuthentication','yes'))) { $dName = $pair[0]; $dWant = $pair[1] # Effective value = FIRST active occurrence (sshd is first-match-wins); # a commented or absent directive falls back to the sshd default, which # for PasswordAuthentication is YES - so absence must be fixed too. $curVal = $null $curLine = $cfgLines | Where-Object { $_ -match ('^\s*' + $dName + '\s+\S') } | Select-Object -First 1 if ($curLine -and $curLine -match ('^\s*' + $dName + '\s+(\S+)')) { $curVal = $Matches[1] } if ($curVal -ne $dWant) { $cfgLines = @($cfgLines | Where-Object { $_ -notmatch ('^\s*' + $dName + '\s+') }) $newLine = ($dName + ' ' + $dWant) $matchIdx = -1 for ($li = 0; $li -lt $cfgLines.Count; $li++) { if ($cfgLines[$li] -match '^\s*Match\b') { $matchIdx = $li; break } } if ($matchIdx -eq 0) { $cfgLines = @($newLine) + $cfgLines } elseif ($matchIdx -gt 0) { $cfgLines = @($cfgLines[0..($matchIdx - 1)]) + $newLine + @($cfgLines[$matchIdx..($cfgLines.Count - 1)]) } else { $cfgLines += $newLine } $cfgDirty = $true Info ('sshd_config: setting ' + $dName + ' ' + $dWant + ' (was: ' + $(if ($curVal) { $curVal } else { 'unset/commented' }) + ').') } } if ($cfgDirty) { Set-Content -Path $sshdConfig -Value $cfgLines -Encoding ascii Ok 'sshd_config hardened (rewritten): PasswordAuthentication no + PubkeyAuthentication yes.' Warn2 'A RUNNING sshd keeps its old config until restarted - it applies at the next sshd (re)start. Deliberately NOT restarting here: this session may ride that sshd.' } else { Ok 'sshd_config already hardened: PasswordAuthentication no + PubkeyAuthentication yes.' } # VERIFY by re-reading the file (never trust the write blindly). $verLines = @(Get-Content $sshdConfig) $pwOk = [bool]($verLines | Where-Object { $_ -match '^\s*PasswordAuthentication\s+no\b' }) $pkOk = [bool]($verLines | Where-Object { $_ -match '^\s*PubkeyAuthentication\s+yes\b' }) if ($pwOk -and $pkOk) { Set-Result 'sshd_config hardening' 'PASS' 'PasswordAuthentication no + PubkeyAuthentication yes (key-only auth)' } else { Set-Result 'sshd_config hardening' 'FAIL' ('verify failed: PasswordAuthentication-no=' + $pwOk + ' PubkeyAuthentication-yes=' + $pkOk) } } else { Warn2 'No sshd_config and no sshd_config_default to seed one from - sshd will use compiled-in defaults (PasswordAuthentication YES).' Set-Result 'sshd_config hardening' 'FAIL' 'sshd_config missing and could not be seeded - password auth may be enabled' } } catch { Bad ('OpenSSH install failed: ' + $_.Exception.Message) if (-not $script:Results.Contains('OpenSSH ACLs')) { Set-Result 'OpenSSH ACLs' 'FAIL' $_.Exception.Message } if (-not $script:Results.Contains('sshd service')) { Set-Result 'sshd service' 'FAIL' $_.Exception.Message } if (-not $script:Results.Contains('sshd_config hardening')) { Set-Result 'sshd_config hardening' 'FAIL' 'not reached' } } # ============================================================================ # 5. HOST KEYS (lesson 2: "no hostkeys available" = sshd will not start) # ============================================================================ Banner 'STEP 5/11: SSH host keys' try { New-Item -ItemType Directory -Path $sshData -Force | Out-Null $hostKeys = @(Get-ChildItem "$sshData\ssh_host_*_key" -ErrorAction SilentlyContinue) if ($hostKeys.Count -eq 0) { Info 'No host keys found - generating (ssh-keygen -A)...' Invoke-Native "$osshDir\ssh-keygen.exe" @('-A') | Out-Null $hostKeys = @(Get-ChildItem "$sshData\ssh_host_*_key" -ErrorAction SilentlyContinue) } if ($hostKeys.Count -eq 0) { throw 'ssh-keygen -A produced no host keys - sshd CANNOT start without them' } Ok ('Host keys present: ' + (($hostKeys | ForEach-Object { $_.Name }) -join ', ')) # Lock down private keys. CRITICAL (root cause of a multi-day sshd-won't-start). # OpenSSH rejects a host key ("Permissions ... are too open ... will be ignored" -> # "sshd: no hostkeys available -- exiting", service fails to start) unless the file is # OWNED by SYSTEM/Administrators AND accessible ONLY by them. # A stray EXPLICIT ACE for a normal user (e.g. BETTINA4AB9\BHAGEDORN:(M), left when # ssh-keygen first ran in that user's logon context) is the real killer: `icacls # /inheritance:r` only strips INHERITED ACEs and `/grant:r` only rewrites the NAMED # principals, so that user ACE SURVIVES and sshd keeps rejecting the key (confirmed # 2026-07-16, BETTINA4AB9). Fix: rebuild the DACL from scratch (Set-SshAclExact) so # ONLY SYSTEM + Administrators remain - folder first (inheritable, so regenerated keys # are safe by construction), then each key. try { Set-SshAclExact $sshData $true foreach ($k in $hostKeys) { Set-SshAclExact $k.FullName $false } } catch { # Fallback if Set-Acl is unavailable/blocked: old icacls sequence, PLUS remove the # specific stray user that sshd -t names ("removing permissions for user: DOMAIN\USER"). Warn2 ('Set-Acl rebuild failed (' + $_.Exception.Message + ') - falling back to icacls + explicit stray-ACE removal.') Invoke-Native 'icacls.exe' @($sshData, '/setowner', '*S-1-5-32-544', '/C') | Out-Null Invoke-Native 'icacls.exe' @($sshData, '/inheritance:r') | Out-Null Invoke-Native 'icacls.exe' @($sshData, '/grant:r', '*S-1-5-18:(OI)(CI)(F)', '*S-1-5-32-544:(OI)(CI)(F)') | Out-Null foreach ($k in $hostKeys) { Invoke-Native 'icacls.exe' @($k.FullName, '/setowner', '*S-1-5-32-544', '/C') | Out-Null Invoke-Native 'icacls.exe' @($k.FullName, '/inheritance:r') | Out-Null Invoke-Native 'icacls.exe' @($k.FullName, '/grant:r', '*S-1-5-18:(F)', '*S-1-5-32-544:(F)') | Out-Null $probe = ((Invoke-Native "$osshDir\sshd.exe" @('-t')) -join "`n") if ($probe -match 'removing permissions for user:\s*([^\s(]+)') { Invoke-Native 'icacls.exe' @($k.FullName, '/remove:g', $matches[1]) | Out-Null } } } # Verify sshd actually ACCEPTS the keys now (catches a lingering perms problem HERE, not # at silent service-start time). Use Invoke-Native: sshd writes complaints to STDERR and # under $EAP='Stop' a raw '& ... 2>&1' can throw NativeCommandError on the failure path. $keyPermTest = ((Invoke-Native "$osshDir\sshd.exe" @('-t')) -join "`n") if ($keyPermTest -match 'too open|UNPROTECTED PRIVATE KEY|no hostkeys') { Warn2 'Host keys STILL flagged too-open after owner+ACL repair - sshd will NOT start; investigate ACEs on C:\ProgramData\ssh.' Set-Result 'Host keys' 'FAIL' ($hostKeys.Count.ToString() + ' key(s) present but sshd rejects them (permissions still too open)') Emit-SshdDiag 'step5-hostkeys-rejected-after-repair' } else { Ok 'Host private keys: owner=Administrators, ACL=SYSTEM+Administrators only, sshd accepts them.' Set-Result 'Host keys' 'PASS' ($hostKeys.Count.ToString() + ' key(s), owner+ACL locked, sshd -t accepts them') } } catch { Bad ('Host key step failed: ' + $_.Exception.Message) Set-Result 'Host keys' 'FAIL' $_.Exception.Message } # ============================================================================ # 6. Windows built-in OpenSSH Server feature -- KEEP POLICY (B4 FIX 2026-07-16). # We no longer remove the capability, EVER: Remove-WindowsCapability + # sc delete (B3 and earlier) put the 'sshd' service name into a MARKED-FOR- # DELETION limbo until reboot, in which NO flavor can register -> the # repeated "no service named sshd" failure on BETTINA4AB9. The built-in # shares C:\ProgramData\ssh with the standalone, so coexistence is harmless; # if a Windows update ever re-registers its service over the standalone's, # the box still has a WORKING sshd (same config/keys/firewall) and the # self-heal task keeps it alive. Only its wide-open firewall rule is still # dropped (ours is Tailscale-scoped). # ============================================================================ Banner 'STEP 6/11: Windows built-in OpenSSH Server feature (KEEP policy) + wide-open firewall rule' try { if ($script:SshdFlavor -eq 'builtin') { Ok 'Built-in OpenSSH.Server is the ACTIVE sshd flavor - keeping it (by design since B4).' } else { $cap = Get-WindowsCapability -Online -Name 'OpenSSH.Server~~~~0.0.1.0' -ErrorAction SilentlyContinue if ($cap -and $cap.State -eq 'Installed') { Warn2 'Built-in OpenSSH.Server capability installed alongside the standalone service - LEAVING it (removal risks the delete-pending limbo; coexistence is harmless).' } else { Ok 'Windows OpenSSH Server feature not installed - nothing to do.' } } $builtinRule = Get-NetFirewallRule -Name 'OpenSSH-Server-In-TCP' -ErrorAction SilentlyContinue if ($builtinRule) { Info 'Removing the built-in wide-open "OpenSSH SSH Server (sshd)" firewall rule...' $builtinRule | Remove-NetFirewallRule Ok 'Built-in firewall rule removed (our Tailscale-scoped rule replaces it).' } } catch { Warn2 ('Feature removal issue (non-fatal): ' + $_.Exception.Message) } # ============================================================================ # 7. AUTHORIZED KEYS (admin keystore, ACL locked or sshd refuses the file) # ============================================================================ Banner 'STEP 7/11: administrators_authorized_keys' try { $ak = Join-Path $sshData 'administrators_authorized_keys' ($PUBKEYS -join "`n") | Out-File -FilePath $ak -Encoding ascii -Force Invoke-Native 'icacls.exe' @($ak, '/inheritance:r') | Out-Null # /grant:r (REPLACE) not /grant (add): /inheritance:r only strips INHERITED ACEs; # a pre-existing EXPLICIT user ACE would survive /grant and sshd would refuse the file. Invoke-Native 'icacls.exe' @($ak, '/grant:r', '*S-1-5-18:F', '*S-1-5-32-544:F') | Out-Null # Verify by SID, NOT by name: on a non-English box Get-Acl renders e.g. # 'VORDEFINIERT\Administratoren', which a name -match 'Administrators' wrongly flags # as a stray ACE -> FALSE-FAILs the step every run (the German-box bug, 2026-07). $okSids = @('S-1-5-18','S-1-5-32-544') $stray = (Get-Acl $ak).Access | Where-Object { try { $okSids -notcontains $_.IdentityReference.Translate([Security.Principal.SecurityIdentifier]).Value } catch { $true } } if ($stray) { Warn2 ('authorized_keys has unexpected ACEs: ' + (($stray | ForEach-Object { $_.IdentityReference }) -join ', ')) Set-Result 'authorized_keys' 'FAIL' 'unexpected ACEs remain - sshd will refuse the file' } else { Ok ('authorized_keys written (' + $PUBKEYS.Count + ' key(s)), ACL = SYSTEM + Administrators only.') Set-Result 'authorized_keys' 'PASS' ($PUBKEYS.Count.ToString() + ' key(s), ACL locked to SYSTEM + Administrators') } } catch { Bad ('authorized_keys step failed: ' + $_.Exception.Message) Set-Result 'authorized_keys' 'FAIL' $_.Exception.Message } # ============================================================================ # 8. FIREWALL (lesson 4: -Profile Any so the Tailscale adapter's profile can # never scope the rule away; RemoteAddress restricted to the Tailscale CGNAT # range 100.64.0.0/10 so port 22 is NOT wide open) # ============================================================================ Banner 'STEP 8/11: Firewall rule (TCP 22, Profile Any, Tailscale range only)' try { Get-NetFirewallRule -DisplayName $fwName -ErrorAction SilentlyContinue | Remove-NetFirewallRule New-NetFirewallRule -DisplayName $fwName -Direction Inbound -Protocol TCP -LocalPort 22 ` -Action Allow -Profile Any -RemoteAddress '100.64.0.0/10' | Out-Null # SMB (445) for the AV-PROOF remote-management channel (impacket/admin creds over # LocalAccountTokenFilterPolicy=1). Windows' built-in File-Sharing rules only open 445 on # SOME network profiles, so on a box whose Tailscale adapter is 'Public' the channel would # silently not exist. Make it explicit + Tailscale-scoped so it's present on EVERY box. Get-NetFirewallRule -DisplayName 'MAD-SMB-In (Tailscale)' -ErrorAction SilentlyContinue | Remove-NetFirewallRule New-NetFirewallRule -DisplayName 'MAD-SMB-In (Tailscale)' -Direction Inbound -Protocol TCP -LocalPort 445 ` -Action Allow -Profile Any -RemoteAddress '100.64.0.0/10' | Out-Null # Belt-and-braces (Mike, 2026-11-04): allow ALL inbound from the Tailscale CGNAT range, # Profile Any, so turning the Windows Firewall ON can NEVER sever our tailnet access to this # box (SSH, RDP, or any future service), not just the two ports above. Safe, not "wide open": # the one-way tailnet ACL already restricts who can reach it (only tag:mike), so this is # "open to Mike over Tailscale only". When support is OFF the node is `tailscale down` and # this rule is moot; when ON, it keeps the firewall from scoping our access away. Get-NetFirewallRule -DisplayName 'MAD-Tailnet-In (all)' -ErrorAction SilentlyContinue | Remove-NetFirewallRule New-NetFirewallRule -DisplayName 'MAD-Tailnet-In (all)' -Direction Inbound ` -Action Allow -Profile Any -RemoteAddress '100.64.0.0/10' | Out-Null # VERIFY (do not trust creation blindly - the P558 rule "existed" but was profile-scoped away) $rule = Get-NetFirewallRule -DisplayName $fwName -ErrorAction Stop $addr = ($rule | Get-NetFirewallAddressFilter).RemoteAddress $port = ($rule | Get-NetFirewallPortFilter) # Tolerant RemoteAddress check: Windows may echo the CIDR back as the range # form '100.64.0.0-100.127.255.255', and the property can be an ARRAY - a # strict -eq '100.64.0.0/10' false-FAILs on both. Join + regex instead. $addrOk = (($addr -join ',') -match '100\.64\.0\.0') if ($rule.Enabled -eq 'True' -and $rule.Profile -eq 'Any' -and $addrOk -and $port.LocalPort -eq '22') { Ok ('Rule verified: Enabled=' + $rule.Enabled + ' Profile=' + $rule.Profile + ' RemoteAddress=' + ($addr -join ',') + ' LocalPort=' + $port.LocalPort) Set-Result 'Firewall rule' 'PASS' ('TCP 22 in, Profile=Any, RemoteAddress=' + ($addr -join ',')) } else { Bad ('Rule verification MISMATCH: Enabled=' + $rule.Enabled + ' Profile=' + $rule.Profile + ' RemoteAddress=' + ($addr -join ',') + ' LocalPort=' + $port.LocalPort) Set-Result 'Firewall rule' 'FAIL' ('unexpected rule state: Profile=' + $rule.Profile + ' RemoteAddress=' + ($addr -join ',')) } } catch { Bad ('Firewall step failed: ' + $_.Exception.Message) Set-Result 'Firewall rule' 'FAIL' $_.Exception.Message } # ============================================================================ # 9. PERSISTENCE PRE-STAGE: Automatic startup + crash-recovery + self-heal # tasks. Done on BOTH paths -- on PATH B this is deliberate pre-staging so # that the moment McAfee is removed, a re-run (or the next boot) flips # cleanly to the permanent reboot-proof service with zero extra work. # ============================================================================ Banner 'STEP 9/11: Persistence (Automatic + crash recovery + self-heal tasks)' try { Set-Service sshd -StartupType Automatic Ok 'sshd startup type = Automatic.' # Service crash-recovery actions (restart on failure). NOTE: useless against a # third-party AV that actively kills sshd - see STEP 2 warning - but harmless # to have staged, and exactly right once the AV is gone. Invoke-Native 'sc.exe' @('failure','sshd','reset=','86400','actions=','restart/5000/restart/10000/restart/30000') | Out-Null Ok 'sc failure actions set (auto-restart on crash).' # Self-heal script + tasks (boot + every 5 min), run as SYSTEM. # PORT-22 GUARD (new vs enroll.ps1): if something is ALREADY listening on 22 - # e.g. the detached interactive-session sshd that PATH B uses while a # third-party AV is present - the self-heal must NOT try to start the service: # it cannot bind anyway, and service restart churn destabilises live # connections (the AV kills per-connection workers). With the guard, the # self-heal becomes a no-op while the stopgap sshd holds the port, and does # its real job (start the service) the moment nothing is serving port 22. $shDir = 'C:\ProgramData\sshd-selfheal' New-Item -ItemType Directory -Path $shDir -Force | Out-Null $shScript = @' # sshd-selfheal -- ensure OpenSSH sshd exists, has host keys, is Automatic, running. # Installed by boot.ps1 (boot.mad.sg). Runs as SYSTEM at boot + every 5 minutes. $logDir = 'C:\ProgramData\sshd-selfheal' $log = Join-Path $logDir 'log.txt' $beat = Join-Path $logDir 'last-run.txt' if (-not (Test-Path $logDir)) { New-Item -ItemType Directory -Path $logDir -Force | Out-Null } $now = Get-Date -Format 'yyyy-MM-dd HH:mm:ss' $action = '' # GUARD: if port 22 is already served (e.g. a detached interactive-session sshd # kept alive because a third-party AV kills the service - see boot.ps1 PATH B), # leave everything alone. Starting the service would fail to bind and the churn # destabilises live connections. if (Get-NetTCPConnection -LocalPort 22 -State Listen -ErrorAction SilentlyContinue) { "$now status=port22-already-served action=none" | Out-File -Encoding utf8 $beat return } $svc = Get-Service -Name sshd -ErrorAction SilentlyContinue if (-not $svc) { $installer = 'C:\Program Files\OpenSSH\install-sshd.ps1' if (Test-Path $installer) { # Invoke via -ExecutionPolicy Bypass -File: never trust the machine policy. # Pre-clear ssh-agent: a running agent keeps a deleted service name # delete-pending (1072) and install-sshd.ps1 dies before creating sshd. Stop-Service ssh-agent -Force -ErrorAction SilentlyContinue Get-Process ssh-agent -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue Start-Process powershell.exe -Wait -WindowStyle Hidden -ArgumentList @('-NoProfile','-ExecutionPolicy','Bypass','-File',('"' + $installer + '"')) $action = 're-registered' } else { $action = 'MISSING-installer' } if (-not (Get-Service -Name sshd -ErrorAction SilentlyContinue)) { # Standalone re-register failed (e.g. name delete-pending): reclaim the # name via the built-in capability - same C:\ProgramData\ssh config, so # keys/config/authorized_keys all keep working (B4 flavor-adaptive fix). try { Add-WindowsCapability -Online -Name 'OpenSSH.Server~~~~0.0.1.0' | Out-Null; $action = 're-added-builtin' } catch { $action = 'FAILED-reregister' } } } # Host keys: sshd refuses to start without them ("no hostkeys available"). if (-not (Get-ChildItem 'C:\ProgramData\ssh\ssh_host_*_key' -ErrorAction SilentlyContinue)) { & 'C:\Program Files\OpenSSH\ssh-keygen.exe' -A 2>$null # CRITICAL: lock regenerated keys (owner=Administrators + SYSTEM/Admins only) or # OpenSSH rejects them as "too open" -> "no hostkeys available" and sshd stays dead # while this task falsely reports success (the multi-day root cause, on the # unattended path). SIDs are locale-safe. foreach ($hk in (Get-ChildItem 'C:\ProgramData\ssh\ssh_host_*_key' -ErrorAction SilentlyContinue)) { & icacls.exe $hk.FullName /setowner '*S-1-5-32-544' /C 2>$null | Out-Null & icacls.exe $hk.FullName /inheritance:r 2>$null | Out-Null & icacls.exe $hk.FullName /grant:r '*S-1-5-18:(F)' '*S-1-5-32-544:(F)' 2>$null | Out-Null } if (-not $action) { $action = 'regenerated-hostkeys' } } # ALWAYS ensure EXISTING host keys are not "too open": strip any stray user ACE # (e.g. a logon-context ssh-keygen leaves DOMAIN\user:(M)) that icacls /inheritance:r # + /grant:r cannot remove - rebuild the DACL to SYSTEM+Administrators only. This # repairs keys that already exist (the missing-keys branch above never fires then), # which was the multi-day BETTINA4AB9 failure. Idempotent; SID-based (locale-safe). $sA = New-Object System.Security.Principal.SecurityIdentifier('S-1-5-32-544') $sS = New-Object System.Security.Principal.SecurityIdentifier('S-1-5-18') foreach ($hk in (Get-ChildItem 'C:\ProgramData\ssh\ssh_host_*_key' -ErrorAction SilentlyContinue)) { try { $sd = Get-Acl -LiteralPath $hk.FullName $sd.SetAccessRuleProtection($true, $false) @($sd.Access) | ForEach-Object { [void]$sd.RemoveAccessRule($_) } $sd.AddAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule($sS,'FullControl','None','None','Allow'))) $sd.AddAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule($sA,'FullControl','None','None','Allow'))) try { $sd.SetOwner($sA) } catch {} Set-Acl -LiteralPath $hk.FullName -AclObject $sd } catch {} } Set-Service -Name sshd -StartupType Automatic -ErrorAction SilentlyContinue & sc.exe failure sshd reset= 86400 actions= restart/5000/restart/10000/restart/30000 | Out-Null $svc = Get-Service -Name sshd -ErrorAction SilentlyContinue if ($svc -and $svc.Status -ne 'Running') { Start-Service -Name sshd -ErrorAction SilentlyContinue if (-not $action) { $action = 'started' } } $status = (Get-Service -Name sshd -ErrorAction SilentlyContinue).Status "$now status=$status action=$action" | Out-File -Encoding utf8 $beat if ($action -ne '') { "$now action=$action status=$status" | Out-File -Append -Encoding utf8 $log } '@ Set-Content -Path (Join-Path $shDir 'sshd-selfheal.ps1') -Value $shScript -Encoding utf8 Invoke-Native 'schtasks.exe' @('/Create','/TN','sshd-selfheal','/TR','powershell.exe -NoProfile -ExecutionPolicy Bypass -File C:\ProgramData\sshd-selfheal\sshd-selfheal.ps1','/SC','ONSTART','/RU','SYSTEM','/RL','HIGHEST','/F','/DELAY','0000:30') | Out-Null Invoke-Native 'schtasks.exe' @('/Create','/TN','sshd-selfheal-5min','/TR','powershell.exe -NoProfile -ExecutionPolicy Bypass -File C:\ProgramData\sshd-selfheal\sshd-selfheal.ps1','/SC','MINUTE','/MO','5','/RU','SYSTEM','/RL','HIGHEST','/F') | Out-Null Ok 'Self-heal tasks installed (boot + every 5 min, SYSTEM, port-22 guard).' Set-Result 'Persistence staged' 'PASS' 'Automatic + sc failure actions + self-heal tasks (port-22 guarded)' } catch { Bad ('Persistence step failed: ' + $_.Exception.Message) Set-Result 'Persistence staged' 'FAIL' $_.Exception.Message } # ============================================================================ # 10. START THE SSH CHANNEL -- the path decision. REPORT-ONLY: this step # leaves the operator with WORKING ACCESS no matter what, and NOTHING # more. It never removes an AV, never reboots, never touches boot config. # PATH A (no third-party AV): permanent service, stability-watched. # PATH B (AV present, INTERACTIVE session >= 1): detached interactive- # session sshd the AV tolerates (the LIFELINE), then REPORT + STOP. # AV removal is PHASE 2 - a separate, operator-directed step (Zoho # remote-GUI once installed, or the console). The integrated # auto-removal that used to run here was REMOVED 2026-07-11 after it # broke a machine's reachability (see the header). # PATH B (AV present, SESSION 0 / headless): never stop/kill any sshd - # it may be serving this very connection. Report and stop. # ============================================================================ # ---- Enable remote management (so ONE command covers everything) ---- # Operator request 2026-07-12: a single `irm boot.mad.sg | iex` should leave the # box fully manageable. This box is WORKGROUP, so Windows hands remote logons a # FILTERED non-admin token and blocks all remote administration over SMB/RPC. # Setting LocalAccountTokenFilterPolicy=1 gives the local admin a full token, # so the operator can manage the box remotely (impacket/admin creds over the # already-open 445/135) EVEN WHEN sshd is down. Tailscale-scoped; no reboot. # SECURITY: this enables remote admin logon for local admins -> pair with a # strong admin password (hardening step; the default weak pw must be changed). try { $polPath = 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System' if (-not (Test-Path $polPath)) { New-Item -Path $polPath -Force | Out-Null } New-ItemProperty -Path $polPath -Name 'LocalAccountTokenFilterPolicy' -Value 1 -PropertyType DWord -Force | Out-Null Ok 'Remote management enabled (LocalAccountTokenFilterPolicy=1) - box manageable over SMB/RPC even if sshd is down.' Set-Result 'Remote mgmt' 'PASS' 'LocalAccountTokenFilterPolicy=1 (remote admin over SMB/RPC; harden admin pw)' } catch { Warn2 ('Could not set LocalAccountTokenFilterPolicy: ' + $_.Exception.Message) Set-Result 'Remote mgmt' 'FAIL' $_.Exception.Message } Banner 'STEP 10/11: Start SSH channel (path decision - report-only, no AV removal)' $script:ChannelOk = $false $script:ChannelDesc = 'none' if (-not $avPresent) { # ========================================================================== # PATH A -- permanent service (reboot-survivable) # ========================================================================== $script:PathTaken = 'A' Info 'PATH A: starting the permanent sshd service...' try { # IDEMPOTENCY: a previous PATH-B run may have left a detached interactive # `sshd -D` (the AV-era stopgap) holding port 22, which would make the # service start fail-to-bind and false-FAIL this step. PATH A only runs # when NO third-party AV is present, so that leftover is obsolete - stop # it and hand the port to the permanent service. $deferHandover = $false if ((Test-Port22Listening) -and ((Get-Service sshd).Status -ne 'Running')) { if ($mySession -lt 1) { # SESSION 0 (this run likely arrived OVER that lifeline sshd, e.g. remote re-run): # killing every sshd here would sever THIS connection and abort the script before it # reports (RESULT/KEY-ACTIONS/Zoho/log-flush all skipped). Hand over via a DETACHED # one-shot so this run finishes + reports first, THEN the service takes the port. Warn2 'Port 22 held by a lifeline sshd that may be serving THIS session - deferring takeover ~15s (detached) so this run finishes reporting. Reconnect after; the service takes over.' Start-Process powershell.exe -WindowStyle Hidden -ArgumentList @('-NoProfile','-Command', 'Start-Sleep 15; Get-Process sshd -EA SilentlyContinue | Stop-Process -Force -EA SilentlyContinue; Start-Sleep 2; Start-Service sshd -EA SilentlyContinue') | Out-Null Set-Result 'sshd channel' 'WARN' 'takeover DEFERRED (this session rides the lifeline) - permanent service starts in ~15s; reconnect' $script:ChannelOk = $true; $script:ChannelDesc = 'permanent sshd service (deferred takeover)' $deferHandover = $true } else { # Interactive session (not riding the lifeline sshd) - safe to swap now. Warn2 'Port 22 is held by a NON-SERVICE sshd (leftover PATH-B stopgap). Stopping it to hand the port to the permanent service.' Get-Process sshd -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue Start-Sleep 1 } } if (-not $deferHandover) { if ((Get-Service sshd).Status -ne 'Running') { Start-Service sshd -ErrorAction Stop } Start-Sleep 3 $st = (Get-Service sshd).Status if ($st -ne 'Running') { throw ('sshd is ' + $st + ' immediately after start') } Ok 'sshd service started.' # Stability watch: a quick "Running" right after Start-Service proves # nothing (lesson 5) - watch it briefly even on the clean path. $watch = 5 Info ('Watching sshd stability for ' + $watch + 's...') Start-Sleep $watch $st = (Get-Service sshd).Status if ($st -eq 'Running') { Ok ('sshd still Running after ' + $watch + 's.') Set-Result 'sshd channel' 'PASS' ('service Running (survived ' + $watch + 's stability watch)') $script:ChannelOk = $true $script:ChannelDesc = 'permanent sshd service' } else { Bad ('sshd DIED within ' + $watch + 's of starting (status: ' + $st + ').') Bad 'No third-party AV detected, so check: host keys, sshd_config, Event Viewer (OpenSSH/Operational).' Set-Result 'sshd channel' 'FAIL' ('service died within ' + $watch + 's; check host keys / sshd_config / event log') Emit-SshdDiag 'step10-service-died-within-watch' } } } catch { Bad ('sshd service start failed: ' + $_.Exception.Message) Set-Result 'sshd channel' 'FAIL' $_.Exception.Message Emit-SshdDiag 'step10-start-threw' } } else { # ========================================================================== # PATH B -- AV present: guarantee a channel the AV tolerates (lesson 8) # ========================================================================== $script:PathTaken = 'B' if ($mySession -ge 1) { # ======================================================================== # PATH B, INTERACTIVE (session >= 1): LIFELINE FIRST, then REPORT + STOP. # ======================================================================== # ---- ACCESS: detached interactive-session sshd lifeline ---- # The trick that restored access on 2026-07-10. A DETACHED, hidden sshd -D # in THIS session survives the console window being closed, and the AV does # not kill it. This is the operator's guaranteed channel while the AV is # still installed. Info ('PATH B (LIFELINE): launching a detached interactive-session sshd in session ' + $mySession + '...') $lifelineUp = $false try { # Free the port first: cleanly stop the (doomed) service so its sc-failure # actions do not fight us, then kill any stale sshd processes. Warn2 'Stopping the sshd service and killing stale sshd processes to free port 22 (any existing SSH sessions will drop).' Stop-Service sshd -Force -ErrorAction SilentlyContinue Get-Process sshd -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue Start-Sleep 2 $proc = Start-Process -FilePath "$osshDir\sshd.exe" -ArgumentList '-D' -WindowStyle Hidden -PassThru Start-Sleep 3 # Stability watch (the AV kills session-0 sshd within seconds; an # interactive-session one should sail through this). $watch = 10 Info ('Watching the detached sshd for ' + $watch + 's...') Start-Sleep $watch $alive = Get-Process -Id $proc.Id -ErrorAction SilentlyContinue if ($alive -and (Test-Port22Listening)) { Ok ('Detached sshd is up: PID ' + $proc.Id + ', session ' + $mySession + ', port 22 LISTENING.') Ok 'This sshd SURVIVES closing this console window. It does NOT survive reboot or logoff.' Set-Result 'sshd channel' 'PASS' ('interactive-session sshd (PID ' + $proc.Id + ', session ' + $mySession + ') - NOT reboot/logoff-proof') $script:ChannelOk = $true $script:ChannelDesc = 'interactive-session sshd' $lifelineUp = $true } elseif ($alive) { Bad ('Detached sshd (PID ' + $proc.Id + ') is running but port 22 is NOT listening - check sshd_config / host keys.') Set-Result 'sshd channel' 'FAIL' 'detached sshd alive but no listener on TCP 22' } else { Bad ('Detached sshd died within ' + $watch + 's - even the interactive trick failed. Check host keys, sshd_config, and whether the AV policy changed.') Set-Result 'sshd channel' 'FAIL' ('detached interactive sshd died within ' + $watch + 's') } } catch { Bad ('Detached sshd launch failed: ' + $_.Exception.Message) Set-Result 'sshd channel' 'FAIL' $_.Exception.Message } # ---- REPORT + STOP: AV removal is deliberately NOT done here ---- if ($lifelineUp) { Info 'PATH B (REPORT): lifeline is UP. This script does NOT remove the AV (report-only by design, 2026-07-11).' } else { Bad 'Lifeline sshd is NOT confirmed up - fix the channel (host keys / sshd_config) and re-run from the console. The AV was NOT touched.' } Warn2 ('AV present: ' + ($thirdPartyAV -join ', ') + ' - see the STEP 2 advisor for per-vendor risk + removal path.') Warn2 'AV removal is a SEPARATE, operator-directed step - recommended: uninstall' Warn2 'via the Zoho remote-GUI once it is installed, or at the console. This box' Warn2 ('is NOT reboot-safe until the AV is removed and boot.ps1 is re-run: irm ' + $BOOT_URL.Replace('https://','') + ' | iex') Set-Result 'AV removal' 'WARN' ('not attempted BY DESIGN (report-only boot loader) - operator-directed step (Zoho remote-GUI or console), then re-run boot.ps1; AV: ' + ($thirdPartyAV -join ', ')) } else { # Session 0 (we are probably running OVER SSH). Whatever sshd is serving # this connection is the LIFELINE - killing "stale" sshd here would sever # our own access (that mistake caused a temporary lockout on 2026-07-10). # Report-only here as everywhere: this script never removes an AV. Info 'PATH B (headless, session 0): report-only - this script does NOT remove the AV.' Info ('AV present: ' + ($thirdPartyAV -join ', ') + ' - removal is a SEPARATE, operator-directed step: uninstall via the Zoho remote-GUI once it is installed, or at the console (see the STEP 2 advisor).') Set-Result 'AV removal' 'WARN' ('not attempted BY DESIGN (report-only boot loader) - operator-directed step (Zoho remote-GUI or console), then re-run boot.ps1; AV: ' + ($thirdPartyAV -join ', ')) Info 'PATH B (headless, session 0): will NOT stop/kill any sshd - it may be serving this very connection.' if (Test-Port22Listening) { $svcState = (Get-Service sshd -ErrorAction SilentlyContinue).Status if ($svcState -eq 'Running') { # The service itself is serving - with the AV present it may die at any # moment. Watch it and report honestly (lesson 5's 25s watch). Warn2 'Port 22 is served by the sshd SERVICE - the AV may kill it. Watching 25s...' Start-Sleep 25 if ((Get-Service sshd -ErrorAction SilentlyContinue).Status -eq 'Running') { Warn2 'Service still up after 25s, but with the AV present treat it as UNSTABLE.' Set-Result 'sshd channel' 'WARN' 'sshd SERVICE serving port 22 despite AV - unstable, expect kills; get someone to the console' $script:ChannelOk = $true $script:ChannelDesc = 'sshd service (unstable under AV)' } else { Bad 'The AV killed the sshd service during the watch (the known kill).' Set-Result 'sshd channel' 'FAIL' 'service killed by AV during stability watch; a human at the console must run boot.ps1' } } else { Ok 'Port 22 is listening via a non-service sshd (almost certainly an interactive-session lifeline) - leaving it untouched.' Set-Result 'sshd channel' 'PASS' 'existing interactive-session sshd serving port 22 (left untouched) - NOT reboot/logoff-proof' $script:ChannelOk = $true $script:ChannelDesc = 'existing interactive-session sshd' } } else { # No listener and no interactive session to launch one in: best-effort # service start (a flapping port beats a dead one), report honestly. Warn2 'No listener on port 22 and no interactive session available. Best-effort service start (the AV will likely kill it)...' try { Start-Service sshd -ErrorAction Stop } catch { Bad ('Service start failed: ' + $_.Exception.Message) } Start-Sleep 25 if ((Get-Service sshd -ErrorAction SilentlyContinue).Status -eq 'Running' -and (Test-Port22Listening)) { Warn2 'Service is up for now - treat as UNSTABLE while the AV is present.' Set-Result 'sshd channel' 'WARN' 'service running under AV - unstable; get a human to the console to run boot.ps1' $script:ChannelOk = $true $script:ChannelDesc = 'sshd service (unstable under AV)' } else { Bad 'NO WORKING CHANNEL from here. A human must open an ADMIN console on the box and run:' Write-Host (' irm ' + $BOOT_URL.Replace('https://','') + ' | iex') -ForegroundColor Yellow Bad '(that interactive run launches the AV-tolerated detached sshd automatically)' Set-Result 'sshd channel' 'FAIL' 'no channel possible headlessly under the AV - needs a human at the console to re-run boot.ps1' } } } } # Port 22 final check (both paths) if (Test-Port22Listening) { $lsn = Get-NetTCPConnection -LocalPort 22 -State Listen -ErrorAction SilentlyContinue | Select-Object -First 1 Ok 'Port 22 is LISTENING.' Set-Result 'Port 22 listening' 'PASS' ($lsn.LocalAddress + ':22') } else { Bad 'Nothing is listening on port 22.' Set-Result 'Port 22 listening' 'FAIL' 'no listener on TCP 22' } # ============================================================================ # 11. REBOOT-SURVIVAL VERIFICATION (+ the REBOOT SAFETY GATE, lesson 9) # ============================================================================ Banner 'STEP 11/11: Reboot-survival verification' $survival = $true function Test-Survival([string]$label, [bool]$ok, [string]$detail) { if ($ok) { Ok ($label + ' - ' + $detail) } else { Bad ($label + ' - ' + $detail); $script:survival = $false } } $svcObj = Get-Service sshd -ErrorAction SilentlyContinue Test-Survival 'sshd startup type' ($svcObj -and $svcObj.StartType -eq 'Automatic') $(if ($svcObj) { [string]$svcObj.StartType } else { 'service missing' }) $t1 = Invoke-Native 'schtasks.exe' @('/Query','/TN','sshd-selfheal'); $t1ok = ($LASTEXITCODE -eq 0) $t2 = Invoke-Native 'schtasks.exe' @('/Query','/TN','sshd-selfheal-5min'); $t2ok = ($LASTEXITCODE -eq 0) Test-Survival 'Self-heal task (boot)' $t1ok $(if ($t1ok) { 'present' } else { 'MISSING' }) Test-Survival 'Self-heal task (5min)' $t2ok $(if ($t2ok) { 'present' } else { 'MISSING' }) $um = (Get-ItemProperty -Path 'HKLM:\SOFTWARE\Tailscale IPN' -Name 'UnattendedMode' -ErrorAction SilentlyContinue).UnattendedMode Test-Survival 'Tailscale unattended' ($um -eq 'always') ('UnattendedMode=' + $um) $fwOk = [bool](Get-NetFirewallRule -DisplayName $fwName -ErrorAction SilentlyContinue | Where-Object { $_.Enabled -eq 'True' }) Test-Survival 'Firewall rule persists' $fwOk $(if ($fwOk) { 'present + enabled' } else { 'MISSING or disabled' }) $tsSvc = Get-Service 'Tailscale' -ErrorAction SilentlyContinue Test-Survival 'Tailscale service' ($tsSvc -and $tsSvc.StartType -eq 'Automatic') $(if ($tsSvc) { [string]$tsSvc.StartType } else { 'service missing' }) Write-Host '' if ($avPresent) { # REBOOT SAFETY GATE (lesson 9): with the AV installed, the box is NEVER # reboot-safe, regardless of how green the checks above look. A reboot with # nobody logged in leaves only the session-0 service - which the AV kills - # and the machine is locked out until a human reaches the console. Write-Host ' REBOOT-SURVIVAL: BLOCKED-BY-AV' -ForegroundColor Red Write-Host '' Write-Host ('*' * 70) -ForegroundColor Red Write-Host '**** REBOOT SAFETY GATE ****' -ForegroundColor Red Write-Host ('*' * 70) -ForegroundColor Red Write-Host '**** DO NOT REBOOT THIS MACHINE while the third-party AV is' -ForegroundColor Red Write-Host '**** installed. After a reboot with nobody logged in, only the' -ForegroundColor Red Write-Host '**** session-0 sshd service exists - the AV kills it - and the' -ForegroundColor Red Write-Host '**** box is LOCKED OUT until a human reaches the console.' -ForegroundColor Red Write-Host '****' -ForegroundColor Red Write-Host '**** Before ANY reboot: have the OPERATOR remove the AV (Zoho' -ForegroundColor Yellow Write-Host '**** remote-GUI uninstall once Zoho is installed, or Settings >' -ForegroundColor Yellow Write-Host '**** Apps at the console - this script never removes it), re-run' -ForegroundColor Yellow Write-Host ('**** irm ' + $BOOT_URL.Replace('https://','') + ' | iex, and confirm REBOOT-SURVIVAL: OK.') -ForegroundColor Yellow Write-Host '****' -ForegroundColor Red Write-Host '**** FALLBACK if nobody can remove the AV soon: configure Windows' -ForegroundColor Yellow Write-Host '**** AUTO-LOGIN for the admin account + a logon task running' -ForegroundColor Yellow Write-Host '**** "sshd -D" - the interactive-session sshd then survives' -ForegroundColor Yellow Write-Host '**** reboots even with the AV. Tradeoff: the desktop auto-unlocks' -ForegroundColor Yellow Write-Host '**** at boot (security downgrade), and it needs the account' -ForegroundColor Yellow Write-Host '**** password - so it is NOT configured automatically here.' -ForegroundColor Yellow Write-Host ('*' * 70) -ForegroundColor Red Set-Result 'Reboot safety' 'FAIL' 'BLOCKED-BY-AV: AV present (NOT removed by design - report-only loader); operator-directed removal (Zoho remote-GUI or console), then re-run and confirm REBOOT-SURVIVAL: OK' } elseif ($survival) { Write-Host ' REBOOT-SURVIVAL: OK' -ForegroundColor Green Set-Result 'Reboot safety' 'PASS' 'reboot-survivable (service Automatic + self-heal + unattended Tailscale)' } else { Write-Host ' REBOOT-SURVIVAL: NOT-OK (fix the [XX] items above)' -ForegroundColor Red Set-Result 'Reboot safety' 'FAIL' 'one or more persistence checks failed - see STEP 11' } } catch { # An UNGUARDED terminating error escaped a step above. Record it; the finally below # still emits the full RESULT report + flush, so a failure is never silent. Bad ('UNCAUGHT top-level error: ' + $_.Exception.Message) try { Set-Result 'Run integrity' 'FAIL' ('uncaught: ' + $_.Exception.Message) } catch {} } finally { # ============================================================================ # RESULT (inside the finally -> ALWAYS emits, even after an uncaught abort above) # ============================================================================ Banner 'RESULT' if ($script:PathTaken -eq 'B') { Write-Host ' PATH TAKEN: B - THIRD-PARTY AV PRESENT (lifeline + report only; removal deferred to the operator)' -ForegroundColor Yellow Send-RemoteLog 'PATH TAKEN: B - THIRD-PARTY AV PRESENT (lifeline + report only; removal deferred to the operator)' } elseif ($script:PathTaken -eq 'A') { Write-Host ' PATH TAKEN: A - CLEAN (permanent sshd service, reboot-survivable)' -ForegroundColor Green Send-RemoteLog 'PATH TAKEN: A - CLEAN (permanent sshd service, reboot-survivable)' } else { # PathTaken never advanced past its '?' init -> the run aborted before the # STEP 10 path decision. Report honestly, not as a clean PATH A. (finding 10.) Write-Host ' PATH TAKEN: ? - run ended BEFORE the path decision (aborted early - see FAIL rows below)' -ForegroundColor Red Send-RemoteLog 'PATH TAKEN: ? - run ended BEFORE the path decision (aborted early - see FAIL rows)' } Write-Host '' $anyFail = $false foreach ($k in $script:Results.Keys) { $r = $script:Results[$k] if ($r.Status -eq 'FAIL') { $anyFail = $true } $color = switch ($r.Status) { 'PASS' { 'Green' } 'WARN' { 'Yellow' } default { 'Red' } } Write-Host (' {0,-22} {1,-5} {2}' -f $k, $r.Status, $r.Detail) -ForegroundColor $color Send-RemoteLog ('{0,-22} {1,-5} {2}' -f $k, $r.Status, $r.Detail) } Write-Host '' if ($tsip) { if ($script:ChannelOk) { if ($script:PathTaken -eq 'B') { Write-Host (' ACCESS NOW: SSH UP via ' + $script:ChannelDesc + ' (ssh ' + $env:USERNAME + '@' + $tsip + ').') -ForegroundColor Yellow Write-Host ' AV PRESENT - NOT removed, BY DESIGN (this boot loader is report-only).' -ForegroundColor Yellow Write-Host ' AV removal is a SEPARATE, operator-directed step: uninstall via the' -ForegroundColor Yellow Write-Host ' Zoho remote-GUI once it is installed, or at the console (see the' -ForegroundColor Yellow Write-Host ' STEP 2 advisor). NOT reboot-safe until the AV is removed; then re-run:' -ForegroundColor Yellow Write-Host (' irm ' + $BOOT_URL.Replace('https://','') + ' | iex') -ForegroundColor Yellow Send-RemoteLog ('ACCESS NOW: SSH UP via ' + $script:ChannelDesc + ' (ssh ' + $env:USERNAME + '@' + $tsip + ') - AV present (not removed by design); removal is a separate operator-directed step; NOT reboot-safe until the AV is removed') } else { Write-Host (' ACCESS NOW: working via ' + $script:ChannelDesc + '.') -ForegroundColor Green Write-Host (' CONNECT WITH: ssh ' + $env:USERNAME + '@' + $tsip) -ForegroundColor White Send-RemoteLog ('ACCESS NOW: working via ' + $script:ChannelDesc + '. CONNECT WITH: ssh ' + $env:USERNAME + '@' + $tsip) } Write-Host ' (any local admin account name works; auth is via the embedded public key)' } else { Write-Host ' NO WORKING SSH CHANNEL - see the sshd channel line above for what a human must do.' -ForegroundColor Red Write-Host (' (Tailscale itself is up at ' + $tsip + ' - the tailnet path is fine, only sshd is down.)') -ForegroundColor Red Send-RemoteLog ('[XX] NO WORKING SSH CHANNEL (Tailscale up at ' + $tsip + ' - only sshd is down; see the sshd channel result line)') } } else { Write-Host ' NO TAILSCALE IP - the machine is NOT remotely reachable yet.' -ForegroundColor Red Write-Host ' Get a tagged pre-auth key and re-run with -AuthKey (or paste it at the prompt).' -ForegroundColor Red Send-RemoteLog '[XX] NO TAILSCALE IP - machine NOT remotely reachable yet; needs a tagged pre-auth key + re-run' } Write-Host '' if ($anyFail) { Write-Host ('*' * 70) -ForegroundColor Red Write-Host '**** ONE OR MORE COMPONENTS FAILED - see the RED lines above. ****' -ForegroundColor Red Write-Host ('*' * 70) -ForegroundColor Red Send-RemoteLog '[XX] ONE OR MORE COMPONENTS FAILED - see the result table above.' } else { Write-Host ' ALL COMPONENTS PASS.' -ForegroundColor Green Send-RemoteLog '[OK] ALL COMPONENTS PASS.' } Write-Host '' # ---------------------------------------------------------------------------- # CONCISE END-OF-RUN SUMMARY -> mad-messages (human-facing; the full detail # already streamed to the mad1 log). Verdict + failed rows + a LOUD flag if a # 3rd-party AV needs uninstalling (the one thing that always needs a human). # Best-effort, never blocks. (Added 2026-07-16 per the feedback-loop redesign.) # ---------------------------------------------------------------------------- try { $sum = @() if ($anyFail) { $sum += ('[XX] ' + $env:COMPUTERNAME + ' enrollment FAILED') } else { $sum += ('[OK] ' + $env:COMPUTERNAME + ' enrollment ALL PASS') } $sum += ('run=' + $script:RunId + ' bootloader v' + $BOOT_VERSION) $failRows = @($script:Results.Keys | Where-Object { $script:Results[$_].Status -eq 'FAIL' }) $warnRows = @($script:Results.Keys | Where-Object { $script:Results[$_].Status -eq 'WARN' }) if ($failRows.Count) { $sum += ('FAILED: ' + ($failRows -join ', ')) } else { $sum += 'all components pass' } if ($warnRows.Count) { $sum += ('warnings: ' + ($warnRows -join ', ')) } if ($avPresent) { $sum += ('!!! 3RD-PARTY AV NEEDS UNINSTALL: ' + ($thirdPartyAV -join ', ') + ' - sshd is not reliable until it is removed (operator action needed).') } if ($tsip -and $script:ChannelOk) { $sum += ('SSH up: ssh ' + $env:USERNAME + '@' + $tsip) } elseif ($tsip) { $sum += ('SSH DOWN - only sshd is down (Tailscale up at ' + $tsip + ')') } else { $sum += 'NO Tailscale IP - machine not reachable' } $sum += 'full log: mad1:/srv/boot-logs/boot.log' # State-fingerprint dedupe (Fable review 2026-07-30, finding 4): under the # boot-retry loop this summary would otherwise POST every tick (spamming the # operator's phone). POST only when the MEANINGFUL state changes - verdict, # path, channel, failed rows, AV presence - or on the very first run. The full # per-run detail still streams to mad1:/srv/boot-logs unconditionally; only the # human-facing summary is deduped. RunId is deliberately NOT in the fingerprint. $fp = (@( $(if ($anyFail) { 'FAIL' } else { 'PASS' }), ('path=' + $script:PathTaken), ('chan=' + [string]$script:ChannelOk), ('av=' + [string][bool]$avPresent), ('rows=' + (($failRows | Sort-Object) -join ',')) ) -join '|') $fpFile = Join-Path $script:LocalLogDir 'last-summary-state.txt' $prevFp = '' try { if (Test-Path $fpFile) { $prevFp = (Get-Content $fpFile -Raw -ErrorAction SilentlyContinue).Trim() } } catch { } if ($fp -ne $prevFp) { $sBody = @{ host = $env:COMPUTERNAME; text = ($sum -join "`n") } | ConvertTo-Json Invoke-RestMethod -Uri $script:SummaryUrl -Method Post -Body $sBody -ContentType 'application/json' -TimeoutSec 6 | Out-Null try { Set-Content -Path $fpFile -Value $fp -Encoding ascii -Force } catch { } } } catch { } # ============================================================================ # POST-RESULT: ZOHO ASSIST SECONDARY CHANNEL -- DECOUPLED SPIN-OFF # ---------------------------------------------------------------------------- # Runs AFTER the SSH RESULT above is fully reported, per the decoupling design # (hetzner/docs/mad-approve-design.md + windows-enroll/ZOHO-SECONDARY.md): # SSH done = done. This section only REGISTERS a separate hourly SYSTEM task # that runs install-zoho.ps1 - Zoho waiting on an approval/token, failing, or # needing a GUI can NEVER block or delay the SSH channel, and any failure here # is deliberately non-fatal (try/catch, warnings only). # # TOKEN STAYS OUT OF THIS PUBLIC SCRIPT. install-zoho.ps1 (also public, also # token-free) fetches the tokenised installer at runtime from a PRIVATE source: # C:\ProgramData\zoho-secondary\config.json / drop\ files placed over SSH, or # the future MAD Approve broker endpoint (stubbed in Get-ApprovalAndToken). # The task self-deletes once ZohoURSService is verified Running. # ============================================================================ Banner 'POST-RESULT: Zoho secondary channel (decoupled spin-off - never blocks SSH)' try { $zohoInstalled = (Test-Path 'HKLM:\SOFTWARE\Zoho Assist\Unattended Remote Support') -or (Test-Path 'HKLM:\SOFTWARE\WOW6432Node\Zoho Assist\Unattended Remote Support') -or [bool](Get-Service -Name 'ZohoURSService' -ErrorAction SilentlyContinue) if ($zohoInstalled) { Ok 'Zoho Assist Unattended agent already installed - no spin-off needed (5-device free cap respected).' } else { $zohoDir = 'C:\ProgramData\zoho-secondary' $zohoScript = Join-Path $zohoDir 'install-zoho.ps1' $zohoTask = 'zoho-secondary-install' New-Item -ItemType Directory -Path $zohoDir -Force | Out-Null Info ('Fetching install-zoho.ps1 from ' + $BOOT_URL + '/install-zoho.ps1 ...') Invoke-WebRequest -Uri ($BOOT_URL + '/install-zoho.ps1') -OutFile $zohoScript -UseBasicParsing -TimeoutSec 60 if (-not (Test-Path $zohoScript)) { throw 'install-zoho.ps1 missing after download' } $zohoTr = 'powershell.exe -NoProfile -ExecutionPolicy Bypass -File "' + $zohoScript + '"' Invoke-Native 'schtasks.exe' @('/Create','/TN',$zohoTask,'/TR',$zohoTr,'/SC','HOURLY','/RU','SYSTEM','/RL','HIGHEST','/F') | Out-Null Invoke-Native 'schtasks.exe' @('/Query','/TN',$zohoTask) | Out-Null if ($LASTEXITCODE -eq 0) { Ok ('Spin-off task registered: "' + $zohoTask + '" (SYSTEM, hourly, self-deletes on success).') Invoke-Native 'schtasks.exe' @('/Run','/TN',$zohoTask) | Out-Null Ok 'First run kicked off in the background. Progress: C:\ProgramData\ssh\logs\install-zoho.log' Info 'It installs silently once a token/artifacts appear via config.json, drop\, or the MAD Approve broker (lingering-approval model - quiet hourly re-checks, no errors while waiting).' } else { Warn2 ('Could not register the "' + $zohoTask + '" task - Zoho spin-off skipped (SSH is unaffected).') } } } catch { Warn2 ('Zoho spin-off setup failed (non-fatal, SSH is unaffected): ' + $_.Exception.Message) Warn2 'Manual path: download install-zoho.ps1 from boot.mad.sg/install-zoho.ps1 and run it elevated.' } Write-Host '' # ============================================================================ # KEY ACTIONS -- concise, operator-facing "what to do next" (operator feedback # 2026-07-12: the full RESULT table reads fine on a console but is noise on # Telegram; end with a short, phone-readable action list). Derived from state. # ---------------------------------------------------------------------------- # AV LEFTOVER detection: a third-party AV can be "uninstalled" as a product yet # leave self-protected services/drivers behind (e.g. McAfee mfehidk / mfevtp / # mfemms) that STILL kill the session-0 sshd service -- the exact "reports clean # (PATH A) but SSH is dead" trap. Flag it so the action list says "run MCPR". # ============================================================================ $avLeftover = $false; $leftNames = @() try { $mfeSvc = @(Get-Service -ErrorAction SilentlyContinue | Where-Object { $_.Name -match '(?i)^(mfe|mccsp|mcapexe|mcafee)' }) $mfeDrv = @(Get-CimInstance Win32_SystemDriver -ErrorAction SilentlyContinue | Where-Object { $_.Name -match '(?i)^mfe' }) $leftNames = @(@($mfeSvc | ForEach-Object { $_.Name }) + @($mfeDrv | ForEach-Object { $_.Name })) | Sort-Object -Unique if ($leftNames.Count -gt 0) { $avLeftover = $true } } catch { } $zohoUp = [bool](Get-Service -Name 'ZohoURSService' -ErrorAction SilentlyContinue | Where-Object { $_.Status -eq 'Running' }) $mcpr = 'irm ' + $BOOT_URL.Replace('https://','') + '/mcpr | iex' $actions = New-Object System.Collections.Generic.List[string] if (-not $tsip) { $actions.Add('Machine NOT reachable: provide a tagged Tailscale pre-auth key and re-run.') } elseif (-not $script:ChannelOk) { if ($avPresent) { $actions.Add('Remove the AV (' + ($thirdPartyAV -join ', ') + ') at the console, then re-run: irm ' + $BOOT_URL.Replace('https://','') + ' | iex') } elseif ($avLeftover) { $actions.Add('SSH will not stay up but NO AV product is installed = AV LEFTOVER (' + ($leftNames -join ', ') + '). At the console run the McAfee cleanup: ' + $mcpr + ' -- click through MCPR, then REBOOT; SSH auto-recovers.') } else { $actions.Add('No SSH listener despite no AV detected: check host keys / sshd_config / the sshd event log (STEP 10-11 above).') } } else { if ($avPresent) { $actions.Add('SSH is UP but AV present (' + ($thirdPartyAV -join ', ') + ') and NOT reboot-safe. Remove it (MCPR / console / Zoho-GUI), then re-run to flip to the permanent service.') } elseif ($avLeftover) { $actions.Add('SSH is up, but a McAfee LEFTOVER is present (' + ($leftNames -join ', ') + '): at the console run the McAfee cleanup: ' + $mcpr + ' + reboot to make it reboot-survivable.') } } if (-not $zohoUp) { $actions.Add('Zoho secondary channel not up: install it (fresh token from assist.zoho.com -> config.json -> install-zoho.ps1).') } if ($anyFail) { $actions.Add('Investigate the [XX] FAIL rows in the result table above.') } if ($actions.Count -eq 0) { $actions.Add('None -- fully enrolled + reboot-survivable. Tidy up: delete the enrolment gist + short links.') } Send-RemoteLog '' Send-RemoteLog '>>> KEY ACTIONS (what to do next) >>>' Write-Host '' Write-Host ' >>> KEY ACTIONS (what to do next) >>>' -ForegroundColor Cyan $ai = 1 foreach ($a in $actions) { Send-RemoteLog (' ' + $ai + '. ' + $a) Write-Host (' ' + $ai + '. ' + $a) -ForegroundColor Cyan $ai++ } Write-Host '' # ============================================================================ # AUTO-LAUNCH MCPR when AV / an AV leftover is detected in an INTERACTIVE session # (operator request 2026-07-12: "build the uninstall in" -- so the SAME boot # command the operator already runs pops the remover, no second command to learn). # SAFETY: this LAUNCHES McAfee's OWN GUI remover (MCPR) and stops -- the operator # clicks through it and chooses to reboot. It is NOT the headless/forced removal # that bricked a box on 2026-07-11 (that stays banned). Fires ONLY with a desktop # (session >= 1); headless/session-0 runs never auto-launch a GUI. Fully cancellable. # ============================================================================ $mySess = (Get-Process -Id $PID).SessionId # McAfee ONLY: MCPR is McAfee's remover - do NOT pop it for a Norton/Avast/etc box. # $avLeftover is McAfee-specific (mfe* svcs/drivers); $thirdPartyAV may name any vendor. $mcafeeHere = (@($thirdPartyAV | Where-Object { $_ -match 'McAfee' }).Count -gt 0) -or $avLeftover if ($mcafeeHere -and ($mySess -ge 1)) { try { Banner 'McAfee present - launching the MCPR remover on screen (click through it, then REBOOT)' $mcprExe = Join-Path $env:USERPROFILE 'Downloads\MCPR.exe' Info ('Downloading MCPR to ' + $mcprExe + ' ...') Invoke-WebRequest -Uri 'https://download.mcafee.com/molbin/iss-loc/SupportTools/MCPR/MCPR.exe' -OutFile $mcprExe -UseBasicParsing -TimeoutSec 300 if ((Test-Path $mcprExe) -and ((Get-Item $mcprExe).Length -gt 100000)) { Start-Process -FilePath $mcprExe Ok 'MCPR is now OPEN on screen. Click: Next > Agree > type the CAPTCHA > let it run (~3-5 min) > RESTART.' Ok 'After that reboot, McAfee + its leftover driver are gone and SSH comes up on its own.' Send-RemoteLog '[OK] MCPR remover LAUNCHED on screen - click through it (Next/Agree/CAPTCHA), then REBOOT. SSH auto-recovers after the reboot.' } else { Warn2 'MCPR download failed or too small - run irm boot.mad.sg/mcpr | iex manually, or get it from mcafee.com/support (article TS101331).' Send-RemoteLog '[!!] MCPR auto-download failed - run irm boot.mad.sg/mcpr | iex manually.' } } catch { Warn2 ('Could not auto-launch MCPR: ' + $_.Exception.Message + ' - run irm boot.mad.sg/mcpr | iex manually.') Send-RemoteLog ('[!!] MCPR auto-launch failed: ' + $_.Exception.Message + ' - run irm boot.mad.sg/mcpr | iex manually.') } } # ============================================================================ # BOOT-RETRY SELF-MANAGEMENT (operator design 2026-07-16): boot.ps1 manages its # own retry loop, so ONE `irm boot.mad.sg | iex` is all a human ever runs. # - Run ended WITHOUT a working sshd channel + port-22 listener, but Tailscale # IS up and no third-party AV is in the way (i.e. a retry can actually # converge): self-install scheduled task 'boot-retry' (SYSTEM, highest, # every 2 min) that re-runs `irm boot.mad.sg | iex`. Every retry fetches the # CURRENT boot.ps1, so each fix deployed to boot.mad.sg is picked up # automatically until the box converges. Survives reboots - so when the fix # IS a reboot (delete-pending service name), it converges right after one. # - Run ended WITH a working channel + listener: remove any 'boot-retry' task. # The loop self-stops; it never lingers on (or lands on) a healthy machine. # - Tailscale down (needs an auth key) or AV present (needs a human): do NOT # install - a blind retry cannot help there and would just spam the log. # Manual stop, any time (admin PowerShell on the box): # schtasks /End /TN boot-retry; schtasks /Delete /TN boot-retry /F # ============================================================================ try { $retryName = 'boot-retry' $retryTask = Get-ScheduledTask -TaskName $retryName -ErrorAction SilentlyContinue $sshdUp = [bool]($script:ChannelOk -and (Test-Port22Listening)) # CONVERGED = the PERMANENT sshd is safe. A lifeline sshd under a 3rd-party AV # (PATH B) is a channel, NOT convergence - it dies on reboot. Only remove the # retry loop when genuinely converged. Keeping the loop alive on AV boxes is the # self-update path Mike asked for: the moment the operator removes the AV # (Zoho/console), the next tick re-runs the CURRENT boot.mad.sg, flips to PATH A, # makes the box reboot-proof, and self-removes - with NO manual SSH re-run. Every # fix pushed to boot.mad.sg in the meantime propagates automatically. # (Fable review 2026-07-30, finding 2.) $converged = $sshdUp -and (-not $avPresent) if ($converged) { if ($retryTask) { Unregister-ScheduledTask -TaskName $retryName -Confirm:$false -ErrorAction Stop Ok ('boot-retry task REMOVED on ' + $env:COMPUTERNAME + ' - converged (permanent sshd up, no AV).') Send-RemoteLog ('[OK] boot-retry task removed on ' + $env:COMPUTERNAME + ' - converged (sshd up, port 22 listening, no 3rd-party AV).') } } elseif ($tsip) { # Slow the loop while it is waiting on a human (AV removal) - nothing is # time-critical there, and it keeps the log/summary noise down. Fast otherwise. $retryMins = if ($avPresent) { 30 } else { 2 } $why = if ($avPresent) { 'a 3rd-party AV is present (lifeline only; converges automatically once the AV is removed)' } else { 'enrolment ended without a working permanent sshd' } if (-not $retryTask) { $rAct = New-ScheduledTaskAction -Execute 'powershell.exe' -Argument ('-NoProfile -ExecutionPolicy Bypass -Command "irm ' + $BOOT_URL + ' | iex"') $rTrg = New-ScheduledTaskTrigger -Once -At (Get-Date).AddMinutes($retryMins) -RepetitionInterval (New-TimeSpan -Minutes $retryMins) -RepetitionDuration (New-TimeSpan -Days 3650) $rSet = New-ScheduledTaskSettingsSet -MultipleInstances IgnoreNew -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Minutes 30) Register-ScheduledTask -TaskName $retryName -Action $rAct -Trigger $rTrg -User 'SYSTEM' -RunLevel Highest -Settings $rSet -Force ` -Description ('boot.mad.sg auto-retry for ' + $env:COMPUTERNAME + ' (installed ' + (Get-Date -Format 'yyyy-MM-dd HH:mm') + ' because ' + $why + '). Re-runs irm boot.mad.sg | iex every ' + $retryMins + ' min as SYSTEM, fetching the CURRENT script each time; boot.ps1 removes this task automatically once the PERMANENT sshd is up with no AV. Manual stop: schtasks /Delete /TN boot-retry /F') | Out-Null Warn2 ('boot-retry task INSTALLED on ' + $env:COMPUTERNAME + ': re-runs boot.mad.sg every ' + $retryMins + ' min (SYSTEM), fetching the CURRENT script, until the permanent sshd is up with no AV - then self-removes. Survives reboots.') Send-RemoteLog ('[!!] boot-retry task installed on ' + $env:COMPUTERNAME + ' - auto-retrying every ' + $retryMins + ' min (' + $why + '); self-removes on convergence; each retry fetches the CURRENT boot.ps1.') } else { Info ('boot-retry task already present - still retrying every ' + $retryMins + ' min until converged (self-removes on success).') Send-RemoteLog ('[..] boot-retry task already present - still retrying every ' + $retryMins + ' min until converged.') } } } catch { Warn2 ('boot-retry self-management failed (non-fatal): ' + $_.Exception.Message) Send-RemoteLog ('[!!] boot-retry self-management failed: ' + $_.Exception.Message) } # ============================================================================ # CLEAR ENDING for the Telegram stream (per operator feedback: it was not # obvious when the run had finished). One unmistakable end-of-run summary line. # ============================================================================ $endStatus = if ($anyFail) { 'COMPLETE - with FAILURES (see above)' } else { 'COMPLETE - ALL PASS' } if ($script:ChannelOk) { $endAccess = 'ACCESS OK: ' + $script:ChannelDesc + $(if ($tsip) { ' -> ssh ' + $env:USERNAME + '@' + $tsip } else { '' }) } else { $endAccess = 'ACCESS: NONE established - see the failures above' } Send-RemoteLog '' Send-RemoteLog ('=====================================================') Send-RemoteLog ('=== ENROLLMENT ' + $endStatus) Send-RemoteLog ('=== ' + $endAccess) Send-RemoteLog ('=== RunId ' + $script:RunId + ' - finished. You can stop watching now.') Send-RemoteLog ('=== Local install log kept for reference at: ' + $script:LocalLog) Send-RemoteLog ('=====================================================') Write-Host '' Write-Host ('=== ENROLLMENT ' + $endStatus + ' ===') -ForegroundColor Cyan Write-Host ('=== ' + $endAccess) -ForegroundColor Cyan Write-Host ('=== Local install log: ' + $script:LocalLog) -ForegroundColor Cyan # Final remote-log flush: force-send whatever is still buffered so the tail of # the run (RESULT table, verdict, Zoho spin-off, and this ENDING) is never lost. Flush-RemoteLog } # ---- end TOP-LEVEL RESILIENCE try/catch/finally: the RESULT + KEY ACTIONS + ENDING # + Flush above run on EVERY exit path, so a failure is always reported ----