Eleven faults in an Exchange 2019 lab, and their real causes

Six virtual machines: a domain controller with DNS and a certification authority, an Exchange 2019 server, an Edge Transport server in a real DMZ, a firewall, a security gateway standing in for a cloud service, and a client simulating the outside world.

Over the course of building it, eleven things broke. In nine of them the real cause turned out to be different from the first hypothesis — and the pattern in how they differed is more useful than any individual fix.

The eleven

Symptom observed	Actual cause	Fix
No SCL header on the test message	The Content Filter ignores authenticated internal mail	Simulate an external sender over an anonymous SMTP session
PowerShell session hangs during SMTP send	A blocking read at a point where the protocol sends no reply	Read responses only where the protocol defines one
No ping between machines	ICMP not permitted by the firewall — not promiscuous mode	Explicit ICMP rule on the DMZ interface, plus an inbound rule on Windows
Edge installation aborts	No fully qualified domain name on a workgroup server	Assign the primary DNS suffix manually
-RequestFile not recognised	Parameter removed in the 2022 cumulative updates	Collect the request into a variable and write the file
Error importing the subscription	The internal server wants -FileData, not -FileName	Read the file as bytes and pass it as data
Outbound mail goes to the gateway instead of Edge	A manual connector with a lower cost than EdgeSync’s	Disable the manual connector
454 Relay access denied from the gateway	Sending to port 25, which is reserved for inbound	Use port 26 on the send connector
HTTP 500.19 on the virtual directories	NTFS permissions missing the IIS_IUSRS group	Grant read and execute with icacls
Bridge adapter list is empty	The hypervisor’s bridged networking component was not installed	Reinstall the virtualisation product
VM will not start, 0xc0000409	Hypervisor kernel driver blocked or absent	Reboot the host and reinstall

Six patterns worth keeping

The sophisticated hypothesis is not the likely one

The machines could not ping each other. The first theory was promiscuous mode on the virtual adapters — a refined explanation, and one that concerns the ability to observe other people’s traffic rather than to receive your own.

The real cause was a missing firewall rule. The most ordinary possible explanation in a network that has a firewall in it.

The new component works; the old one is interfering

Mail was bypassing the Edge server. All the attention went to the EdgeSync subscription, which had just been created and was therefore the obvious suspect.

The subscription was working perfectly. What interfered was a send connector created days earlier and never removed, which won on cost.

A system that refuses may be right to refuse

The gateway answered 454 Relay access denied. It was applying its own policy exactly as configured: port 25 accepts mail only for the relay domains it knows about. The message was addressed to an external domain, so refusing was correct.

There was nothing to repair in the gateway. There was a port to correct on the sender.

Documentation ages

-RequestFile appears in countless guides for generating a certificate request. It was removed in a cumulative update, for a specific security reason: it accepted UNC paths.

Following a tutorial without checking the version produces an error that reads like a syntax mistake of your own.

Absence is information

Three of the eleven diagnoses rest entirely on something that was not in the logs:

• the absence of a DELIVER entry proved the antimalware agent had intercepted the message
• the absence of any event on the Edge server proved the mail had taken a different path
• the absence of ANONYMOUS LOGON in the connector permissions proved there was no open relay

The layer underneath is part of the system

Two of the eleven faults had nothing to do with Exchange or with networking. They were the hypervisor: a missing bridged-networking component, and a kernel driver that stopped the VM from starting at all with 0xc0000409.

Two details worth stealing

The Content Filter ignores authenticated internal mail. This is by design and it makes testing anti-spam from inside the organisation meaningless: no SCL header will ever appear. To test it you have to arrive the way real spam arrives — an anonymous SMTP session from outside.

Reading SMTP by hand blocks where the protocol says nothing. Scripting an SMTP conversation means reading a response after each command — except where the protocol does not define one. A blocking read at that point hangs the session, and it looks exactly like a network problem.

The common thread

Nine of eleven had a cause different from the first hypothesis, and the direction of the error is consistent: the first hypothesis was almost always more interesting than the truth.

Promiscuous mode rather than a firewall rule. A broken new subscription rather than an old connector nobody removed. A gateway misconfiguration rather than the gateway doing its job.

There is probably something to that. When a system is unfamiliar, the parts you have just learned about are the parts that come to mind — and they are rarely the parts that are wrong.