Hi,

Great article -- this was quite an interesting demo of how much can be
built from a simple primitive.

As you said, XXTEA is not a proper compression function and doesn't fit
the Merkle-Damgard assumptions, letting the attacker forge MACs. This
doesn't break the scheme fully since the attacker can't compute a
ciphertext from a plaintext without the key, but it is reduced to
unauthenticated XXTEA-CTR.

The attacker needs to see a single ciphertext and the corresponding MAC.
Let's say the ciphertext is (key || c0 || c1 || ... || cn), where ci are
32-bit blocks. The MAC is then computed as

MAC = enc(cn, enc(c[n-1], enc(..., enc(c0, HK)...)))

where HK is the inner hash state after processing the key blocks.

(this ignores the tie-off step for simplicity, but handling it only
requires an extra swap in the attack)

The attacker knows all keys ci, and can decrypt each layer one by one:

HK = dec(c0, dec(..., dec(c[n-1], dec(cn, MAC))...))

and now compute MAC(key || M) for an arbitrary message M by starting
from this value.

There are several constructions that turn block ciphers into one-way
compression functions, such as Davies-Meyer: instead of using E(in,ctx)
as the compression function, you can use E(in,ctx)^ctx. This prevents
the attacker from simply decrypting the block cipher -- an attack that
relies on properties of XXTEA might be possible, but it wouldn't be
independent of the primitive.

Thanks, Dimitrije! I've learned something new, so my exercise continues to 
serve its purpose. My mistake seems really obvious after having it spelled 
out. Alternative to your suggested fix, it seems HMAC would have rescued 
this MAC as well. I'd have been better off skipping the "swap" trick 
altogether and just relying on that.