Puzzling.
Downy mildews are almost always on the underside of leaves – though often showing a corresponding yellowish area on the upper surface, so as you suggest hariotii can be ruled out. The two powdery mildews on Buddleia are restricted to southern Africa and India respectively and neither occurs on Buddleja davidii which is native to parts of China and Japan.
So probably not fungal, it looks like abnormal growth of hairs/down – I’ll keep an eye out for anything similar. My only thought is that the plant appears to be growing in the gutter / drainage area? Perhaps that’s significant.
Chris