There’s already been one attempt to exploit XPI the same way ActiveX is routinely exploited.
The upshot is that unlike Internet Explorer, Mozilla products are being actively developed and will get fixes for any real exploits (not just install prompts) that come along fairly quickly.