IMHO unsafe should be used in a way that provides the most value. And therefore there is a large grey area where some judgement is required.
Any short rule (like "only things that could cause UB at the language level should be unsafe" or "everything dangerous should be unsafe") will not provide as much value as possible.
The goal should be that it's impossible to trigger UB in safe code (ie. avoid unsoundness), and also mark "dangerous" things unsafe as long as it stays possible to write meaningful real-world software without cluttering it with unsafe blocks. It should be possible to write whole applications where all uses of unsafe blocks are in very isolated locations (eg. for a typical embedded application: only some short initialization code needs unsafe blocks, and definitely not any business logic)
But it's totally fine to mark a function unsafe if it's very likely that calling it without observing some non-obvious preconditions will break your software, even if it's technically not UB. Eg. if you have to wait for a PLL to settle before using it as a clock source, a function that changes the clock source without checking that precondition could be unsafe. And if there's a reasonable way to provide a safe wrapper that always checks the precondition first, by all means provide that wrapper!