So, the three relevant safety items:

* How long is it live? if it's statically allocated, you're fine. If it's stack allocated, you have to make sure it never leaves the given thread, ideally with a lifetime annotation. If it can be dropped at runtime, you have to represent the "owner" of it, boxing it is one way of achieving it
* Mutability - there never may be more than one mutable access, probably mutexes and such, you seem to have a better idea here.
* Pinning - is it necessary that (temporary) mutable access always represents the SAME item? In rust, `&mut` must always be a VALID instance, but may be replaced, indiana jones style with `core::mem::swap`. If that's NOT acceptable, then you must use `Pin`.