Should I be worried about Rockwool dust?

No! The World Health Organization has confirmed it to be bio-soluble and safe.