Most secure method is removable media. Flash memory stick (geek drive, thumb drive). After that, anything retained on the system is hackable via a key logging program. Additionally, if your spouse is persistent, they could employ timed screen captures via a watchdog program, and thus see anything you did on that PC.

Personally, I would use a removable hard drive on a laptop. Secure it in a combination locked safe, one that would be registered at the local lock and key locations to prevent his fabricating a story to break into the safe.

On top of that, operating offsite for your privacy would be the only additional way to safeguard your information.

I do need to mention data recovery capabilities. Depending on the cost the spouse is willing to accept, any activity that was done on a particular hard drive is recoverable unless the hard drive is physically destroyed. The only way to permanently destroy a hard drive is to disassemble the drive, and put each disk against a permanent magnet to erase all data. Formating the drive via any software method does not erase all data.