No, you are NOT wrong! Do NOT let him buffalo you into that kind of relationship. If you don't want it, there's no way you would be happy. He would be happy, for damned sure, but you wouldn't.

Even in a M/s relationship, unless you are going for the full deal, you have to consent to whatever is going on. The kind of deal he's talking about wouldn't be good for either you or your daughter. He is the only one who benefits. My advice is to walk away, give him the divorce and soak him for alimony and child support. Then, maybe, find someone who cares about you, and your daughter, more than about himself.

Sure, it's a hard step to take. But for your own happiness and, I think, safety, it's one you must take.

Good luck.