A good book on recreational/non-relationship sex (aka swinging) is “Together sex”.

Poly is having multiple relationship, it does not means that there must be on equal footings, I have learned that everybody is different and each situation is unique so no need to fit yourself in a one-size fits all.

What you describe are two relationships. One can say that it is unfair to place the burden of fulfilling all of your needs on one relationship, for exemple you might want a hiking partner or a sex partner if your relationships do not fulfill these needs. Onnthe other hand if a relationship becomes empty you might consider moving away – loving the person is not a good criteria for deciding to stay with somebody, but it can be a factor. I know I love a good number of people and want them to be happy, but that does not mean I want or can live with them or even have a continuing relationship with them, even if they state that this would make them happy.

In other words, you have to decide what to do. First be happy, then decide what relationships you want to have, then find non-relationship partners for the remaining needs.