If two people in a close relationship grow apart, or if they grow away from each other, they gradually begin to have a less close relationship, usually because they no longer have the same interests and want the same things.

There was nobody else involved - we just grew apart. 没有任何外人插足，我们只是渐渐疏远了。