Ever since Prince Harry and Meghan Markle decided to step down as working members of the royal family and relocate to California, there has been a rift between them and rest of Harry’s family.

Obviously turning your back on your family and their expectations can have consequences, but when you’re a royal, it’s even more complicated. Still, they are family, and according to royal expert Sarah Hewson, King Charles III, Harry’s dad, “doesn’t want this distance with his son, and he certainly doesn’t want this barrier between him and his grandchildren.”

Hewson told The Sun that Charles has hardly even seen Harry’s children, Archie and Lilibet. She explained, “He’s met Lilibet once, Archie just a handful of times.” She added, “That’s really, really sad. Obviously geography plays a part in that, we aren’t going to see the King jumping on a plane to California to see them. We know things with the Waleses and the Sussexes are extremely tense. I don’t see any signs of any thawing there — and it’s really not a priority.”

Despite it being sad that the grandchildren are growing up without spending much time with their grandfather, there is one big reason why it might be best for the Sussex’s to stay away. If Meghan and Harry jumped on a plane with their children with the intention of visiting Harry’s family, that would be quite stressful for everyone, and stress is the last thing you need when you’re already dealing with major health issues.

The King is battling cancer, and Harry’s sister-in-law, Princess Catherine, is also recovering from preventative chemotherapy treatments for cancer that was discovered earlier this year.

Hewson explained, “One of the things the Waleses really want to focus on now is keeping things calm, stress levels down, during the Princess’ recuperation. I really don’t think a reunion with Harry and Meghan is going to do anything for stress levels, and it’s just not their focus.”

Another reason the Sussex’s are probably unlikely to have a reunion any time soon is that Meghan doesn’t want to. Hewson shared, “There have been suggestions that there will be this huge full family summit and everybody gathered to thrash things once and for all, I don’t buy into that. I can’t see that happening. And all the signs around Meghan are that she doesn’t really have any desire to set foot in this country again.”

The last few times Harry has visited England, such as for the 10th anniversary of the Invictus Games, Meghan did not go with him.