Do not add to it for ever.
Do it differently
for each chunk of your world in memory, repeat when reloading
Determine if traversable northsourt, eastwest,.
Then only store 4 boolean per tile, that contain where you can go.
Now if you need a way, find based on those using a* a way from start to target chunk.
For the current chunk the npc is in, generate/load a navmesh, to do the detail navigation within the chunk.
Advanced:_add a cost metric for each chunk. So climbing over a mountain is avoided, if it is possible to go a slightly longer flat way.
Bonus:
Store the 4 booleans as the first 4 bits of a byte, and use the remaining 4 for the cost metric.
That way you can have really large worlds at nearly 0 memory cost from the navigation side.