I just learned that this one is missable!

For this achievement you have to unlock 4 hermits using the secondary ingame currency "gems" and then sail away to another island. One hermit can be found on every island waiting for you to be unlocked somewhere in the woods in a cottage. When unlocked the hermits will automatically hop on the boat when you set sail.
There are two types of chests on the islands. One type just has some coins in them and the other type has 2 to 3 gems per chest in them. As there are more unlockables using gems then gems in a single playtrough make sure you safe some gems for unlocking at least 4 hermits.
As the chests never despawn unless opened my advise is to not open the gem chests unless you want to spend the gems shortly after. If you don't want to spend them they will take up space in your money bag and can be stolen by the greed. You get back some gems when you destroy the greed but as FaSCoRP stated in the comments you do not get back all gems.
In total there are 5 unlockable hermits:
Ballista Hermit1st island
3 gems
Turns triplet towers into ballista towers
Stable Hermit2nd island
1 Gem
Turns mill houses into stables
Bakery Hermit3rd island
4 gems
Turns triplet towers into bakeries
Knight Hermit4th island
2 gems
Turns triplet towers into knight towers
Horn Hermit5th island
3 gems
Turns tall stone walls or iron walls into rally walls
You need at least 9 gems to unlock 4 hermits for this achievement.