Which cliff? Which ladder? Are those cliffs done using the cliff autotile (the one near the trees on the A tab) or the one at the very bottom? If it's the one at the very bottom, it's probably not passable downwards like the autotile one is, and it looks like your ladder only reaches to the top of the tile below it. I'd just change the 4-dir passage setting on the cliff tile right at the bottom of the A tab, to allow you to move down. That's how the passage on the autotile works. Otherwise, you're going to have to make a B-E tile that has no sprite but a O passage setting, just to override it.