There are numerous reasons why people will not visit or return to specific spots. Those are woods, deserted islands, caverns, relinquished houses, and numerous different spots. There are several distinct purposes behind that, similar to dread of death, superstition, strict reasons, and so on.
This one is a particularly terrible spot for all people. It has some extremely dim history and individuals simply would prefer not to go there.