This is probably more of a web-based question. I have a basic web crawler I wrote with Java (using JSoup) which crawls through our repositories and stuff to locate the paths to certain snapshots and whatnot. It crawls through using the file tree which gets exposed when you try to go to a directory in a URL.
The problem I am having is that some of the directories contain index files, which means the connection to the directory redirects from the file tree to the index file. Does anyone know if there is a way to prevent that redirection?