dglp's Avatar
dglp 4
4 Asked
6 Answered
1 Best
0
No one has voted on this question yet :(
3 years, 5 months ago

What search engine/method will yield files in website subdirectories?

Suppose I want to find all the files in subdirectories of www.site.net, e.g. files at www.site.net/sub1/sub2/*.*. The major search engines don't yield results from anything but the root. Is there a reliable method of searching subdirectories on a given site?
Tip for best answer: M$0.00
Separate topics with commas, or by pressing return. Use the delete or backspace key to edit or remove existing topics.

You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.

M$

What is Your Answer?

0
0
0

4 Answers

0
tko's Avatar
tko | 3 years, 5 months ago
4
All of the search engines parse subdirectories. Provided those subdirectories are linked from somewhere, and don't have code in them, or on the site telling search engines *not* to parse them.

Search engines are big money these days. They'll search as exhaustively as their programmers can devise. (Heck, look at the amount of PDF's that are searchable ..image and audio recognition are going to start playing a big part soon too. Soon there won't be much you *can't* find.)

You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.

M$
dglp's Avatar
dglp | 3 years, 5 months ago Report

I would like to give this a tie for 'best answer' since it answers part of my question very clearly and emphatically.

Report Abuse

Post Reply Cancel
0
loopy1's Avatar
loopy1 | 3 years, 5 months ago
4
ALL of them will, as long as the website doesn't restrict the search engine from looking in subdirectories with a restrictive robots.txt file. For example, I did a search on google for chicagotribune.com, and the third result was archives.chicagotribune.com/2008/nov/11/health/chi-081111bishops. While it's not technically a subdirectory (for the reason that most modern websites don't run off of flat html files that are stored in a directory structure) it may be the type of result you are looking for.
source(s):
Experience.

You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.

M$

Report Abuse

Post Reply Cancel
0
jeffhoard's Avatar
jeffhoard | 3 years, 5 months ago
8
I think you might have luck if you go to Google.com and type in your search query followed by site:websitetitlehere.com

So if you wanted to find all the pages on Mahalo related to Sarah Palin you would get these results...

http://www.google.ca/search?hl=en&q=sarah+palin+site%3Amahalo.com&btnG=Google+Search&meta=

You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.

M$
dglp's Avatar
dglp | 3 years, 5 months ago Report

This is the best answer in that it provides a method and a direct answer to the question.

dglp's Avatar
dglp | 3 years, 5 months ago Report

Very fast result Jeff. I had tried this various times in the past, and didn't get results. But I've just tried it in 3 places and it's worked on two of them. The 3rd one seems to have locked subdirectories - and maybe that's why the search fails.

Report Abuse

Post Reply Cancel
0
ilaksh's Avatar
ilaksh | 3 years, 5 months ago
4
You could use a sitemap generator like http://www.xml-sitemaps.com . xml-sitemaps.com has options to download the sitemaps in html, xml or text lists of urls. Or try http://www.sitemapdoc.com/

You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.

M$
dglp's Avatar
dglp | 3 years, 5 months ago Report

This is my favourite answer among the first four, because it gave me an alternative way of looking. Very useful!

Report Abuse

Post Reply Cancel

Learn something new with our FREE educational apps!

Private lessons in the comfort of your own home. Get back in shape or finally pick up a guitar with our great experts guiding you the whole way!
Learn Guitar
Learn Hip Hop
Learn Pilates