Help Keep Boards Alive. Support us by going ad free today. See here: https://subscriptions.boards.ie/.
If we do not hit our goal we will be forced to close the site.

Current status: https://keepboardsalive.com/

Annual subs are best for most impact. If you are still undecided on going Ad Free - you can also donate using the Paypal Donate option. All contribution helps. Thank you.

https://www.boards.ie/group/1878-subscribers-forum

Private Group for paid up members of Boards.ie. Join the club.

Automatically generate a content inventory

destraynor · 2007-04-30 15:06:23

Hi all, Just a quick question. Does anyone know of any software that can generate a full website inventory. i.e. given a url, generate all the valid URLs within that domain name, by following links on the site. I thought wget or curl could do it, but the --spider option doesn't seem too useful. Any ideas?

30-04-2007 04:06PM

#1

destraynor

Registered Users, Registered Users 2 Posts: 100 ✭✭

Join Date: August 2005

Posts: 83

Hi all,
Just a quick question. Does anyone know of any software that can generate a full website inventory.

i.e. given a url, generate all the valid URLs within that domain name, by following links on the site.

I thought wget or curl could do it, but the --spider option doesn't seem too useful. Any ideas?

0

Comments

#2 30-04-2007 08:33PM

grahamor

Registered Users, Registered Users 2 Posts: 872 ✭✭✭

Join Date: August 2005

Posts: 557

I use XML Sitemaps to generate all the links contained in a site i developed.

You can see a sample result below (it saved me typing in 100+ URL's )

http://www.cheap-hotel-dublin.com/sitemap.xml

Hope it helps

0
#3 30-04-2007 11:13PM

destraynor

Registered Users, Registered Users 2 Posts: 100 ✭✭

Join Date: August 2005

Posts: 83

Nice one grahamor, I can use that plus some XSLT or Perl to whittle it down to what I need no bother.

Thanks.

Des

0