[NTLUG:Discuss] Use wget to generate all urls from a site?

[Wayne Walker]

wget --mirror --no-verbose http://www.bybent.com/
or
wget --mirror --no-verbose http://www.bybent.com/ 2>&1 | sed -e 's/^.* URL://' -e 's/ \[.*//'

You must "rm -rf www.bybent.com" before subsequent runs as it won't
retry existing pages.

NOTE: this will make a local copy of the site at the same time.

On Sun, Aug 05, 2007 at 03:51:17PM -0500, Neil Aggarwal wrote:
> Hello:
> 
> I have been trying to figure this out but cant:
> 
> Is there a way to get a list of all urls in a site
> using wget?
> 
> Thanks,
> 	Neil
> 
> 
> --
> Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com
> FREE! Eliminate junk email and reclaim your inbox.
> Visit http://www.spammilter.com for details.
> 
> 
> _______________________________________________
> http://www.ntlug.org/mailman/listinfo/discuss

-- 

Wayne Walker

www.unwiredbuyer.com - when you just can't be by the computer

wwalker at bybent.com                    Do you use Linux?!
http://www.bybent.com                 Get Counted!  http://counter.li.org/
Perl - http://www.perl.org/           Perl User Groups - http://www.pm.org/
Jabber:  wwalker at jabber.gnumber.com   AIM:     lwwalkerbybent
IRC:     wwalker on freenode.net