Running wget 10
Options may come after the URLs, and a short option can carry its value without a space: -olog is the same as -o log.
| Command | What it does |
|---|---|
wget [OPTION]... [URL]... | Basic syntaxEvery URL given is downloaded in turn, and options may stand before or after the URLs. |
wget URL | Download fileSaves into the current directory; a slow or broken transfer is retried up to 20 times by default. |
wget -drc URL | Bundle short optionsFlags that take no argument can share one dash; this equals wget -d -r -c URL. |
-b | Run in the backgroundLong form --background; without -o LOGFILE, all output is written to a file named wget-log. |
-e COMMAND | Run a wgetrc commandLong form --execute; it takes precedence over .wgetrc and can be repeated for several commands. |
wget -e robots=off URL... | Ignore robots exclusionEquivalent to setting robots to off in .wgetrc, so robots.txt and nofollow meta tags stop being honoured. |
--config=FILE | Use another startup fileRead instead of the default .wgetrc files; it wins when --no-config is given as well. |
--no-config | Skip startup filesStarts with no wgetrc settings at all; ignored when --config=FILE is also present. |
-V | Show the versionLong form --version; reports which GNU Wget release is installed. |
-h | Show helpLong form --help; lists each command-line option with a short explanation. |
Saving files 16
-c only matters for a file left by an earlier run; within one run Wget retries a dropped connection by itself.
| Command | What it does |
|---|---|
-O FILE | Write all output to one fileLong form --output-document=FILE; works like shell redirection, so every document the run fetches lands in FILE. |
-O - | Print to standard outputSends the downloaded documents to stdout for a pipe and turns off link conversion. |
-nc | Skip existing filesLong form --no-clobber; a local copy is kept and newer copies on the server are not fetched. |
-c | Resume a partial downloadLong form --continue; meant for a file left by an earlier run, since retries within one run happen anyway. |
--start-pos=OFFSET | Start at a byte offsetZero-based, accepts k and m suffixes, and overrides --continue when both are given. |
-N | Download only newer filesLong form --timestamping; a file is fetched only when it is missing locally or newer on the server. |
--no-use-server-timestamps | Keep local download timeLocal files get the time they were downloaded rather than the server's modification time that -N compares. |
-P PREFIX | Target directoryLong form --directory-prefix=PREFIX; the top of the retrieval tree, the current directory by default. |
-nd | No directory hierarchyLong form --no-directories; recursive runs save everything in one place and number repeated names. |
-x | Force directoriesLong form --force-directories; recreates host and path as folders even for a single file. |
-nH | Drop the host directoryLong form --no-host-directories; recursive downloads no longer start with a folder named after the host. |
--cut-dirs=NUMBER | Strip leading path partsLeaves out that many remote directory levels when building local paths, often together with -nH. |
-E | Add the .html extensionLong form --adjust-extension; HTML pages served from .asp or CGI addresses get .html appended locally. |
--content-disposition | Name files from headersExperimental and off by default; it can cost an extra HEAD request per file. |
--trust-server-names | Name files after redirectsAfter a redirect the local name comes from the final address, not from the URL you typed. |
--default-page=NAME | Set the default file nameApplies when the URL ends with a slash and gives no file name; otherwise wget picks index.html. |
Logging and input 15
| Command | What it does |
|---|---|
-o LOGFILE | Write messages to a logLong form --output-file=LOGFILE; messages that normally go to standard error end up in the file instead. |
-a LOGFILE | Append to a logLong form --append-output=LOGFILE; like -o, but an existing log is extended rather than replaced. |
-q | Quiet modeLong form --quiet; suppresses all of Wget's output, the progress bar included. |
-nv | Less verbose outputLong form --no-verbose; errors and basic information still appear, unlike with -q. |
-v | Verbose outputLong form --verbose; prints every available detail and is already the default mode. |
-d | Debug outputLong form --debug; works only when this Wget build was compiled with debug support. |
--show-progress | Always show the progress barDisplays the bar even with -nv or -q, which otherwise hide it outside verbose mode. |
--progress=TYPE | Choose the progress styleTYPE is bar, the default, or dot, which wget uses on its own when output is not a terminal. |
-S | Show server headersLong form --server-response; prints HTTP response headers and FTP server replies as they arrive. |
-i FILE | Download from listLong form --input-file=FILE; no URL is needed on the command line, and any given there are fetched first. |
-i - | Read URLs from stdinA dash in place of FILE makes wget take its list of addresses from a pipe. |
-F | Treat input as HTMLLong form --force-html; with -i, links are extracted from an HTML page rather than a plain list. |
-B URL | Base for relative linksLong form --base=URL; resolves relative links in an HTML input file, like a BASE tag would. |
wget --spider --force-html -i bookmarks.html | Check a bookmarks fileReads the links of a saved HTML bookmarks page and confirms that each target exists, saving nothing. |
--rejected-log=LOGFILE | Log rejected URLsWrites each rejection as CSV with the reason, the address and the page it came from. |
Retries, timeouts and speed 18
| Command | What it does |
|---|---|
-t NUMBER | Set the number of triesLong form --tries=NUMBER; 0 or inf retries forever, while refused connections and 404 errors are never retried. |
--retry-connrefused | Retry refused connectionsTreats a refused connection as transient, for mirroring servers that tend to vanish briefly. |
--retry-on-host-error | Retry name lookup failuresErrors such as a temporary DNS failure become transient, so the normal retry rules apply. |
--retry-on-http-error=CODE[,CODE,...] | Retry on HTTP status codesTakes three-digit codes such as 503 or 429 that are normally final; the usual retry limits still apply. |
-T SECONDS | Set all network timeoutsLong form --timeout=SECONDS; sets the DNS, connect and read timeouts in one go. |
--dns-timeout=SECONDS | Limit DNS lookup timeLookups that take longer fail; by default only the system resolver limits them. |
--connect-timeout=SECONDS | Limit connection setup timeTCP connections not established within that time are aborted; there is no limit of Wget's own otherwise. |
--read-timeout=SECONDS | Limit idle read timeCounts idle time only: if no data arrives for that long, the read fails and the download restarts. |
--limit-rate=AMOUNT | Limit speedBytes per second, with k or m suffixes; --limit-rate=20k caps the transfer at about 20 KB/s. |
-w SECONDS | Pause between downloadsLong form --wait=SECONDS; eases the load on the server and accepts m, h or d suffixes. |
--waitretry=SECONDS | Back off between retriesPauses only after failures, one second longer per retry, up to the given maximum. |
--random-wait | Vary the wait timeEach pause becomes a random 0.5 to 1.5 times the --wait value. |
-Q QUOTA | Set a download quotaLong form --quota=QUOTA; checked after each whole file, so it never cuts off a single-file download. |
--spider | Check links without savingWget only confirms that the pages are there and stores nothing on disk. |
--no-proxy | Bypass proxiesIgnores the proxy environment variables for this run, even when they are set. |
-4 | Use IPv4 onlyLong form --inet4-only; AAAA records and IPv6 addresses in URLs are ignored. |
-6 | Use IPv6 onlyLong form --inet6-only; A records and IPv4 addresses are ignored. |
--bind-address=ADDRESS | Bind to a local addressTakes a host name or IP; useful when the machine has several addresses. |
HTTP requests and logins 25
| Command | What it does |
|---|---|
--user=USER --password=PASSWORD | Log in with user and passwordCovers HTTP and FTP alike; --http-user and --ftp-user override it per protocol. |
--ask-password | Prompt for the passwordAsks interactively on each connection instead of taking the password as an argument; not allowed with --password. |
--http-user=USER --http-password=PASSWORD | HTTP loginWget answers the server's challenge with basic, digest or NTLM authentication. |
--no-netrc | Ignore .netrc credentialsWithout it, wget looks in .netrc when a server wants credentials that were not given on the command line. |
--auth-no-challenge | Send credentials unaskedSends plaintext Basic credentials with every request; meant only for odd servers and not recommended. |
--header=HEADER-LINE | Add a request headerGive a name, colon and value on one line; repeat the option for more headers. |
wget --header="Host: foo.bar" http://localhost/ | Override the Host headerConnects to localhost but sends foo.bar as the Host value, since --header can replace headers wget would generate. |
-U AGENT-STRING | Set the user agentLong form --user-agent=AGENT-STRING; replaces the default Wget/VERSION identification. |
--referer=URL | Set the Referer headerFor pages that only work when they appear to be reached from a link on another page. |
--post-data=STRING | POST a stringSwitches every request to POST and sends key=value pairs joined by & as the body. |
--post-file=FILE | POST a file's contentsSame as --post-data, except that the body is read from FILE. |
--method=HTTP-METHOD | Use another HTTP methodWhatever string follows is sent as the method name, which suits RESTful scripting. |
--body-data=DATA-STRING | Send a request bodyNeeded when a method set with --method must carry data; --body-file reads it from a file instead. |
--load-cookies FILE | Load cookiesReads a Netscape-style cookies.txt before the first request, typically to reuse a browser login. |
--save-cookies FILE | Save cookiesWrites cookies on exit; session cookies are skipped unless --keep-session-cookies is added. |
--keep-session-cookies | Keep session cookiesMakes --save-cookies store session cookies too, for sites that need a login first. |
--no-cookies | Disable cookiesWget then neither accepts cookies from the server nor sends any back. |
wget --save-cookies cookies.txt --post-data 'user=foo&password=bar' http://example.com/auth.php | Log in and store cookiesPosts a login form once and keeps the returned cookies; add --keep-session-cookies if the site uses session cookies. |
wget --load-cookies cookies.txt -p http://example.com/interesting/article.php | Fetch a page while logged inSecond step: reuses the saved cookies to download a members-only page with its requisites. |
--save-headers | Save headers with the fileThe response headers go at the top of the saved file, separated from the content by a blank line. |
--max-redirect=NUMBER | Limit redirectsCaps how many redirections are followed for one resource; the default is 20. |
--no-cache | Bypass server cachesSends Cache-Control and Pragma no-cache directives so a proxy returns a fresh copy. |
--compression=TYPE | Choose compressionauto or gzip asks the server for gzip-compressed transfer, and none turns that request off. |
--content-on-error | Keep error page contentSaves the body even when the server answers with an HTTP error status. |
--proxy-user=USER --proxy-password=PASSWORD | Proxy loginCredentials for an authenticating proxy, always sent with the basic scheme. |
HTTPS and certificates 9
| Command | What it does |
|---|---|
--no-check-certificate | Skip certificate checkAlso drops the host name match; verification has been on by default since Wget 1.10. |
--ca-certificate=FILE | Use a CA bundle fileA PEM bundle of certificate authorities, used instead of the system-wide locations. |
--ca-directory=DIRECTORY | Use a CA directoryOne PEM certificate per file, named by hash as the c_rehash utility produces. |
--certificate=FILE | Send a client certificateFor servers that demand a certificate from clients; PEM unless --certificate-type=TYPE says DER. |
--certificate-type=TYPE | Set the certificate formatPEM is assumed; DER, also called ASN1, is the other accepted value. |
--private-key=FILE | Client private keyLets the key live in a separate file from the client certificate. |
--secure-protocol=PROTOCOL | Choose the TLS versionAccepts auto, TLSv1_3, TLSv1_2, PFS and older SSL names; auto lets the SSL library pick. |
--https-only | Follow HTTPS links onlyApplies in recursive mode, where plain HTTP links are then skipped. |
--no-hsts | Ignore HSTSStrict-Transport-Security headers and stored HSTS policies are no longer applied. |
FTP 5
| Command | What it does |
|---|---|
--ftp-user=USER --ftp-password=PASSWORD | FTP loginWithout it the password defaults to -wget@, the usual anonymous FTP login. |
--no-passive-ftp | Use active FTPPassive mode, the default, has the better chance behind firewalls and NAT; this forces active transfers. |
--no-glob | Turn off FTP wildcardsCharacters such as * and ? in an FTP URL are then taken literally. |
--no-remove-listing | Keep FTP listingsLeaves the raw .listing files from FTP directories on disk, for example for debugging. |
--preserve-permissions | Keep remote permissionsFiles keep the permissions they had on the FTP server instead of those from the local umask. |
Recursive downloads 15
| Command | What it does |
|---|---|
-r | Download recursivelyLong form --recursive; follows links up to five levels deep and stays on the starting host. |
-l DEPTH | Set the recursion depthLong form --level=DEPTH; the default is 5, and 0 means the same as inf. |
-l inf | Unlimited depthRemoves the depth limit; -l 0 has the same effect, so it does not restrict a run to one page. |
-m | Mirror siteLong form --mirror; turns on recursion and timestamps with no depth limit, and keeps FTP listings. |
-k | Convert links for offline useLong form --convert-links; runs once the download ends and rewrites links to images, style sheets and pages. |
-K | Back up before convertingLong form --backup-converted; keeps the unconverted file with a .orig suffix, which -N then takes into account. |
--convert-file-only | Convert only file namesRewrites just the file part of each URL and pairs well with --adjust-extension. |
-p | Get page requisitesLong form --page-requisites; also fetches the images, sounds and style sheets a page needs to render. |
--delete-after | Delete after downloadingEach file is removed once fetched, which is useful for warming a proxy cache. |
wget -p http://SITE/1.html | One page with its requisitesLeave out -r and -l: only this page and what it needs are fetched, no linked pages. |
wget -E -H -k -K -p http://SITE/DOCUMENT | Save a page for offline viewingPulls requisites from other hosts too and fixes extensions and links so the copy displays locally. |
wget -p --convert-links -nH -nd -Pdownload http://www.example.com/dir/page.html | Save a page into one folderSame idea, but every file goes into download/ without host or path folders. |
wget -nc -r https://www.gnu.org/ | Resume a recursive downloadAfter an interruption, files already on disk are left alone instead of being fetched again. |
wget -m -k -K -E https://www.gnu.org/ -o /home/me/weeklog | Mirror a site for offline useShort form of the manual's mirror with converted links, .orig backups and .html extensions, logged to a file. |
0 0 * * 0 wget --mirror https://www.gnu.org/ -o /home/me/weeklog | Weekly mirror from cronA crontab line that refreshes the mirror every Sunday at midnight and logs to weeklog. |
Following links 9
A recursive run never visits other hosts unless -H is given; -D DOMAIN-LIST then narrows which hosts count.
| Command | What it does |
|---|---|
-np | No parentLong form --no-parent; guarantees the run stays below the directory it started in. |
-H | Span hostsLong form --span-hosts; without it, recursion never leaves the host given on the command line. |
-D DOMAIN-LIST | Limit to domainsLong form --domains=DOMAIN-LIST; comma-separated, and it only matters together with -H. |
--exclude-domains DOMAIN-LIST | Exclude domainsHosts in these domains are never followed, which narrows a run that spans hosts. |
wget -rH -Dexample.com http://www.example.com/ | Recurse across subdomainsFollows links to hosts such as images.example.com while staying inside example.com. |
-I LIST | Include directoriesLong form --include-directories=LIST; only these comma-separated directories are followed, wildcards allowed. |
-X LIST | Exclude directoriesLong form --exclude-directories=LIST; comma-separated with wildcards, and an empty list clears the .wgetrc value. |
-L | Relative links onlyLong form --relative; even absolute links to the same host are skipped. |
--follow-ftp | Follow FTP linksFTP links found in HTML pages are ignored unless this is set. |
Filtering files 7
| Command | What it does |
|---|---|
-A ACCLIST | Accept file typesLong form --accept; a comma-separated list of suffixes, or quoted patterns when it contains *, ?, [ or ]. |
-R REJLIST | Reject file typesLong form --reject; same list rules as -A, but matching files are skipped. |
--accept-regex URLREGEX | Accept URLs by regexMatches against the complete URL rather than only the file name. |
--reject-regex URLREGEX | Reject URLs by regexSkips every URL whose full address matches the expression. |
--regex-type REGEXTYPE | Choose the regex flavorposix or pcre; pcre needs a wget built with libpcre. |
--ignore-case | Case-insensitive matchingAffects -A, -R, -I and -X as well as FTP globbing, so *.txt also matches .TXT files. |
wget -r -l1 --no-parent -A.gif http://www.example.com/dir/ | Get all files of one typeHTTP has no wildcards, so recurse one level, stay below dir and accept only .gif. |
No row on this page matches that filter.
Notes
One file
wget URL saves into the current directory, and -P PREFIX chooses another one. Fetching a name that already exists keeps the old file and saves the new one as FILE.1; -nc skips such files and -N fetches only newer ones. -O FILE is not a rename: it works like shell redirection and writes everything the run downloads into that one file, which is why -N does not work with it. A dropped connection is retried within the run (20 tries by default, see -t NUMBER); -c is for finishing a file that an earlier run or another program left behind. -b detaches and logs to wget-log, -q silences output in scripts, --limit-rate=AMOUNT caps bandwidth and -i FILE reads a list of URLs.
Many files
-r follows links five levels deep (-l DEPTH changes that, -l inf removes the limit) and does not leave the starting host unless -H is given; it can still climb into parent directories, which -np prevents. -A ACCLIST keeps only matching files: wget -r -l1 --no-parent -A.gif http://www.example.com/dir/ fetches every GIF in one directory, because HTTP has no wildcards. -m stands for -r -N -l inf --no-remove-listing; with -k, -K and -E added, as in wget -m -k -K -E https://www.gnu.org/ -o /home/me/weeklog, the copy can be browsed offline. A recursive run honours robots.txt, and -w SECONDS spaces the requests out to spare the server.
Authentication and TLS
--user=USER --password=PASSWORD applies to HTTP and FTP alike. A password on the command line, or inside the URL, is visible to anyone on the machine who runs ps; --ask-password prompts for it instead, and a .netrc file protected with chmod also keeps it out of sight. A browser session can be reused with --load-cookies FILE. --no-check-certificate turns off both the certificate check and the host-name match, which have been on by default since Wget 1.10; for an internal server signed by its own CA, --ca-certificate=FILE keeps verification on.
Questions
Does this page cover Wget2?
No. The rows come from the manual of GNU Wget 1.25.0, the latest 1.x release (November 2024). Wget2 is a separate program with its own manual and is not covered here.
Why was my download saved as file.1?
A file with that name already existed. Without -N, -nc, -r or -p, Wget keeps the original and numbers each new copy (.1, .2 and so on). -nc skips the download instead, -N replaces the file only when the server copy is newer, and -O FILE writes to a name you choose.
How do I download every PDF in one directory of a site?
Recurse one level, stay below the directory and accept a single suffix. The manual shows the pattern for GIFs as wget -r -l1 --no-parent -A.gif http://www.example.com/dir/; use -A.pdf for PDFs. A wildcard in the URL does not work over HTTP.
How do I save one web page with its images and style sheets?
wget -p http://SITE/1.html fetches the page and the files it needs to display. wget -E -H -k -K -p http://SITE/DOCUMENT also takes requisites from other hosts and rewrites the links so the copy opens offline.
Can wget check links without downloading them?
Yes. --spider only checks that each page exists; wget --spider --force-html -i bookmarks.html runs that check over every link in a saved bookmarks file.
What do Wget's exit codes mean?
0 no problems, 1 a generic error, 2 a parse error in options or startup files, 3 a file I/O error, 4 a network failure, 5 SSL verification failure, 6 an authentication failure, 7 a protocol error and 8 an error response from the server. When several occur, the lowest code from 2 upwards is returned.
Sources and checks
- GNU Wget 1.25.0 Manual (Free Software Foundation), Info edition from the wget-1.25.0 release Invoking chapter (option syntax and every option group), Recursive Download, Following Links, Robot Exclusion and the Examples chapter
- GNU Wget 1.25.0 source, src/main.c (the text printed by wget --help) The --no-config option, which the manual only mentions in passing
Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.
Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.