Sites not accepting wget user agent header

It seems Yahoo server does some heuristic based on User-Agent in a case Accept header is set to */*.

Accept: text/html

did the trick for me.

e.g.

wget  --header="Accept: text/html" --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0"  http://yahoo.com

Note: if you don’t declare Accept header then wget automatically adds Accept:*/* which means give me anything you have.

Leave a Comment

Hata!: SQLSTATE[HY000] [1045] Access denied for user 'divattrend_liink'@'localhost' (using password: YES)