Selenium headless: How to bypass Cloudflare detection using Selenium

Using the latest Google Chrome v96.0 if you retrive the useragent For the google-chrome browser the following user-agent is in use: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36 Where as for google-chrome-headless browser the following user-agent is in use: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/96.0.4664.110 … Read more

How can a program control another program?

I’ve written a bunch of bots at one time or another (from Pogo games to Yohoho Puzzle Pirates). For windows, you’re usually going to either be sending Win32 events to simulate mouse movements, or spoof the actually low-level messages sent between windows when the mouse is actually clicked. A lot of it really depends on … Read more

Pickle: TypeError: a bytes-like object is required, not ‘str’ [duplicate]

You need to open the file in binary mode: file = open(fname, ‘rb’) response = pickle.load(file) file.close() And when writing: file = open(fname, ‘wb’) pickle.dump(response, file) file.close() As an aside, you should use with to handle opening/closing files: When reading: with open(fname, ‘rb’) as file: response = pickle.load(file) And when writing: with open(fname, ‘wb’) as … Read more

Detecting honest web crawlers

You said matching the user agent on ‘bot’ may be awkward, but we’ve found it to be a pretty good match. Our studies have shown that it will cover about 98% of the hits you receive. We also haven’t come across any false positive matches yet either. If you want to raise this up to … Read more