How to force scrapy to crawl duplicate url?

You’re probably looking for the dont_filter=True argument on Request().
See http://doc.scrapy.org/en/latest/topics/request-response.html#request-objects