HTTP request inside a for loop. - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: HTTP request inside a for loop. (/thread-3244.html) |
HTTP request inside a for loop. - ayanda83 - May-08-2017 I am writing a web scraping script where I have a list of urls to scrap and I iterate through these urls using a for loop. These urls will return the same page structure but with different data hence I am using the same code to scrap all these urls. My problem is that when I run the script, it returns a 404 html error for all the urls in the list and I think its because the for loop is running faster than the 'session.get()' can return the pages. I may be wrong, please advice what you think the problem is. Please see code below. session = requests.session() url_list = list(current_urls_set) if len(url_list) > 0: payload = { 'UserName': 'myemail', 'Password': 'mypassword' } session.post('website url here', data=payload) rfq_dir = 'C:/Projects/TenderBot_Python/Tenders_and_RFQs/cpt/RFQs/{}'.format(datetime.today().strftime('%d-%m-%Y')) if not os.path.exists(rfq_dir): os.mkdir(rfq_dir) for url in url_list: data_list = url.split(',') closing_date = datetime.strptime(data_list[1], '%m/%d/%Y %I:%M:%S %p') if closing_date > datetime.now(): rfqHtml = session.get(websit url here{}'.format(data_list[3].strip())) print(rfqHtml.text) RE: HTTP request inside a for loop. - ayanda83 - May-08-2017 Never mind this, I had the urls wrong. |