Python Forum
Image Scraper (beautifulsoup), stopped working, need to help see why
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Image Scraper (beautifulsoup), stopped working, need to help see why
#2
Try set User-Agent.
Here a quick test.
import requests
from bs4 import BeautifulSoup

url = 'https://archive.4plebs.org/hr/thread/2866456/'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36'}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, 'lxml')
print(soup.find('title').text)
img_test = soup.select_one('div.thread_image_box > a > img')
print(img_test)
print(img_test.get('src'))
Output:
/hr/ - High Resolution ยป Thread #2866456 <img class="thread_image" data-md5="6SrRaBpXnbWSj3IZ9bTr+g==" height="176" loading="lazy" src="https://i.4pcdn.org/hr/1487896415236s.jpg" width="250"/> https://i.4pcdn.org/hr/1487896415236s.jpg
Reply


Messages In This Thread
RE: Image Scraper (beautifulsoup), stopped working, need to help see why - by snippsat - Jan-04-2021, 11:33 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Web scraper tomenzo123 8 4,598 Aug-18-2023, 12:45 PM
Last Post: Gaurav_Kumar
  Web scraper not populating .txt with scraped data BlackHeart 5 1,632 Apr-03-2023, 05:12 PM
Last Post: snippsat
  BeautifulSoup Showing none while extracting image url josephandrew 0 1,994 Sep-20-2021, 11:40 AM
Last Post: josephandrew
  Web scrapping - Stopped working peterjv26 2 3,186 Sep-23-2020, 08:30 AM
Last Post: peterjv26
  not getting image src in my BeautifulSoup csv file farhan275 11 3,920 Sep-14-2020, 04:52 PM
Last Post: buran
  Court Opinion Scraper in Python w/ BS4 (Currently exports to CSV) need help with SQL MidnightDreamer 4 3,123 Mar-12-2020, 09:57 AM
Last Post: BrandonKastning
  Python using BS scraper paulfearn100 1 2,630 Feb-07-2020, 10:22 PM
Last Post: snippsat
  web scraper using pathlib Larz60+ 1 3,274 Oct-16-2017, 05:27 PM
Last Post: Larz60+
  Need alittle hlpl with an image scraper. Blue Dog 8 7,880 Dec-24-2016, 08:09 PM
Last Post: Blue Dog
  Made a very simple email grabber(scraper) Blue Dog 4 7,013 Dec-13-2016, 06:25 AM
Last Post: wavic

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020