Jan-28-2021, 03:16 AM
Hello,
I have this problem trying to scrape a website using Beautifulsoup.
I'm trying to find a "span" in multiple "div", but I can't find anything deeper than the very first div
Here's my code
if I call page_soup.body i get this result:
I have this problem trying to scrape a website using Beautifulsoup.
I'm trying to find a "span" in multiple "div", but I can't find anything deeper than the very first div
Here's my code
from urllib.request import urlopen as uReq from bs4 import BeautifulSoup as soup my_url = 'https://www.newegg.com/p/pl?d=graphicscard' uClient = uReq(my_url) page_html = uClient.read() uClient.close() page_soup = soup(page_html, "html.parser") containers = page_soup.findAll("span", {"class":"fs-13"})and here's the result I have in my console
>>> containers = page_soup.findAll("div",{"id":"app"}) >>> len(containers) 1 >>> containers [<div id="app"></div>] >> containers = page_soup.findAll("span",{"class":"fs-11"}) >>> len(containers) 0see the <div id="app"> is the very first div, but there's a whole bunch of stuff in this div. I can see it when I inspect the webpage, but if I try to find the <span class="fs-11"> using the function findAll i get nothing
if I call page_soup.body i get this result:
>>> page_soup.body <body> <div id="app"></div> <div id="modal"></div> <script> if (window.location.port !== '80') window.__env__ = 'dev'; </script> <script> window.appHash = 'b0b815fdc589074946ba'; </script> <script src="https://polyfill.io/v3/polyfill.min.js"></script> <script src="https://cdn.polyfill.io/v......(cut for the sake of brievety)So my question is: How do I scrape a <span> in a website which is embedded in multiple <div>?