Python Forum
Webscraping news articles by using selenium
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Webscraping news articles by using selenium
#8
(Aug-28-2023, 07:19 AM)cate16 Wrote: I got the following error:
Now it try to parse.can be a problem with site in your country or speed of search.
Try set time.sleep(3) on line 17,to make it wait a bit before looking for tag.
Sometime the network search can be to fast,time sleep is just a quick first test,Selenium has own Waits .

Here site that works for all,to make sure your setup works.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
import time

# Setup
#https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/116.0.5795.0/win64/chromedriver-win64.zip
options = Options()
options.add_argument("--headless=new")
ser = Service(r"C:\cmder\bin\chromedriver.exe")
browser = webdriver.Chrome(service=ser, options=options)
# Parse or automation
url = 'https://www.python.org/'
browser.get(url)
python_about = browser.find_element(By.CSS_SELECTOR, '#touchnav-wrapper > header > div > div.introduction > p')
print(python_about.text) 
Output:
Python is a programming language that lets you work quickly and integrate systems more effectively. Learn More
Reply


Messages In This Thread
RE: Webscraping news articles by using selenium - by snippsat - Aug-28-2023, 09:58 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Webscraping with beautifulsoup cormanstan 3 2,318 Aug-24-2023, 11:57 AM
Last Post: snippsat
  Webscraping returning empty table Buuuwq 0 1,519 Dec-09-2022, 10:41 AM
Last Post: Buuuwq
  WebScraping using Selenium library Korgik 0 1,138 Dec-09-2022, 09:51 AM
Last Post: Korgik
  How to get rid of numerical tokens in output (webscraping issue)? jps2020 0 2,020 Oct-26-2020, 05:37 PM
Last Post: jps2020
  Python Webscraping with a Login Website warriordazza 0 2,718 Jun-07-2020, 07:04 AM
Last Post: warriordazza
  Help with basic webscraping Captain_Snuggle 2 4,082 Nov-07-2019, 08:07 PM
Last Post: kozaizsvemira
  Can't Resolve Webscraping AttributeError Hass 1 2,395 Jan-15-2019, 09:36 PM
Last Post: nilamo
  How to exclude certain links while webscraping basis on keywords Prince_Bhatia 0 3,330 Oct-31-2018, 07:00 AM
Last Post: Prince_Bhatia
  Webscraping homework Ghigo1995 1 2,730 Sep-23-2018, 07:36 PM
Last Post: nilamo
  Intro to WebScraping d1rjr03 2 3,544 Aug-15-2018, 12:05 AM
Last Post: metulburr

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020