Python Forum
Code worked in shell but not when I tried in my project.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Code worked in shell but not when I tried in my project.
#2
(Apr-18-2021, 03:48 AM)yoohooos Wrote: Could you guys point out to me what's wrong with my code?
The data that I'm interested in: https://imgur.com/a/mz3bTnr time in the red box
You do not find data because data is generated bye JavaScript.
This is a common problem that all faces when start doing some scraping.
A solution is to use Selenium can use it with Scrapy,
or if only want data from this site is easier to just use it alone.
Example.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from time import sleep

#--| Setup
options = Options()
options.add_argument("--headless")
options.add_argument("--user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36")
#options.add_argument("--window-size=1980,1020")
browser = webdriver.Chrome(executable_path=r'C:\cmder\bin\chromedriver.exe', options=options)
#--| Parse or automation
url = "https://finviz.com/quote.ashx?t=ANPC"
browser.get(url)
sleep(3)
date_1 = browser.find_elements_by_css_selector('#news-table > tbody > tr:nth-child(1) > td:nth-child(1)')[0]
date_2 = browser.find_elements_by_css_selector('#news-table > tbody > tr:nth-child(2) > td:nth-child(1)')[0]
print(f'{date_1.text.strip()}\n{date_2.text.strip()}')
Output:
Apr-16-21 04:15PM Mar-10-21 07:25AM
Reply


Messages In This Thread
RE: Code worked in shell but not when I tried in my project. - by snippsat - Apr-19-2021, 11:18 AM

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020