Apr-19-2021, 11:18 AM
(Apr-18-2021, 03:48 AM)yoohooos Wrote: Could you guys point out to me what's wrong with my code?You do not find data because data is generated bye JavaScript.
The data that I'm interested in: https://imgur.com/a/mz3bTnr time in the red box
This is a common problem that all faces when start doing some scraping.
A solution is to use Selenium can use it with Scrapy,
or if only want data from this site is easier to just use it alone.
Example.
from selenium import webdriver from selenium.webdriver.chrome.options import Options from time import sleep #--| Setup options = Options() options.add_argument("--headless") options.add_argument("--user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36") #options.add_argument("--window-size=1980,1020") browser = webdriver.Chrome(executable_path=r'C:\cmder\bin\chromedriver.exe', options=options) #--| Parse or automation url = "https://finviz.com/quote.ashx?t=ANPC" browser.get(url) sleep(3) date_1 = browser.find_elements_by_css_selector('#news-table > tbody > tr:nth-child(1) > td:nth-child(1)')[0] date_2 = browser.find_elements_by_css_selector('#news-table > tbody > tr:nth-child(2) > td:nth-child(1)')[0] print(f'{date_1.text.strip()}\n{date_2.text.strip()}')
Output:Apr-16-21 04:15PM
Mar-10-21 07:25AM