+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: how to scrape a website from a keyword list (/thread-31259.html)
how to scrape a website from a keyword list - greenpine - Nov-30-2020
Hello;
I am new to Python,
I am trying to scrape a website using search keywords from a list (text file), loop through each line of keywords until each and every keyword in the file has been searched.
the search block of code works and prints the result.
Here is my code:
driver.get('https://www.website.com/?q=dog%20care')
html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')
a = soup.select('div.class_name a')
for a in soup.select('div.class_name a'):
print(a['title'])
# read file sction:
with open ("keyword_list.txt", "r") as f:
for line in f:
print(line.strip()))the read file block of code works, but I am not sure how to make it work together with the search function. Can anyone help me with this code, please?
RE: how to scrape a website from a keyword list - metulburr - Dec-02-2020
you can just use readlines assuming each keyword is on its own line
with open("keyword_list.txt") as f:
lines = f.readlines().strip()
...
for line in lines:
if line in soup:
print(f"keyword {line} is in website")
RE: how to scrape a website from a keyword list - greenpine - Dec-04-2020
thanks for the reply,
I'll try it.