Beautiful soup and tags - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: Beautiful soup and tags (/thread-19583.html) Pages:
1
2
|
Beautiful soup and tags - starter_student - Jul-05-2019 Hi everyone, I'm trying to do a loop inside a div tag with others div tags but how to find all the children of that div! <div id ="storelist" class> <ul> <li id ="00021455" class> . . . <div class ="wr-store-details"> <h3> class "Store abc"</h3> <span> class ="20005 dc"> <span> class ="LA"> </ul> </div> #code from urllib.request import urlopen as uReq from bs4 import BeautifulSoup as soup import csv import requests URL = "http:www.abc.com" r = requests.get(URL) soup = BeautifulSoup(r.content, 'html5lib') store_details_tab = [] # a list to store store_details-tab table = soup.find('div', attrs = {'id':'storelist'}) for row in table.findAll('div', attrs = {'class':'store_details'}): store_details = {} store_details['store_name'] = row.div.text store_details['store_address'] = row.span.text store_details['store_city'] = row.span.text store_details_tab.append(store_details) filename = 'store_details_tab.csv' with open(filename, 'wb') as f: w = csv.DictWriter(f,['store_name','store_address','store_city']) w.writeheader() for store_details in store_details_tab: w.writerow(store_details)error: Traceback (most recent call last): File "haagen_store.py", line 4, in <module> import requests ModuleNotFoundError: No module named 'requests' RE: Beautiful soup and tags - perfringo - Jul-05-2019 (Jul-05-2019, 07:10 AM)starter_student Wrote: error: Traceback (most recent call last): This error message states that Python can't find requests module. Have you installed it (it's not built-in module)? RE: Beautiful soup and tags - starter_student - Jul-05-2019 (Jul-05-2019, 07:18 AM)perfringo Wrote:(Jul-05-2019, 07:10 AM)starter_student Wrote: error: Traceback (most recent call last): Hey, I installed both via anaconda soup and request RE: Beautiful soup and tags - perfringo - Jul-05-2019 (Jul-05-2019, 08:46 AM)starter_student Wrote: I installed both via anaconda You can make some easy checks from terminal: pip list - list all packages and their versionspip show requests detailed information, something like this: If there is nothing to show it will return to prompt.
RE: Beautiful soup and tags - Larz60+ - Jul-05-2019 managing multiple versions is a snap with pyenv and virtual environments. With pyenv, you have access to (in code tags for scrolling capability): Available versions: 2.1.3 2.2.3 2.3.7 2.4.0 2.4.1 2.4.2 2.4.3 2.4.4 2.4.5 2.4.6 2.5.0 2.5.1 2.5.2 2.5.3 2.5.4 2.5.5 2.5.6 2.6.6 2.6.7 2.6.8 2.6.9 2.7.0 2.7-dev 2.7.1 2.7.2 2.7.3 2.7.4 2.7.5 2.7.6 2.7.7 2.7.8 2.7.9 2.7.10 2.7.11 2.7.12 2.7.13 2.7.14 2.7.15 2.7.16 3.0.1 3.1.0 3.1.1 3.1.2 3.1.3 3.1.4 3.1.5 3.2.0 3.2.1 3.2.2 3.2.3 3.2.4 3.2.5 3.2.6 3.3.0 3.3.1 3.3.2 3.3.3 3.3.4 3.3.5 3.3.6 3.3.7 3.4.0 3.4-dev 3.4.1 3.4.2 3.4.3 3.4.4 3.4.5 3.4.6 3.4.7 3.4.8 3.4.9 3.4.10 3.5.0 3.5-dev 3.5.1 3.5.2 3.5.3 3.5.4 3.5.5 3.5.6 3.5.7 3.6.0 3.6-dev 3.6.1 3.6.2 3.6.3 3.6.4 3.6.5 3.6.6 3.6.7 3.6.8 3.7.0 3.7-dev 3.7.1 3.7.2 3.7.3 3.8-dev activepython-2.7.14 activepython-3.5.4 activepython-3.6.0 anaconda-1.4.0 anaconda-1.5.0 anaconda-1.5.1 anaconda-1.6.0 anaconda-1.6.1 anaconda-1.7.0 anaconda-1.8.0 anaconda-1.9.0 anaconda-1.9.1 anaconda-1.9.2 anaconda-2.0.0 anaconda-2.0.1 anaconda-2.1.0 anaconda-2.2.0 anaconda-2.3.0 anaconda-2.4.0 anaconda-4.0.0 anaconda2-2.4.0 anaconda2-2.4.1 anaconda2-2.5.0 anaconda2-4.0.0 anaconda2-4.1.0 anaconda2-4.1.1 anaconda2-4.2.0 anaconda2-4.3.0 anaconda2-4.3.1 anaconda2-4.4.0 anaconda2-5.0.0 anaconda2-5.0.1 anaconda2-5.1.0 anaconda2-5.2.0 anaconda2-5.3.0 anaconda2-5.3.1 anaconda2-2018.12 anaconda2-2019.03 anaconda3-2.0.0 anaconda3-2.0.1 anaconda3-2.1.0 anaconda3-2.2.0 anaconda3-2.3.0 anaconda3-2.4.0 anaconda3-2.4.1 anaconda3-2.5.0 anaconda3-4.0.0 anaconda3-4.1.0 anaconda3-4.1.1 anaconda3-4.2.0 anaconda3-4.3.0 anaconda3-4.3.1 anaconda3-4.4.0 anaconda3-5.0.0 anaconda3-5.0.1 anaconda3-5.1.0 anaconda3-5.2.0 anaconda3-5.3.0 anaconda3-5.3.1 anaconda3-2018.12 anaconda3-2019.03 ironpython-dev ironpython-2.7.4 ironpython-2.7.5 ironpython-2.7.6.3 ironpython-2.7.7 jython-dev jython-2.5.0 jython-2.5-dev jython-2.5.1 jython-2.5.2 jython-2.5.3 jython-2.5.4-rc1 jython-2.7.0 jython-2.7.1 micropython-dev micropython-1.9.3 micropython-1.9.4 micropython-1.10 miniconda-latest miniconda-2.2.2 miniconda-3.0.0 miniconda-3.0.4 miniconda-3.0.5 miniconda-3.3.0 miniconda-3.4.2 miniconda-3.7.0 miniconda-3.8.3 miniconda-3.9.1 miniconda-3.10.1 miniconda-3.16.0 miniconda-3.18.3 miniconda2-latest miniconda2-3.18.3 miniconda2-3.19.0 miniconda2-4.0.5 miniconda2-4.1.11 miniconda2-4.3.14 miniconda2-4.3.21 miniconda2-4.3.27 miniconda2-4.3.30 miniconda3-latest miniconda3-2.2.2 miniconda3-3.0.0 miniconda3-3.0.4 miniconda3-3.0.5 miniconda3-3.3.0 miniconda3-3.4.2 miniconda3-3.7.0 miniconda3-3.8.3 miniconda3-3.9.1 miniconda3-3.10.1 miniconda3-3.16.0 miniconda3-3.18.3 miniconda3-3.19.0 miniconda3-4.0.5 miniconda3-4.1.11 miniconda3-4.2.12 miniconda3-4.3.11 miniconda3-4.3.14 miniconda3-4.3.21 miniconda3-4.3.27 miniconda3-4.3.30 pypy-c-jit-latest pypy-c-nojit-latest pypy-dev pypy-stm-2.3 pypy-stm-2.5.1 pypy-1.5-src pypy-1.5 pypy-1.6 pypy-1.7 pypy-1.8 pypy-1.9 pypy-2.0-src pypy-2.0 pypy-2.0.1-src pypy-2.0.1 pypy-2.0.2-src pypy-2.0.2 pypy-2.1-src pypy-2.1 pypy-2.2-src pypy-2.2 pypy-2.2.1-src pypy-2.2.1 pypy-2.3-src pypy-2.3 pypy-2.3.1-src pypy-2.3.1 pypy-2.4.0-src pypy-2.4.0 pypy-2.5.0-src pypy-2.5.0 pypy-2.5.1-src pypy-2.5.1 pypy-2.6.0-src pypy-2.6.0 pypy-2.6.1-src pypy-2.6.1 pypy-4.0.0-src pypy-4.0.0 pypy-4.0.1-src pypy-4.0.1 pypy-5.0.0-src pypy-5.0.0 pypy-5.0.1-src pypy-5.0.1 pypy-5.1-src pypy-5.1 pypy-5.1.1-src pypy-5.1.1 pypy-5.3-src pypy-5.3 pypy-5.3.1-src pypy-5.3.1 pypy-5.4-src pypy-5.4 pypy-5.4.1-src pypy-5.4.1 pypy-5.6.0-src pypy-5.6.0 pypy-5.7.0-src pypy-5.7.0 pypy-5.7.1-src pypy-5.7.1 pypy2-5.3-src pypy2-5.3 pypy2-5.3.1-src pypy2-5.3.1 pypy2-5.4-src pypy2-5.4 pypy2-5.4.1-src pypy2-5.4.1 pypy2-5.6.0-src pypy2-5.6.0 pypy2-5.7.0-src pypy2-5.7.0 pypy2-5.7.1-src pypy2-5.7.1 pypy2.7-5.8.0-src pypy2.7-5.8.0 pypy2.7-5.9.0-src pypy2.7-5.9.0 pypy2.7-5.10.0-src pypy2.7-5.10.0 pypy2.7-6.0.0-src pypy2.7-6.0.0 pypy2.7-7.0.0-src pypy2.7-7.0.0 pypy2.7-7.1.0-src pypy2.7-7.1.0 pypy2.7-7.1.1-src pypy2.7-7.1.1 pypy3-dev pypy3-2.3.1-src pypy3-2.3.1 pypy3-2.4.0-src pypy3-2.4.0 pypy3.3-5.2-alpha1-src pypy3.3-5.2-alpha1 pypy3.3-5.5-alpha-src pypy3.3-5.5-alpha pypy3.5-c-jit-latest pypy3.5-5.7-beta-src pypy3.5-5.7-beta pypy3.5-5.7.1-beta-src pypy3.5-5.7.1-beta pypy3.5-5.8.0-src pypy3.5-5.8.0 pypy3.5-5.9.0-src pypy3.5-5.9.0 pypy3.5-5.10.0-src pypy3.5-5.10.0 pypy3.5-5.10.1-src pypy3.5-5.10.1 pypy3.5-6.0.0-src pypy3.5-6.0.0 pypy3.5-7.0.0-src pypy3.5-7.0.0 pypy3.6-7.0.0-src pypy3.6-7.0.0 pypy3.6-7.1.0-src pypy3.6-7.1.0 pypy3.6-7.1.1-src pypy3.6-7.1.1 pyston-0.5.1 pyston-0.6.0 pyston-0.6.1 stackless-dev stackless-2.7-dev stackless-2.7.2 stackless-2.7.3 stackless-2.7.4 stackless-2.7.5 stackless-2.7.6 stackless-2.7.7 stackless-2.7.8 stackless-2.7.9 stackless-2.7.10 stackless-2.7.11 stackless-2.7.12 stackless-2.7.14 stackless-3.2.2 stackless-3.2.5 stackless-3.3.5 stackless-3.3.7 stackless-3.4-dev stackless-3.4.1 stackless-3.4.2 stackless-3.4.7 stackless-3.5.4If interested, see: https://python-forum.io/Thread-pyenv-Simple-Python-Version-Management?highlight=pyenv RE: Beautiful soup and tags - starter_student - Jul-08-2019 (Jul-05-2019, 09:19 AM)perfringo Wrote:(Jul-05-2019, 08:46 AM)starter_student Wrote: I installed both via anaconda Thks for your response but I checked it and its already installed also the details were shown RE: Beautiful soup and tags - perfringo - Jul-08-2019 (Jul-08-2019, 06:18 AM)starter_student Wrote: Thks for your response but I checked it and its already installed also the details were shown Is the location of requests on your path so that Python can find it? You can check it within Python: >>> import sys >>> sys.path ['/anaconda/bin', '/anaconda/lib/python3.7', '/anaconda/lib/python3.7/lib-dynload', '/anaconda/lib/python3.7/site-packages' /.../]Now check whether requests location is on path. If not, the you should add it to path. This issue can also be related to Python versions (did you install it to correct one) or virtual environment (do you use them?). One can always install request again with command: -m pip install requests which will install it into same Python version which is running in shell.
RE: Beautiful soup and tags - starter_student - Jul-08-2019 (Jul-08-2019, 06:39 AM)perfringo Wrote:(Jul-08-2019, 06:18 AM)starter_student Wrote: Thks for your response but I checked it and its already installed also the details were shown I think i was not importing the module in the right way! Thks but now I have another error TypeError: a bytes-like object is required, not 'str'... can you help me with the code? RE: Beautiful soup and tags - snippsat - Jul-08-2019 (Jul-08-2019, 07:32 AM)starter_student Wrote: but now I have another error TypeError: a bytes-like object is required, not 'str'... can you help me with the code?Next time post whole Traceback ,it will also tell line number that problem occur in this case 39.Change filename, 'wb' to filename, 'w' .
RE: Beautiful soup and tags - starter_student - Jul-08-2019 (Jul-08-2019, 10:15 AM)snippsat Wrote:(Jul-08-2019, 07:32 AM)starter_student Wrote: but now I have another error TypeError: a bytes-like object is required, not 'str'... can you help me with the code?Next time post whole Thks for the advice ... already done ! :) I removed the 'b' and now there is no error but the output file is empty just with headers |