A problem with installing scrapy - Truman - May-21-2019
Error: rial\_dist
creating build\lib.win32-3.6\twisted\trial\_dist\test
copying src\twisted\trial\_dist\test\test_distreporter.py -> build\lib.win32
-3.6\twisted\trial\_dist\test
copying src\twisted\trial\_dist\test\test_disttrial.py -> build\lib.win32-3.
6\twisted\trial\_dist\test
copying src\twisted\trial\_dist\test\test_options.py -> build\lib.win32-3.6\
twisted\trial\_dist\test
copying src\twisted\trial\_dist\test\test_worker.py -> build\lib.win32-3.6\t
wisted\trial\_dist\test
copying src\twisted\trial\_dist\test\test_workerreporter.py -> build\lib.win
32-3.6\twisted\trial\_dist\test
copying src\twisted\trial\_dist\test\test_workertrial.py -> build\lib.win32-
3.6\twisted\trial\_dist\test
copying src\twisted\trial\_dist\test\__init__.py -> build\lib.win32-3.6\twis
ted\trial\_dist\test
creating build\lib.win32-3.6\twisted\web\test
copying src\twisted\web\test\requesthelper.py -> build\lib.win32-3.6\twisted
\web\test
copying src\twisted\web\test\test_agent.py -> build\lib.win32-3.6\twisted\we
b\test
copying src\twisted\web\test\test_cgi.py -> build\lib.win32-3.6\twisted\web\
test
copying src\twisted\web\test\test_client.py -> build\lib.win32-3.6\twisted\w
eb\test
copying src\twisted\web\test\test_distrib.py -> build\lib.win32-3.6\twisted\
web\test
copying src\twisted\web\test\test_domhelpers.py -> build\lib.win32-3.6\twist
ed\web\test
copying src\twisted\web\test\test_error.py -> build\lib.win32-3.6\twisted\we
b\test
copying src\twisted\web\test\test_flatten.py -> build\lib.win32-3.6\twisted\
web\test
copying src\twisted\web\test\test_html.py -> build\lib.win32-3.6\twisted\web
\test
copying src\twisted\web\test\test_http.py -> build\lib.win32-3.6\twisted\web
\test
copying src\twisted\web\test\test_http2.py -> build\lib.win32-3.6\twisted\we
b\test
copying src\twisted\web\test\test_httpauth.py -> build\lib.win32-3.6\twisted
\web\test
copying src\twisted\web\test\test_http_headers.py -> build\lib.win32-3.6\twi
sted\web\test
copying src\twisted\web\test\test_newclient.py -> build\lib.win32-3.6\twiste
d\web\test
copying src\twisted\web\test\test_proxy.py -> build\lib.win32-3.6\twisted\we
b\test
copying src\twisted\web\test\test_resource.py -> build\lib.win32-3.6\twisted
\web\test
copying src\twisted\web\test\test_script.py -> build\lib.win32-3.6\twisted\w
eb\test
copying src\twisted\web\test\test_stan.py -> build\lib.win32-3.6\twisted\web
\test
copying src\twisted\web\test\test_static.py -> build\lib.win32-3.6\twisted\w
eb\test
copying src\twisted\web\test\test_tap.py -> build\lib.win32-3.6\twisted\web\
test
copying src\twisted\web\test\test_template.py -> build\lib.win32-3.6\twisted
\web\test
copying src\twisted\web\test\test_util.py -> build\lib.win32-3.6\twisted\web
\test
copying src\twisted\web\test\test_vhost.py -> build\lib.win32-3.6\twisted\we
b\test
copying src\twisted\web\test\test_web.py -> build\lib.win32-3.6\twisted\web\
test
copying src\twisted\web\test\test_webclient.py -> build\lib.win32-3.6\twiste
d\web\test
copying src\twisted\web\test\test_web__responses.py -> build\lib.win32-3.6\t
wisted\web\test
copying src\twisted\web\test\test_wsgi.py -> build\lib.win32-3.6\twisted\web
\test
copying src\twisted\web\test\test_xml.py -> build\lib.win32-3.6\twisted\web\
test
copying src\twisted\web\test\test_xmlrpc.py -> build\lib.win32-3.6\twisted\w
eb\test
copying src\twisted\web\test\_util.py -> build\lib.win32-3.6\twisted\web\tes
t
copying src\twisted\web\test\__init__.py -> build\lib.win32-3.6\twisted\web\
test
creating build\lib.win32-3.6\twisted\web\_auth
copying src\twisted\web\_auth\basic.py -> build\lib.win32-3.6\twisted\web\_a
uth
copying src\twisted\web\_auth\digest.py -> build\lib.win32-3.6\twisted\web\_
auth
copying src\twisted\web\_auth\wrapper.py -> build\lib.win32-3.6\twisted\web\
_auth
copying src\twisted\web\_auth\__init__.py -> build\lib.win32-3.6\twisted\web
\_auth
creating build\lib.win32-3.6\twisted\words\im
copying src\twisted\words\im\baseaccount.py -> build\lib.win32-3.6\twisted\w
ords\im
copying src\twisted\words\im\basechat.py -> build\lib.win32-3.6\twisted\word
s\im
copying src\twisted\words\im\basesupport.py -> build\lib.win32-3.6\twisted\w
ords\im
copying src\twisted\words\im\interfaces.py -> build\lib.win32-3.6\twisted\wo
rds\im
copying src\twisted\words\im\ircsupport.py -> build\lib.win32-3.6\twisted\wo
rds\im
copying src\twisted\words\im\locals.py -> build\lib.win32-3.6\twisted\words\
im
copying src\twisted\words\im\pbsupport.py -> build\lib.win32-3.6\twisted\wor
ds\im
copying src\twisted\words\im\__init__.py -> build\lib.win32-3.6\twisted\word
s\im
creating build\lib.win32-3.6\twisted\words\protocols
copying src\twisted\words\protocols\irc.py -> build\lib.win32-3.6\twisted\wo
rds\protocols
copying src\twisted\words\protocols\__init__.py -> build\lib.win32-3.6\twist
ed\words\protocols
creating build\lib.win32-3.6\twisted\words\test
copying src\twisted\words\test\test_basechat.py -> build\lib.win32-3.6\twist
ed\words\test
copying src\twisted\words\test\test_basesupport.py -> build\lib.win32-3.6\tw
isted\words\test
copying src\twisted\words\test\test_domish.py -> build\lib.win32-3.6\twisted
\words\test
copying src\twisted\words\test\test_irc.py -> build\lib.win32-3.6\twisted\wo
rds\test
copying src\twisted\words\test\test_ircsupport.py -> build\lib.win32-3.6\twi
sted\words\test
copying src\twisted\words\test\test_irc_service.py -> build\lib.win32-3.6\tw
isted\words\test
copying src\twisted\words\test\test_jabberclient.py -> build\lib.win32-3.6\t
wisted\words\test
copying src\twisted\words\test\test_jabbercomponent.py -> build\lib.win32-3.
6\twisted\words\test
copying src\twisted\words\test\test_jabbererror.py -> build\lib.win32-3.6\tw
isted\words\test
copying src\twisted\words\test\test_jabberjid.py -> build\lib.win32-3.6\twis
ted\words\test
copying src\twisted\words\test\test_jabberjstrports.py -> build\lib.win32-3.
6\twisted\words\test
copying src\twisted\words\test\test_jabbersasl.py -> build\lib.win32-3.6\twi
sted\words\test
copying src\twisted\words\test\test_jabbersaslmechanisms.py -> build\lib.win
32-3.6\twisted\words\test
copying src\twisted\words\test\test_jabberxmlstream.py -> build\lib.win32-3.
6\twisted\words\test
copying src\twisted\words\test\test_jabberxmppstringprep.py -> build\lib.win
32-3.6\twisted\words\test
copying src\twisted\words\test\test_service.py -> build\lib.win32-3.6\twiste
d\words\test
copying src\twisted\words\test\test_tap.py -> build\lib.win32-3.6\twisted\wo
rds\test
copying src\twisted\words\test\test_xishutil.py -> build\lib.win32-3.6\twist
ed\words\test
copying src\twisted\words\test\test_xmlstream.py -> build\lib.win32-3.6\twis
ted\words\test
copying src\twisted\words\test\test_xmpproutertap.py -> build\lib.win32-3.6\
twisted\words\test
copying src\twisted\words\test\test_xpath.py -> build\lib.win32-3.6\twisted\
words\test
copying src\twisted\words\test\__init__.py -> build\lib.win32-3.6\twisted\wo
rds\test
creating build\lib.win32-3.6\twisted\words\xish
copying src\twisted\words\xish\domish.py -> build\lib.win32-3.6\twisted\word
s\xish
copying src\twisted\words\xish\utility.py -> build\lib.win32-3.6\twisted\wor
ds\xish
copying src\twisted\words\xish\xmlstream.py -> build\lib.win32-3.6\twisted\w
ords\xish
copying src\twisted\words\xish\xpath.py -> build\lib.win32-3.6\twisted\words
\xish
copying src\twisted\words\xish\xpathparser.py -> build\lib.win32-3.6\twisted
\words\xish
copying src\twisted\words\xish\__init__.py -> build\lib.win32-3.6\twisted\wo
rds\xish
creating build\lib.win32-3.6\twisted\words\protocols\jabber
copying src\twisted\words\protocols\jabber\client.py -> build\lib.win32-3.6\
twisted\words\protocols\jabber
copying src\twisted\words\protocols\jabber\component.py -> build\lib.win32-3
.6\twisted\words\protocols\jabber
copying src\twisted\words\protocols\jabber\error.py -> build\lib.win32-3.6\t
wisted\words\protocols\jabber
copying src\twisted\words\protocols\jabber\ijabber.py -> build\lib.win32-3.6
\twisted\words\protocols\jabber
copying src\twisted\words\protocols\jabber\jid.py -> build\lib.win32-3.6\twi
sted\words\protocols\jabber
copying src\twisted\words\protocols\jabber\jstrports.py -> build\lib.win32-3
.6\twisted\words\protocols\jabber
copying src\twisted\words\protocols\jabber\sasl.py -> build\lib.win32-3.6\tw
isted\words\protocols\jabber
copying src\twisted\words\protocols\jabber\sasl_mechanisms.py -> build\lib.w
in32-3.6\twisted\words\protocols\jabber
copying src\twisted\words\protocols\jabber\xmlstream.py -> build\lib.win32-3
.6\twisted\words\protocols\jabber
copying src\twisted\words\protocols\jabber\xmpp_stringprep.py -> build\lib.w
in32-3.6\twisted\words\protocols\jabber
copying src\twisted\words\protocols\jabber\__init__.py -> build\lib.win32-3.
6\twisted\words\protocols\jabber
creating build\lib.win32-3.6\twisted\_threads\test
copying src\twisted\_threads\test\test_convenience.py -> build\lib.win32-3.6
\twisted\_threads\test
copying src\twisted\_threads\test\test_memory.py -> build\lib.win32-3.6\twis
ted\_threads\test
copying src\twisted\_threads\test\test_team.py -> build\lib.win32-3.6\twiste
d\_threads\test
copying src\twisted\_threads\test\test_threadworker.py -> build\lib.win32-3.
6\twisted\_threads\test
copying src\twisted\_threads\test\__init__.py -> build\lib.win32-3.6\twisted
\_threads\test
running egg_info
writing src\Twisted.egg-info\PKG-INFO
writing dependency_links to src\Twisted.egg-info\dependency_links.txt
writing entry points to src\Twisted.egg-info\entry_points.txt
writing requirements to src\Twisted.egg-info\requires.txt
writing top-level names to src\Twisted.egg-info\top_level.txt
reading manifest file 'src\Twisted.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*.misc' found under director
y 'src\twisted'
warning: no previously-included files matching '*.bugfix' found under direct
ory 'src\twisted'
warning: no previously-included files matching '*.doc' found under directory
'src\twisted'
warning: no previously-included files matching '*.feature' found under direc
tory 'src\twisted'
warning: no previously-included files matching '*.removal' found under direc
tory 'src\twisted'
warning: no previously-included files matching 'NEWS' found under directory
'src\twisted'
warning: no previously-included files matching 'README' found under director
y 'src\twisted'
warning: no previously-included files matching 'newsfragments' found under d
irectory 'src\twisted'
warning: no previously-included files found matching 'src\twisted\topfiles\C
REDITS'
warning: no previously-included files found matching 'src\twisted\topfiles\C
hangeLog.Old'
warning: no previously-included files found matching 'pyproject.toml'
warning: no previously-included files found matching 'codecov.yml'
warning: no previously-included files found matching 'appveyor.yml'
warning: no previously-included files found matching '.circleci'
warning: no previously-included files matching '*' found under directory '.c
ircleci'
no previously-included directories found matching 'bin'
no previously-included directories found matching 'admin'
no previously-included directories found matching '.travis'
no previously-included directories found matching '.github'
warning: no previously-included files found matching 'docs\historic\2003'
warning: no previously-included files matching '*' found under directory 'do
cs\historic\2003'
writing manifest file 'src\Twisted.egg-info\SOURCES.txt'
copying src\twisted\python\twisted-completion.zsh -> build\lib.win32-3.6\twi
sted\python
creating build\lib.win32-3.6\twisted\python\_pydoctortemplates
copying src\twisted\python\_pydoctortemplates\common.html -> build\lib.win32
-3.6\twisted\python\_pydoctortemplates
copying src\twisted\python\_pydoctortemplates\index.html -> build\lib.win32-
3.6\twisted\python\_pydoctortemplates
copying src\twisted\python\_pydoctortemplates\summary.html -> build\lib.win3
2-3.6\twisted\python\_pydoctortemplates
copying src\twisted\test\cert.pem.no_trailing_newline -> build\lib.win32-3.6
\twisted\test
copying src\twisted\test\key.pem.no_trailing_newline -> build\lib.win32-3.6\
twisted\test
copying src\twisted\test\server.pem -> build\lib.win32-3.6\twisted\test
copying src\twisted\test\test_defer.py.3only -> build\lib.win32-3.6\twisted\
test
copying src\twisted\internet\iocpreactor\notes.txt -> build\lib.win32-3.6\tw
isted\internet\iocpreactor
copying src\twisted\internet\test\_awaittests.py.3only -> build\lib.win32-3.
6\twisted\internet\test
copying src\twisted\internet\test\_yieldfromtests.py.3only -> build\lib.win3
2-3.6\twisted\internet\test
creating build\lib.win32-3.6\twisted\internet\test\fake_CAs
copying src\twisted\internet\test\fake_CAs\chain.pem -> build\lib.win32-3.6\
twisted\internet\test\fake_CAs
copying src\twisted\internet\test\fake_CAs\not-a-certificate -> build\lib.wi
n32-3.6\twisted\internet\test\fake_CAs
copying src\twisted\internet\test\fake_CAs\thing1.pem -> build\lib.win32-3.6
\twisted\internet\test\fake_CAs
copying src\twisted\internet\test\fake_CAs\thing2-duplicate.pem -> build\lib
.win32-3.6\twisted\internet\test\fake_CAs
copying src\twisted\internet\test\fake_CAs\thing2.pem -> build\lib.win32-3.6
\twisted\internet\test\fake_CAs
copying src\twisted\mail\test\rfc822.message -> build\lib.win32-3.6\twisted\
mail\test
copying src\twisted\python\test\_deprecatetests.py.3only -> build\lib.win32-
3.6\twisted\python\test
copying src\twisted\words\im\instancemessenger.glade -> build\lib.win32-3.6\
twisted\words\im
copying src\twisted\words\xish\xpathparser.g -> build\lib.win32-3.6\twisted\
words\xish
running build_ext
building 'twisted.test.raiser' extension
error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual
C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools
----------------------------------------
ERROR: Command "'c:\python36\python.exe' -u -c 'import setuptools, tokenize;__fi
le__='"'"'C:\\Users\\user\\AppData\\Local\\Temp\\pip-install-gcuyud9b\\Twisted\\
setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().
replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'
"'exec'"'"'))' install --record 'C:\Users\user\AppData\Local\Temp\pip-record-6xx
qckqy\install-record.txt' --single-version-externally-managed --compile" failed
with error code 1 in C:\Users\user\AppData\Local\Temp\pip-install-gcuyud9b\Twist
ed\
I found a similar thread but it didn't help me much. Installed C++ 14.0.
Any suggestion?
RE: A problem with installing scrapy - snippsat - May-21-2019
Did you follow my advice in that link?
Also you first download the Twisted wheel.
Then after install Twisted wheel,you do pip install -U scrapy
There there no need to mess with compiler C++ 14.0.
Here a run:
# Test python
C:\Users\Tom\Downloads
λ python -V
Python 3.7.3
# Test pip
C:\Users\Tom\Downloads
λ pip -V
pip 19.1.1 from c:\python37\lib\site-packages\pip (python 3.7)
# Install Twisted wheel
C:\Users\Tom\Downloads
λ pip install Twisted-19.2.0-cp37-cp37m-win32.whl
Processing c:\users\tom\downloads\twisted-19.2.0-cp37-cp37m-win32.whl
Dowloading ....
Successfully installed Twisted-19.2.0
# Now install scrapy
C:\Users\Tom\Downloads
λ pip install -U scrapy
Collecting scrapy
Downloading ....
Successfully installed scrapy-1.6.0
# Test that it work
C:\Users\Tom\Downloads
λ python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import scrapy
>>> scrapy.__version__
'1.6.0'
>>> exit()
RE: A problem with installing scrapy - heiner55 - May-21-2019
You can try https://www.lfd.uci.edu/~gohlke/pythonlibs/.
Then search for scrapy.
RE: A problem with installing scrapy - Truman - May-21-2019
snippsat,
I tried to install Twisted wheel too but got error message.
RE: A problem with installing scrapy - snippsat - May-21-2019
(May-21-2019, 01:42 PM)Truman Wrote: I tried to install Twisted wheel too but got error message. No one here is a mind reader
Type python cmd:
C:\>python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
So i have Python 3.7 and it's 32-bit,then this is wheel i need to use
pip install Twisted-19.2.0-cp37-cp37m-win32.whl You use Python 3.6
Then there are two choices.
Twisted‑19.2.0‑cp36‑cp36m‑win32.whl
Twisted‑19.2.0‑cp36‑cp36m‑win_amd64.whl If you get [MSC v.1916 32 bit (Intel)]
pip install Twisted‑19.2.0‑cp36‑cp36m‑win32.whl If you get [MSC v.1915 64 bit (AMD64)]
pip install Twisted‑19.2.0‑cp36‑cp36m‑win_amd64.whl Basic install tutorial Python 3.6/3.7 and pip installation under Windows
RE: A problem with installing scrapy - Truman - May-21-2019
Error: C:\Python36\kodovi>pip install Twisted-19.2.0-cp36-cp36m-win_amd64.whl
ERROR: Twisted-19.2.0-cp36-cp36m-win_amd64.whl is not a supported wheel on this
platform.
RE: A problem with installing scrapy - snippsat - May-21-2019
Do you have 64-bit Python?
Reads my post again you test in cmd bye typing python .
Do you get [MSC v.1916 32 bit (Intel)] then you have 32-bit Python and install is.
pip install Twisted‑19.2.0‑cp36‑cp36m‑win32.whl
RE: A problem with installing scrapy - Truman - May-21-2019
Thank you. I have windows on 64 bits which confused me.
I installed scrapy too but get this error
Error: Traceback (most recent call last):
File "C:\Python36\kodovi\scrap.py", line 1, in <module>
import scrapy
File "C:\Python36\lib\site-packages\scrapy\__init__.py", line 34, in <module>
from scrapy.spiders import Spider
File "C:\Python36\lib\site-packages\scrapy\spiders\__init__.py", line 10, in <
module>
from scrapy.http import Request
File "C:\Python36\lib\site-packages\scrapy\http\__init__.py", line 8, in <modu
le>
from scrapy.http.headers import Headers
File "C:\Python36\lib\site-packages\scrapy\http\headers.py", line 4, in <modul
e>
from scrapy.utils.python import to_unicode
File "C:\Python36\lib\site-packages\scrapy\utils\python.py", line 14, in <modu
le>
from scrapy.utils.decorators import deprecated
File "C:\Python36\lib\site-packages\scrapy\utils\decorators.py", line 4, in <m
odule>
from twisted.internet import defer, threads
File "C:\Python36\lib\site-packages\twisted\internet\defer.py", line 31, in <m
odule>
from twisted.python import lockfile, failure
File "C:\Python36\lib\site-packages\twisted\python\lockfile.py", line 46, in <
module>
import pywintypes
File "C:\Python36\lib\site-packages\win32\lib\pywintypes.py", line 136, in <mo
dule>
__import_pywin32_system_module__("pywintypes", globals())
File "C:\Python36\lib\site-packages\win32\lib\pywintypes.py", line 39, in __im
port_pywin32_system_module__
for suffix_item in imp.get_suffixes():
AttributeError: module 'imp' has no attribute 'get_suffixes'
when trying to execute code:
import scrapy
from scrapy.crawler import CrawlerProcess
class PythonEventsSpider(scrapy.Spider):
name = 'pythoneventsspider'
start_urls = ['https://www.python.org/events/python-events/',]
found_events = []
def parse(self, response):
for event in response.xpath('//ul[contains(@class, "list-recentevents")]/li'):
event_details = dict()
event_details['name'] = event.xpath('h3[@class="event-title"]/a/text()').extract_first()
event_details['location'] = event.xpath('p/span[@class="event-location"]/text()').extract_first()
event_details['time'] = event.xpath('p/time/text()').extract_first()
self.found_events.append(event_details)
if __name__ == "__main__":
process = CrawlerProcess({ 'LOG_LEVEL': 'ERROR'})
process.crawl(PythonEventsSpider)
spider = next(iter(process.crawlers)).spider
process.start()
for event in spider.found_events:
print(event) As per error message it looks that there are some problems with packages.
Also wondering should I use scrapy at all. This stuff can be done with requests and bs4 in less lines.
RE: A problem with installing scrapy - snippsat - May-22-2019
(May-21-2019, 11:09 PM)Truman Wrote: I installed scrapy too but get this error You should follow tutorial,to make sure that work first.
(May-21-2019, 11:09 PM)Truman Wrote: Also wondering should I use scrapy at all. This stuff can be done with requests and bs4 in less lines. The user case can be very different,Scrapy is the fully fledged Spider library.
Which mean that it eg can follow a start_url and extract data from eg 10 to millions of pages.
start_urls = ['http://alibaba.com/'] alibaba has a lot of product pages,can eg use Srapy to find product and price of product of choice.
It will start and go trough 1000's of pages,and can save data get back in json,csv...ect
Writing the same code with Requests and BS would be a lot of work.
RE: A problem with installing scrapy - Truman - May-23-2019
Ok, will give a try with a tutorial and tell you what happened.
|