It took a lot of time for me to install scrapy in my windows pc. I have tried the Installation Guide by scrapy, and tried the tutorials from YouTube and always ended up with having errors. And i tried for weeks installing and uninstalling components, always got different errors. And finally, with lots of research, I successfully installed Scrapy. So, this is how i did it.
Step 1: Install Python 2.7
You can download Python 2.7 from here. Please make sure that you are downloading and installing Python 2.7, because scrapy don’t support the Python 3 versions. But, scrapy is working on making it compatible with Python 3. If you have already installed Python 3, uninstall it before installing Python 2.7.
Now you need to add C:\Python27 and C:\Python27\Scripts to your Path environment variable. To do this open your command prompt and type the following and hit enter:
c:\python27\python.exe c:\python27\tools\scripts\win_add2path.py
To do it manually, go to your system properties, either through My Computer or Control Panel.
Then Select Advanced Settings , then Environment Variables from Advanced Tab.
Then select path from system variables. Then click edit.
Then copy-paste C:\Python27\;C:\Python27\Scripts; to the end of the path after the semicolon(if there is no semicolon in the end add it before pasting the path.). Then OK button, you have successfully added the path.
To check whether Python have installed properly, go to command prompt and type python and hit enter. If you have installed it properly you can see a message like in the below image showing the version and other details of Python.
Note: The versions of the softwares shown in the image will be different in the website (except python 2.7), since constant updating is happening.
Step 2: Install Pip and Setup tools
All the Python versions come after 2.7.9 already have pip and setup tools installed. To check whether pip have installed correctly and to check the version use the following command in cmd.
pip --
version
If your pip is outdated, you can update pip using the command:
pip install --
upgrade pip
If your python doesn’t have setuptools, download ez_setup.py from here to your python folder. To save the file Right click on the page and then select Save as. Then go to cmd and change the path to python. Then use the following command to install setuptools.
python ez_setup.py
Step 3: Install Microsoft Visual C++ for Python 2.7
Microsoft Visual C++ Compiler for Python 2.7 is really important to install before installing scrapy. Download it from here.
Step 4: Install OpenSSL
For windows, it doesn’t come pre-installed with the os, we have to install it separately. Download the OpenSSL from here to your C:\Python27 folder.
Right click on the .whl file and select Save link as to download the file to your python folder.
To install OpenSSL, open cmd. Then change the path using
cd C:\Python27
Then use the following command:
python -m pip install pyOpenSSL-16.0.0-py2.py3-none-any.whl
Once its installed completely, you can see the following message in the end.
Step 4: Install lxml
Installing lxml is really important. Download lxml from here. We need to download and install the latest version which supports our Python 2.7. Download the file to your Python 2.7 folder, which is C:\Python27.
To install the lxml, open cmd and change path to python as we did in previous step. Then use the following command to install lxml.
python -m pip install lxml-3.6.0-cp27-cp27m-win32.whl
Step 5: Install Pywin32
Download Pywin32 from here and install it. Please make sure that you are downloading the right version for the Python installed in your system.
Step 6: Install Scrapy
Finally! It needs a lot of effort to install scrapy without any errors. After installing all the above components successfully, now, we can install Scrapy. Open a cmd window and type
pip install Scrapy
and hit enter. It will take few mins to complete the installation. And finally we completed installation of scrapy successfully.
Note: When you copy-paste commands to your cmd window, don’t use keyboard shortcuts since it wont work, use mouse to do it.
Reblogged this on Simply Python.
LikeLiked by 1 person
Thank you 🙂
LikeLike
Should thank you instead for the very useful post. 🙂
LikeLiked by 1 person
Thank You so much! It helped me a lot! 🙂
LikeLiked by 1 person
Glad it helped you 🙂
LikeLike
Thank you for sharing your experience, it was very helpful for me.
LikeLike
You are heavenly sent!!! Everything worked like a charm after following your easy steps. I did get errors after following other tutorials, including Scrapy’s own installation manual. But yours…just the first trial and after testing using their simple spider example, it worked!!! Can’t thank you enough for this.
LikeLike
WORKED BEAUTIFULLY!!!!!!!!!!
LikeLike
Thank you so much 🙂 .. It helped to install without any efforts 🙂
LikeLike
Thanks You!
LikeLike
hey
thank for this post… but when i gonna install openssl, i got this :
Could not find a version that satisfies the requirement pyOpenSSL-16.0.0-py2.py3-none-any.wh (from versions: )
No matching distribution found for pyOpenSSL-16.0.0-py2.py3-none-any.wh
What does that means ?
thanx a lot
LikeLike
thanks this is very useful for beginners!
LikeLike
Thanks a lot
LikeLike
thank you a lot
LikeLike
Thanks…this was very helpful
LikeLike
Keeping this here so that it may be helpful for someone.
To add Python2.7 installation path to Registry.
Run cmd in administrator mode.
and type :
reg add HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.7\InstallPath /ve /t REG_SZ /d “C:\Python27” /f
LikeLike
lxml can also be installed using executable installer (exe) which can be downloaded from the following link. I’m posting this because I got an error while trying to install lxml using the ‘wheel’ .
https://pypi.python.org/pypi/lxml/3.4.0
hope it helps 🙂
LikeLike
I followed steps A to Z properly, everything worked well and I deployed my first Scrapy project ! 😀
LikeLike
Thanks a lot. Very clear guidance 🙂
LikeLike
Thanks one Billion times. This install would have killed me on my own. I couldn’t even do it on Linux or a Ubuntu Virtual Box. Thank God you were here to show me the windows alternative!!!
LikeLike
Great resource for setting up Scrapy.
Also take note to the file names like “pyOpenSSL-16.0.0-py2.py3-none-any.whl” for example, as it is changed now. Just copy the new file name instead of the one here.
LikeLike
Excelente post!!! Congratulations!!!
LikeLike
thank you for sharing this, was very helpful !!
LikeLike
thank for sharing!
LikeLike
hey, try this:
go to http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
here you will find unofficial wheel file for your python version and 32 or 64 bit variant
install this wheel using:
pip install name_of_downloadwd_wheel_file.whl
now do:
pip install Scrapy
—————————————————————————————————————-
wheel are already compiled files so that you do not need microsoft c++ compiler
hope this will help:)
LikeLike
needed this link for 3.7
https://github.com/mhammond/pywin32/releases
LikeLike
Thank you so much !
LikeLike
python 2.7 is not availbe on the site. whats should i do now
LikeLike
I guess you have to download the latest version of python. This is an old blog, scrappy supports new versions of python now.
LikeLike
It really helped me even in 2019 :). Thumbs up to you
LikeLiked by 1 person
Thank you, thank you, thank you. I have been trying for four days and now I have it working.
LikeLiked by 1 person