How to Install Scrapy in Windows

^543DDAB9D1F7B62090D7E854E3A49575E5E9C30402B1E8631F^pimgpsh_fullsize_distr

It took a lot of time for me to install scrapy in my windows pc. I have tried the Installation Guide by scrapy, and tried the tutorials from YouTube and always ended up with having errors. And i tried for weeks installing and uninstalling components, always got different errors. And finally, with lots of research, I successfully installed Scrapy. So, this is how i did it.

Step 1: Install Python 2.7

You can download Python 2.7 from here. Please make sure that you are downloading and installing Python 2.7, because scrapy don’t support the Python 3 versions. But, scrapy is working on making it compatible with Python 3. If you have already installed Python 3, uninstall it before installing Python 2.7.

Python

Now you need to add C:\Python27 and C:\Python27\Scripts to your Path environment variable. To do this open your command prompt and type the following and hit enter:

c:\python27\python.exe c:\python27\tools\scripts\win_add2path.py

To do it manually, go to your system properties, either through My Computer or Control Panel.

Then Select Advanced Settings , then Environment Variables from Advanced Tab.

new2

Then select path from system variables. Then click edit.

new-3

Then copy-paste C:\Python27\;C:\Python27\Scripts; to the end of the path after the semicolon(if there is no semicolon in the end add it before pasting the path.). Then OK button, you have successfully added the path.

new-4

To check whether Python have installed properly, go to command prompt and type python and hit enter. If you have installed it properly you can see a message like in the below image showing the version and other details of Python.

cmd

Note: The versions of the softwares shown in the image will be different in the website (except python 2.7), since constant updating is happening.

Step 2: Install Pip and Setup tools

All the Python versions come after 2.7.9 already have pip and setup tools installed. To check whether pip have installed correctly and to check the version use the following command in cmd.

pip --version


pip

If your pip is outdated, you can update pip using the command:

pip install --upgrade pip

If your python doesn’t have setuptools, download ez_setup.py from here to your python folder. To save the file Right click on the page and then select Save as. Then go to cmd and change the path to python. Then use the following command to install setuptools.

python ez_setup.py

Step 3: Install Microsoft Visual C++ for Python 2.7

Microsoft Visual C++ Compiler for Python 2.7 is really important to install before installing scrapy. Download it from here.

Microsoft Visual C++ Compiler for Python 2

Step 4: Install OpenSSL

For windows, it doesn’t come pre-installed with the os, we have to install it separately. Download the OpenSSL from here to your C:\Python27 folder.

pyOpenSSL 16.0

Right click on the .whl file and select Save link as to download the file to your python folder.

To install OpenSSL, open cmd. Then change the path using
cd C:\Python27

Then use the following command:
python -m pip install pyOpenSSL-16.0.0-py2.py3-none-any.whl

openssl

Once its installed completely, you can see the following message in the end.

Untitled - Notepad_2016-03-19_17-31-05

Step 4: Install lxml

Installing lxml is really important. Download lxml from here. We need to download and install the latest version which supports our Python 2.7. Download the file to your Python 2.7 folder, which is C:\Python27.

lxml

To install the lxml, open cmd and change path to python as we did in previous step. Then use the following command to install lxml.
python -m pip install lxml-3.6.0-cp27-cp27m-win32.whl

Step 5: Install Pywin32

Download Pywin32 from here and install it. Please make sure that you are downloading the right version for the Python installed in your system.

Pywin32

Step 6: Install Scrapy

Finally!  It needs a lot of effort to install scrapy without any errors. After installing all the above components successfully, now, we can install Scrapy. Open a cmd window and type

pip install Scrapy

and hit enter. It will take few mins to complete the installation. And finally we completed installation of scrapy successfully.

Note: When you copy-paste commands to your cmd window, don’t use keyboard shortcuts since it wont work, use mouse to do it.

Advertisements

6 thoughts on “How to Install Scrapy in Windows

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s