PyDev key bindings
Switching between multiple IDE, i always forget what is the key to do the following in PyDev eclipse..
Find all references ==> Ctrl + Shift + G
Find definition: F3
For full list of PyDev keybindings see here.
Switching between multiple IDE, i always forget what is the key to do the following in PyDev eclipse..
Find all references ==> Ctrl + Shift + G
Find definition: F3
For full list of PyDev keybindings see here.
The following steps assumes that you already have python installed. To make your installation easy the steps recommend using easy_install tool.
Install Scrapy
1. Download and install easy_install tool from cheeseshop .
2. Easy install “zope.interface”
3. Download and install Twisted 10.2.0 from Twisted site
4. Easy install “lxml”
5. Easy install “pyOpenSSL”
6. Easy install “Scrapy”
Test your Scrapy installation
1. Within python interpreter, import the following batteries and there should not be any errors
1. import zope.interface
2. import twisted
3. import lxml
4. import OpenSSL
5. import scrapy
2. If you need to find the version of a particular battery then use module attribute .version or .__version__ as below
print twisted.version ==> [twisted, version 10.2.0]
I found an excellent framework in python for web scraping. It is Scrapy.
I say “excellent” because the architecture is very simple and based off of another proven and reliable framework – Twisted. I love asynchronous programming and so really excited about this framework.
I’m going to play more with it in coming days. Will capture my thoughts and notes here during my journey….
I was using the target parser method of lxml as suggested here and could not turn off fully qualified name for tags and so wrote quick one to split namespace from tag name…
def _gettagns(self,tag):
""" returns a tuple of namespace,name """
if tag[:1] == "{":
return tag[1:].split("}", 1)
else:
return (None,tag)
I liked the virtualenv to sandbox the development environment for django development. Had to write a big instruction for other contributor/s to follow the steps. Then i found this buildout but did’nt like it initially as it was more of declarative approach but it does the job and perfectly. It is very simple as it is declarative. I’m using it now and i can always nuke my local dev and just redo the whole environment setup with just one or two commands. Look here for further information on buildout and how to use it with django.
I take my words back on what I said about IDLE (Integrated DeveLopment Environment) earlier in “Where to Start?”. In the past few days, I felt that using IDLE is much more easier and faster for learning purpose than other IDE. Yeah, the IDLE should really be an acronym of Integrated Developer’s Learning Environment.
Here is a nice online resource for using IDLE.
Now that I’ve decided to learn python, the next steps are to have the python software installed on my machine. Having worked on other languages I’m used to IDEs and so did not like the IDE that comes with python (IDLE) and so I found a nice and free IDE to work with. It is Stani’s Python Editor. Once I feel that I would adopt this language I might well buy some better IDE that is out there in the market. For now I’ll stick with SPE.
Step 1: Download and Install Python.
Just download latest python software and the installation is pretty straight forward. If you still do need help you can use the screencast on downloading and installing. One customization which I do during installation is to install it directly under the root drive like “c:\Python24″. This makes it easier for me to write bat files in future.
Step 2: Download and Install Stani’s Python Editor.
Now I’m packed for the interesting journey ahead…