Tag: Python

How to setup a Google App Engine Python environment, on Mac OSX, using Homebrew

I want to create Google App Engine apps using Python SDK on my OSX machine: setting up the enviroment should be easy, but I neverthless spent some time putting together all the required pieces. The whole process is composed by four parts: install Homebrew, install Python 2.7 thru brew, install Google App Engine Python SDK and setting up the project environment. Let’s go step by step.

Homebrew

Simple and straightforward, instructions on Homebrew homepage. Xcode Command Line Tools is the only prerequisite.
Personally, I’ve used a different install folder (a different prefix) instead of /usr/local, but it’s just me and some bottles could not work. So the suggestion is to stay with the default prefix. Here the step I used:

xcode-select --install
curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install > ~/brew.rb
sed -i '' 's/HOMEBREW_PREFIX = .*/HOMEBREW_PREFIX = "#{Dir.home}\/tools/homebrew"/' ~/brew.rb
ruby ~/brew.rb

Then I added the Homebrew custom prefix to my PATH adding the following line at the end of ~/.bash_profile

PATH=”$HOME/tools/homebrew/bin:$PATH”

To check that everything works, a call to brew doctor should return just a warning about the custom prefix.

Python

Once Homebrew is in place, install Python is a breeze:

brew install python

Then, I installed the additional tools required to properly work, pip (already installed with Python 2.7, but it may need an update) and virtualenv

pip install -U pip
pip install virtualenv

Google Cloud SDK + App Engine Python

Even if the site suggest to download GoogleAppEngine Launcher, I chose to use Google Cloud SDK, in order to have one single point of access for all the Google Cloud related stuff, instead that different runtimes scattered around my machine:

curl https://sdk.cloud.google.com | bash
exec -l $SHELL
gcloud init

The init command also update ~/.bash_profile, adding auto completion and updating the PATH for the Google Cloud SDK. These two lines

source ~/tools/google-cloud-sdk/path.bash.inc
source ~/tools/google-cloud-sdk/completion.bash.inc

Finally, I added the GAE Python SDK with

gcloud components install app-engine-python

Setup the first GAE project

Time to create the first GAE Python project. Luckily, official documentation has everything to start and succeed, and the section “Download the App Engine SDK for Python” can be skipped because of the Google Cloud SDK already installed.
Real problems arrive when additional Python library are required, like Flask. Following the official steps suggested here has one big problem: when executing the command pip install -t lib, the following error occurs:

DistutilsOptionError: must supply either home or prefix/exec-prefix — not both

It’s a well known bug of Python installed with Homebrew, and the workaround proposed didn’t worked for me. After some searching, I was able to solve the problem using a properly setup virtualenv environment and a symlink. Inside the project folder:

virtualenv env
source env/bin/activate
ln -s env/lib/python2.7/site-packages lib

Then, it’s important to skip the upload of virtualenv environment files, adding the following lines to the app.yaml file. GAE will continue to upload all the required libraries from the lib folder (a symlink to the lib folder inside the virtualenv), so no need of the entire env folder:

skip_files:
- ^env$ #virtual environment's folder

Finally, it’s possible to create the appengine_config.py to enable vendoring and inclusion of libraries in the lib folder, as per the official doc.

from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')

Ahh, time to install some lib, edit files, launch the GAE dev server and start coding!

pip install flask
#(add and edit your files)
dev_appserver.py .

Identify your Twitter followings older that 4 months

Spring is all about cleaning, the saying goes, so why don’t apply the same principle also the the accounts I follow on Twitter? Why? Because I would like to maintain their number under 400 and because I would like to grow my very limited Python skills.

With the help of TweetPony (among the many), the task was pretty straightforward. Final result is a simple script that checks for the people I follow, verifies their last tweet date and alert me if it is older than four months.

Configure the Python environment (Ubuntu 14.04 Trusty)

I don’t want to pollute my system-wide Python installation with libraries and dependencies related to a single project, so I created a virtual environment. Still not a master on that, so forgive my errors:

apt-get install python-pip
sudo pip install virtualenv
cd %projectdir%
virtualenv build_dir
source build_dir/bin/activate

From now ongoing, all the pip commands will be execute inside the (build_dir) virtualdev, and not at system-wide level. Time to install the TweetPony library:

sudo pip install tweetpony

Once installed, I tried some examples from the GitHub repo, to check if it worked. And yes, it did (even without api key and permission, see later), but a boring console message appeared every time the script made a call to Twitter API, caused probably by the old Python 2.7.6 version or libs I was using:

InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning

In order to solve it, I installed some dev libraries required to compile some other Python libraries (again, inside the virtualenv only)

sudo apt-get install libssl-dev
sudo apt-get install libfii-dev
pip install cryptography
pip install pyopenssl ndg-httpsclient pyasn1
pip install urllib3

and added these lines of code at the beginning of the main function of the script, before any Twitter API call:

import urllib3.contrib.pyopenssl
urllib3.contrib.pyopenssl.inject_into_urllib3()

They made the trick! But, as I said, probably you may not need all of these.

The script

The script itself it’s pretty simple. I took the basic code to create the TweetPony API object from the repo’s example folder and I was able to get user’s friends_id (the account the user follows). Then, cycling thru each one, I checked the status of that friend, watching for last tweet date. Some cornercases management (like private tweets or no tweets at all) and voila’, I had all I needed.

Regarding authentication, all Twitter’s libraries require a consumer key and consumer secret to work, in addition to an OAuth access_token and access_token_secret. What made me preferred TweetPony to other libs, like tweepy or python-twitter, was that TweetPony doesn’t required anything. Test consumer key and secret are gently embedded into the lib source, while OAuth tokens are created on the fly for you and persisted over a file, .auth_data.json. To use new credentials, simply delete the file and add somewhere, at the beginning of your code, these two lines, with key and secret obtained from Twitter Dev Console:

tweetpony.CONSUMER_KEY = 'xxxx'
tweetpony.CONSUMER_SECRET = 'xxxxx'

Final consideration about Twitter API usage: there is a limit of 180 calls every 15 minutes, so I added a sleep after every check. Slow, but it worked with my 500+ followers :)
Continue reading “Identify your Twitter followings older that 4 months” →