Using the Twitter API without 3rd Party Libraries

October 27, 2015

I've been playing with Twitter's REST API to collect stats for work. Data like: follower counts, what Tweets are being faved, etc… There are libraries for working with the API but no good examples for using it directly. So, I wrote my own1. It's not super complicated, but there are some tricky parts. I'm posting my code here for posterity's sake.

Each example follows the same basic flow outlined in Twitter's Application-only Authentication approach.

  1. Combine the Consumer Key and Consumer Secret (generated when you create an app) to make a Bearer Token.
  2. Base64 encode the Bearer Token (making sure newlines aren't introduced).
  3. Use the Bearer Token to obtain an Access Token.
  4. Use the Access Token to make the actual API call (in this case the basic info for my Twitter account).
  5. Print a dump of the JSON data that's returned.

Step 5 being where real work would actually happen. It's also where looping for to make additional API requests would take place since the Access Token only needs to be pulled once2.

Here's the code:

Ruby

require "base64"
require "json"
require "net/http"
require "uri"

### Setup access credentials

consumer_key = "YOUR_CONSUMER_KEY_STRING"
consumer_secret = "YOUR_CONSUMER_SECRET_STRING"

### Get the Access Token

bearer_token = "#{consumer_key}:#{consumer_secret}"
bearer_token_64 = Base64.strict_encode64(bearer_token)

token_uri = URI("https://api.twitter.com/oauth2/token")
token_https = Net::HTTP.new(token_uri.host,token_uri.port)
token_https.use_ssl = true

token_request = Net::HTTP::Post.new(token_uri)
token_request["Content-Type"] = "application/x-www-form-urlencoded;charset=UTF-8"
token_request["Authorization"] = "Basic #{bearer_token_64}"
token_request.body = "grant_type=client_credentials"

token_response = token_https.request(token_request).body
token_json = JSON.parse(token_response)
access_token = token_json["access_token"]

### Use the Access Token to make an API request

timeline_uri = URI("https://api.twitter.com/1.1/users/show.json?screen_name=TheIdOfAlan")
timeline_https = Net::HTTP.new(timeline_uri.host,timeline_uri.port)
timeline_https.use_ssl = true

timeline_request = Net::HTTP::Get.new(timeline_uri)
timeline_request["Authorization"] = "Bearer #{access_token}"

timeline_response = timeline_https.request(timeline_request).body
timeline_json = JSON.parse(timeline_response)

puts JSON.pretty_generate(timeline_json)

Python

import base64
import json
import urllib2

### Setup access credentials

consumer_key = "YOUR_CONSUMER_KEY_STRING"
consumer_secret = "YOUR_CONSUMER_SECRET_STRING"

### Get the Access Token

bearer_token = "%s:%s" % (consumer_key, consumer_secret)
bearer_token_64 = base64.b64encode(bearer_token)

token_request = urllib2.Request("https://api.twitter.com/oauth2/token") 
token_request.add_header("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8")
token_request.add_header("Authorization", "Basic %s" % bearer_token_64)
token_request.data = "grant_type=client_credentials"

token_response = urllib2.urlopen(token_request)
token_contents = token_response.read()
token_data = json.loads(token_contents)
access_token = token_data["access_token"]

### Use the Access Token to make an API request

timeline_request = urllib2.Request("https://api.twitter.com/1.1/users/show.json?screen_name=TheIdOfAlan")
timeline_request.add_header("Authorization", "Bearer %s" % access_token)

timeline_response = urllib2.urlopen(timeline_request)
timeline_contents = timeline_response.read()
timeline_data = json.loads(timeline_contents)

print json.dumps(timeline_data, indent=2, sort_keys=True)

Perl

use strict;
use Data::Dumper;
use HTTP::Request::Common;
use JSON;
use LWP::UserAgent;
use MIME::Base64;
use Mozilla::CA; # Gets HTTPS working on Mac OSX (10.10)

### Setup access credentials

my $consumer_key = "YOUR_CONSUMER_KEY_STRING";
my $consumer_secret = "YOUR_CONSUMER_SECRET_STRING";

### Get the Access Token

my $bearer_token = "$consumer_key:$consumer_secret";
my $bearer_token_64 = encode_base64($bearer_token, "");

my $user_agent = LWP::UserAgent->new;

my $token_request = POST(
  "https://api.twitter.com/oauth2/token",
  "Content-Type" => "application/x-www-form-urlencoded;charset=UTF-8",
  "Authorization" => "Basic $bearer_token_64",
  Content => { "grant_type" => "client_credentials" },
);

my $token_response = $user_agent->request($token_request);
my $token_json = decode_json($token_response->content);

my $timeline_request = GET(
  "https://api.twitter.com/1.1/users/show.json?screen_name=TheIdOfAlan",
  "Authorization" => "Bearer " . $token_json->{access_token}
);

my $timeline_response = $user_agent->request($timeline_request);
my $timeline_json = decode_json($timeline_response->content);

print Dumper $timeline_json;

A few things to keep in mind with these examples:

  • Authentication credentials should not be hard coded in source files.
  • They are very high level. Just one step beyond pseudo-code.
  • The code is straight procedural and almost certainly not how you should actually do it.
  • There is no error handling in these examples.
  • There is no accounting for throttling based on the API rate-limits.
  • No step to transform strings into RFC 1738 (which the docs say to do) was added since there is no change at this point.
  • Probably some other caveats that I'm forgetting.
  • Authentication credentials should not be hard coded in source files (in case you missed it the first time).

Generally, I'm all for using code libraries to make life easier. Working with APIs is a different story. Unless whoever makes the API is also providing the library, I'd rather use the service directly. Avoiding third-party dependencies is well worth the little extra work it takes. And, if there are good examples available there's not a huge difference between the levels of effort.


Footnotes

  1. The person I'm helping with this is interested in using the pandas data analysis library which is written in Python. I wrote the first version in that language. I decided to port it to Ruby since that's my go-to language these days. Once I started down the path of multiple versions it was easy to thrown in Perl (my previous language of choice). This was the first time in years that I've written any Perl. It's also the first time I've ever done anything beyond "Hello, World" in Python. All said, it was a good language exercise. It's fun to see the difference between them.

  2. The Access Token only needs to be pulled once for as long as it's valid. It should stay that way unless a specific request to invalidate it occurs.


Configuration Files and Multiple Ruby Object Constructors

October 12, 2015

I mostly write command line apps. Mostly. It's the nature of the gig, even though I work on a relatively big website1. There are lots of moving parts, many of which are well suited for automation, that combine to produce the overall site. As someone who likes avoiding hard-coded variables, I use configuration files to tell my various apps/scripts/services/software what to do.

The simplest way to instantiate a Ruby object with parameters (like a config file path) is to send them directly to the initialize method. For example2:

class Robot

  attr_reader :config

  def initialize(config_path)
    @config = parse_config(config_path)
  end
  
  def parse_config(config_path)
    YAML.load(ERB.new(File.read(config_path)).result)
  end
end

robot = Robot.new("~/robot-config.yml"))

That approach works but makes testing problematic. Unit tests for individual method can't be run without loading a config file. An undesirable tight coupling is created. If the object validates the config, changes to its format can trigger multiple, otherwise unrelated tests. A strong "Shotgun Surgery" Code Smell3 indicating a great refactoring opportunity.

My first attempt at a remedy was to make the config file optional. For example:

class Robot

  attr_reader :config

  def initialize(config_path = nil)
    if config_path
      @config = parse_config(config_path)
    else 
      @config = {}
    end
  end
  
  def parse_config(config_path)
    YAML.load(ERB.new(File.read(config_path)).result)
  end
end

robot = Robot.new("~/robot-config.yml"))

robot_for_unit_tests = Robot.new

That works, but feels rough. Conditional logic has become a red flag ever since attending Sandi Metz's wonderful Practical Object-Oriented Development (POOD) course4. Also, having recently kicked around with The Big Nerd Ranch's Objective-C book5, I'd just seen examples of using multiple object constructors. After some investigation6, experiments, and hacking, I ended up with this approach:

class Robot

  attr_reader :config

  def initialize
    @config = {} # defaults can be applied here too.
  end
  
  def initialize_with_config config_path
    initialize
    @config = parse_config(config_path)
  end
  
  def self.new_with_config config_path
    forerunner = allocate
    forerunner.send(:initialize_with_config, config_path)
    forerunner  
  end
  
  def parse_config(config_path)
    YAML.load(ERB.new(File.read(config_path)).result)
  end
end

The primary way to instantiate objects changes from:

robot = Robot.new("~/robot-config.yml"))

To use the newly created class method:

robot = Robot.new_with_config("~/robot-config.yml"))

The .new_with_config class method bounces to the .initialize_with_config object method which in turn calls the default .initialize object method before doing its work.

Unit tests can now use the built-in .new() without params or worrying about a config file. An added bonus is the nice way this separates setting defaults (and any other initialization requirements) from loading the config.

This approach isn't limited to config files. It works any time there's a need to create objects with different parameters/options from a single class. The few extra extra lines for the class methods are totally worth the clean separation provided by multiple constructors.

If you're interested in more details on how this works, check out Ruby Constructors from @terminalbreaker.


Footnotes

  1. Well, I really work on a few big sites: PGATOUR.COM, PresidentsCup.com, and WorldGolfChampionships.com but they are all directly related and powered by the same tools.

  2. The examples in this post contain working code with two caveats. First, they need require 'erb' and require 'yaml'. Those lines were omitted to save a little vertical height. And, of course, a config file sitting at ~/robot-config.yml. I also used a minimal approach with no error handling to keep the size down. So, while they work, they are only proof-of-concept examples.

  3. The Wikipedia Shotgun Surgery is a pretty formal. The simplified way I've heard it described and think about it is: An update in one place that requires modifying multiple classes/methods/functions in other places.

  4. I can't recommend Sandi's Practical Object-Oriented Design Course course highly enough. It was mind-expanding. The huge focus on "Practical" is hard to beat. (And yes, that's the back of my head with the WWSMD neck-tattoo.)

  5. I haven't read any other Objective-C books. So, I don't have a basis for comparison, but Objective-C Programming: The Big Nerd Ranch Guide seems like a decent intro. More than anything, it makes me want to attend a course at the ranch.

  6. This was a rare case where I wasn't able to score a hit on StackOverflow. I found the solution in Ruby Constructors and the related Constructors, Intro to Ruby Classes Part II slides. Both by Juan '@terminalbreaker' Leal.

P.S. Yes, I know making two calls works just fine. Something like:

robot = Robot.new
robot.prase_config("~/robot-config.yml"))

But since production scripts require the config, I prefer to make it happen in one line.


The Density of Kanji

September 04, 2015

Japanese writing, called Kanji, is beautiful. It's a "logographic writing system1". A fancy way of saying the written symbols primarily represent words (or parts of words) instead of sounds. That approach allows it to be much denser than English. Something I realized after picking up a Japanese Twitter follower.

Take this Tweet2 for example:

最近有科学家研究发现,人类最理想的身高是168厘米,上下变动的范围可在167厘米至170厘米之间。研究指出,身材高大的人血液循环路线较长,心脏负担也较重,因此可能会对寿命有影响。矮个子体表面积相对较小,日常的能量消耗较少,所需营养物质相应减少,身体的耐受力较强。

It's 131 characters long. Nine shy of the Twitter's limit. After running it through Google translation engine it becomes:

Recently, scientists found that the human ideal height is 168 cm, the upper and lower range of variation can be between 167-170 cm. Research indicates that tall people the blood circulation route is longer, it is also heavier burden on the heart, and therefore might affect life expectancy. Shorty body surface area is relatively small, less daily energy consumption, a corresponding reduction in the required nutrients, strong physical endurance.

That's 447 characters. More than 3x the length. I doubt the most efficient human translator/editor could get the same information across in English while staying under Twitter's 140 character limit.

It blew my mind a little to figure this out.


Footnotes

  1. Here's the Wikipedia section on logographic writing systems I used when writing this post.

  2. And here's the original Tweet that got me thinking about all this.


Twitter Feature Request: Show Video Titles and Times

August 31, 2015

Dear Twitter,

I have a feature request for video links. The current "View media" feature is nice.

The in-line, expanded card it opens is great.

But, I'd like be to see the video's name and it's length without having to click/tap anything. Please consider replacing the URL with the title and run time in the initial display. For example, instead of:

Maybe something like:

Thanks,

-a


Reading and Writing Files with SQLite

July 21, 2015

The documentation for SQLite isn't super clear on how to work with external files directly from the command line. Here's how to read and process SQL from an input file:

sqlite3 database.db < input.sql

And this captures the output to a file as well:

sqlite3 database.db < query.sql > results.txt

Super useful for automating work without having to jump into SQLite itself.


(Alternative approaches are cat input.sql | sqlite3 database.db and cat query.sql | sqlite3 database.db > results.txt, respectively, but they aren't as aesthetically pleasing.)


Fifteen (plus) Years Building This Website

July 13, 2015

Today, this site is officially fifteen years old.

Crazy as that seems, it's actually even older. I started it in the late '90s. The records database doesn't go back that far. So, I don't know exactly when. I tried using the Internet Archive's Wayback Machine to figure it out. No help there. The first snapshot wasn't until July 18, 2001.

I still get a kick out of that iteration of the home page. It refreshed every four seconds. Each time presenting a bunch of "all work and no play makes jack a dull boy." sentences in different layouts. An ode to The Shining.

That page was hand built in language from Microsoft called ASP (for Active Server Pages). Every time someone loaded it, the server would pick a random selection from the dozen or so different "all work…" layouts and display it. Nothing fancy. Just a fun little experiment.

In September 2005 I started blogging at Blogger.com. Within two months I decided I was going to keep at it and migrated everything here. The brief post I wrote about the move had this line:

I like the idea of having my stuff be mine and not under someone else's control.

Could've used some editorial help on that one but the sentiment holds. After a decade and a half working on a personal site, I can highly recommend it for everyone. And, thanks to the march of technology, it's easy for anyone to do it for free.

If you're not a tech person, go to Wordpress.com and click the "Create Website" button. You'll be up and running in a few minutes. Over time, you'll be much happier working on your own thing than something in someone else's social network.


Notes

  • Wordpress is "Open Source Software". That means you can take your site off "Wordpress.com" and put it somewhere else.
  • Wordpress is a great choice for tech folks too. It gets you up and running and is easy to migrate away from if you find another platform to use later.

Name dot Whatever Websites are Available

July 12, 2015

The last part of a website's name (e.g. the ".com", ".edu", or ".uk") is called the Top-level Domain. Each country gets their own (e.g. ".au", ".uk", ".us"). There also used to be seven other "generic" possibilities:

  1. .com (commercial)
  2. .org (organization)
  3. .net (network)
  4. .int (international organizations)
  5. .edu (U.S. education institutions)
  6. .gov (U.S. government)
  7. .mil (U.S. military)

That list has been growing in recent years. It's up to 810 at last count1. This makes it harder for cybersquatters2 to snatch up all the good ones. There are simply too many possibilities to make it cost effective. For example, here's the current list of available my name dot whatever:

Site Domain Name Price
alan.accountants $73.98
alan.adult $80.00
alan.aero $79.81
alan.alsace $39.86
alan.apartments $47.81
alan.associates $24.60
alan.auction $24.60
alan.bar $55.50
alan.bargains $24.60
alan.beer $24.37
alan.bid $19.24
alan.bingo $41.81
alan.bio $47.26
alan.black $34.85
alan.boutique $24.60
alan.brussels $31.81
alan.build $55.50
alan.builders $24.60
alan.cab $24.60
alan.camp $24.60
alan.capetown $23.09
alan.cards $24.60
alan.care $24.60
alan.careers $37.99
alan.casino $136.81
alan.chat $27.81
alan.cheap $24.60
alan.church $24.60
alan.city $17.39
alan.claims $37.99
alan.clinic $37.99
alan.coach $37.99
alan.cologne $11.54
alan.community $24.60
alan.condos $37.99
alan.contractors $24.60
alan.cool $24.60
alan.coop $119.81
alan.credit $73.98
alan.creditcard $109.97
alan.cricket $55.50
alan.cruises $37.99
alan.cymru $16.72
alan.dating $37.99
alan.deals $24.60
alan.delivery $37.99
alan.dental $37.99
alan.desi $16.36
alan.diamonds $37.99
alan.direct $24.60
alan.directory $17.39
alan.discount $24.60
alan.domains $24.60
alan.durban $23.09
alan.energy $73.98
alan.enterprises $24.60
alan.equipment $17.39
alan.estate $24.60
alan.exchange $24.60
alan.expert $37.99
alan.fail $24.60
alan.farm $24.60
alan.financial $37.99
alan.fish $24.60
alan.fitness $24.60
alan.flights $37.99
alan.football $16.81
alan.forsale $24.60
alan.foundation $24.60
alan.frl $33.68
alan.fund $37.99
alan.furniture $37.99
alan.gifts $24.60
alan.glass $24.60
alan.golf $54.81
alan.gratis $17.39
alan.green $55.50
alan.guide $24.60
alan.guitars $24.60
alan.hamburg $38.83
alan.haus $24.60
alan.healthcare $37.99
alan.holdings $37.99
alan.holiday $37.99
alan.host $70.95
alan.immo $24.60
alan.immobilien $24.60
alan.industries $24.60
alan.institute $17.39
alan.insure $37.99
alan.investments $71.98
alan.jetzt $17.39
alan.joburg $23.09
alan.juegos $13.61
alan.kaufen $24.60
alan.koeln $11.54
alan.land $24.60
alan.lease $37.99
alan.legal $37.99
alan.lgbt $34.85
alan.life $24.60
alan.lighting $17.39
alan.limited $24.60
alan.limo $37.99
alan.loans $73.98
alan.ltda $32.84
alan.luxury $480.00
alan.market $14.60
alan.marketing $24.60
alan.media $24.60
alan.memorial $37.99
alan.menu $29.75
alan.moda $19.45
alan.money $24.60
alan.nagoya $13.01
alan.ngo $64.81
alan.one $11.81
alan.partners $37.99
alan.party $24.39
alan.physio $65.80
alan.pink $14.25
alan.pizza $37.99
alan.place $24.60
alan.plumbing $24.60
alan.plus $32.81
alan.poker $34.85
alan.porn $80.00
alan.pro $14.30
alan.properties $24.60
alan.quebec $27.61
alan.recipes $37.99
alan.red $14.25
alan.reisen $17.39
alan.rentals $24.60
alan.rest $29.75
alan.restaurant $37.99
alan.reviews $19.45
alan.ruhr $33.89
alan.sale $24.60
alan.sarl $24.60
alan.school $26.81
alan.singles $24.60
alan.solar $24.60
alan.supplies $17.39
alan.supply $17.39
alan.surgery $37.99
alan.tattoo $24.60
alan.tax $37.99
alan.technology $17.39
alan.tennis $41.81
alan.tienda $37.99
alan.tires $73.98
alan.tokyo $13.01
alan.tools $24.60
alan.tours $54.81
alan.town $24.60
alan.trade $24.60
alan.training $24.60
alan.travel $110.00
alan.university $37.99
alan.vacations $24.60
alan.ventures $37.99
alan.viajes $37.99
alan.vlaanderen $31.81
alan.vote $55.50
alan.voto $55.50
alan.voyage $37.99
alan.wedding $24.60
alan.wien $31.11
alan.xxx $80.00
alan.yoga $24.60
alan.yokohama $13.01
alan.zone $24.60

Those prices are per year. For comparison, I pay $14 per for this site (alanwsmith.com).

Historically, having ".com" at the end of a website was important for marketing. My guess it will continue to be for sometime but that'll diminish to being a wash over time.


(Note: The number of top-level domains for the F-word is surprisingly high. They include: .associates, .church, .education, .expert, .golf, .guru, .management, .partners, .photography, .systems, .university.)


Footnotes

  1. The current list of Top-level domains.

  2. From the Wikipedia: Cybersquatting (also known as domain squatting), according to the United States federal law known as the Anticybersquatting Consumer Protection Act, is registering, trafficking in, or using an Internet domain name with bad faith intent to profit from the goodwill of a trademark belonging to someone else. The cybersquatter then offers to sell the domain to the person or company who owns a trademark contained within the name at an inflated price.


Don't use PVC for Fireworks Launch Tubes

July 03, 2015

Once you discover mortar fireworks, nothing else will do.

The fireworks I grew up with were self-contained little packages. You'd light a fuse on a box or stick, it would do its thing, then be tossed. Mortars are different. A basic package contains a reusable launch tube and several shells.

A basic cardboard launch tube and eight double-break shells. The "double break" part means each shell produces two colorful bursts in the sky.

Drop a shell in the tube, light fuse and get away. Then, do it again. They go way higher, look better and make a bigger bang than other fireworks. They're really just smaller scale version of the stuff the pros use.

Industrious pyros line up multiple tubes, run fuse between them and launch several shells at once. Even more industrious pyros use pipe to make their own tubes for more simultaneous launches. This is where things can go sideways.

The tubes in fireworks packages are generally dense cardboard or fiberglass. It's easy to think that if cardboard would work, PVC from your local hardware store would too. Don't make this mistake. If something goes wrong and the shell explodes in the tube, shrapnel flies. Here's what that looks like:

Instead of PVC, professionals recommend High-density polyethylene (HDPE). It's much less likely to maim or kill someone if something goes wrong. Check out the difference:

A final note, don't be fooled by "ABS" pipe at the hardware store. It looks like HDPE but it's different and also unsafe. You're best bet is search online. (The top search result that came up me was PyroBoom which looks like a great place to start.)

Be safe out there and have a happy 4th of July.


Go To Index Page: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107


© Alan W. Smith
RSS Feed