Elevated Code RSS Feed

Facebook IFrame applications and the new Open Graph API

Wed, 12 May 2010 06:36:44 GMT

Facebook has long recommended that platform applications are developed using IFrames instead of FBML. Unfortunately, that has typically been a painful endeavor. Facebook typically has passed session information to the user as parameters on the first request to the iframe. The developer was required to store that information in a cookie if they wanted to persist it. Unfortunately, webkit based browsers (notably Safari and Chrome) don't allow cross domain cookies by default. This has caused some painful workarounds. Inside, I'll show you a painless one.

To workaround the cross domain cookie problem, we'll need to get our application out of the iframe. A typical Facebook connect login button looks like:

  <%= fb_login_and_redirect root_url %>

In an IFrame application, this will redirect the IFrame to our site. When this happens, Safari and Chrome will block cookies. Instead, we want to redirect not just the IFrame, but the whole page. We'll use a new helper for that (Facbooker2 only right now)

<% fb_login do %>
  window.top.location = 'http://localhost:3000/'
<% end %>

This gets us to our site without being in an IFrame. As the facebook connect process loads, we'll get the cookies set. If we were to then redirect back to the IFrame version of our page, we would have session information.

While this is functional, it's an odd user experience to have a page outside of facebook. Let's clean that up a little. I'm going to create an action to handle getting the session information. We don't need to do anything in the controller. When a webkit user visits this page, Facebook will be able to set the login cookie. This will call the auth.login event to be fired. We can hook into that to send the user back to the IFrame page.

Our view looks like:

<% fb_connect_async_js2 do %>
FB.Event.subscribe('auth.login', function(response) {
        window.location.href = "http://apps.facebook.com/canvas_url/";
      });
<% end %>

We could add a spinner or some other message to make it look a little nicer, but that's functional. Now, when a webkit user logs in they will be sent to our login specific page where the cookie will be set. Afterwards, they will be redirected back to the canvas url.

There's still one problem. If a user somehow makes it to our login page while already logged in, they will be stuck there forever. We can add just a little bit of our logic to our controller to handle that.

  def create
    redirect_to "http://apps.facebook.com/canvas_url/" if current_facebook_user
  end

Now, if the user is already logged in, we'll just redirect them back immediately. If they aren't, we'll wait for the login callback and will redirect them in the view.

That eliminates one of the major problems plaguing Facebook IFrame applications. Shortly, I'll talk about how to make session handling work for logged in users so that you can use the built in Rails flash and session objects.

Adding a UIActivityIndicator to a UIToolbar

Thu, 29 Apr 2010 07:18:04 GMT

When building apps that fetch data from the network you may want to add an activity indicator to your application's toolbar. Unfortunately, there is no built in activity indicator bar item. Luckily, it's easy to create one in code.

The key is to set the activity indicator as the custom view in a UIBarButtonItem:

    self.loadingIndicator = [[UIActivityIndicatorView alloc] initWithActivityIndicatorStyle:UIActivityIndicatorViewStyleWhite];
    self.activityBarItem.customView = self.loadingIndicator;

That's all there is to it. You can create the bar item in IB and then create the indicator in the controller. Once you've created the indicator, just assign it to the customView property of the UIBarButtonItem. You'll want to set the width of the UIBarButtonItem to 20px in IB. I love simple solutions!

Speeding up delayed job

Wed, 04 Nov 2009 06:44:28 GMT

Delayed Job is a fantastic plugin for moving work out of the request flow. Unfortunately, it has a few performance problems when running with large numbers of jobs on MySQL. With minor changes, we can remove a large amount of the overhead.

I noticed the performance problem on one of our applications when MySQL was taking large amounts of CPU. The application had been in production for about a year and had been performing well. I had just recently moved image resizing out of the main workflow and into delayed job. When I went to look at the slow query log, I saw this being run frequently:


# Query_time: 0  Lock_time: 0  Rows_sent: 5  Rows_examined: 67604
SELECT * FROM `delayed_jobs` WHERE ((run_at <= '2009-11-04 06:58:49' AND (locked_at IS NULL OR locked_at < '2009-11-04 02:58:49') OR (locked_by = 'host:example.com pid:17731')) AND failed_at IS NULL)  ORDER BY priority DESC, run_at ASC LIMIT 5;

When reading the output of the slow query log, I look mainly at two things. The first is the query time. In this case, the query was being run in less than a second. The second thing I look at is the ratio of the number of rows examined to the number of rows sent back to the app. In this case, the database had to look at 67,604 rows just to give the app 5 rows back. When you see full table scans like this, it is usually a sign of a missing index. I looked at the delayed jobs table and saw that there was only an index on ID. That means the DB was reading every row in the table for each query.

Once I realized that I was seeing a full table scan, I knew I needed to add an index. Normally, when you add indexes to a table, you're doing it to speed up the row selection performed by the where clause. That isn't a good idea here. Delayed job removes successfully completed records by default, and we have very few updates that fail. This means that the vast majority of rows in the table will match the where clause. When this happens, you typically want to add an index to speed up the sort instead of the where clause.

This query sorts on two fields, priority and run_at. We can easily add an index there, but notice the ordering on the sort. It sorts by priority in descending order. Because of the way database indexes work, they are typically only used for ascending order sorts. If you were to create an index on delayed_jobs(priority,run_at) the database won't be able to use the index. (You can try it out. Add the index and explain the query.

Some databases, like Oracle, allow you to create an index with a specified ordering. For example, you can create an index on delayed_jobs(priority desc, run_at asc) which will then allow the query to use the index. While that syntax is supported by MySQL, the ordering is ignored. That means MySQL still won't be able to use the index.

So how did I solve the problem? Simple. I changed the semantics of priority. By default, Delayed Job expects a higher value of priority to be more important. I decided to assign lower numbers to mean more important (similar to Unix nice values.) Once I changed the way I assigned priorities, I simply changed the query to order by priority ASC, run_at ASC and added an index on delayed_jobs(priority,run_at). Now, when the query is run, the log looks like:


# Query_time: 0  Lock_time: 0  Rows_sent: 5  Rows_examined: 5
SELECT * FROM `delayed_jobs` WHERE ((run_at <= '2009-11-04 06:58:49' AND (locked_at IS NULL OR locked_at < '2009-11-04 02:58:49') OR (locked_by = 'host:example.com pid:17731')) AND failed_at IS NULL)  ORDER BY priority ASC, run_at ASC LIMIT 5;

That quickly reduced the load on the system and improved application performance.

Slides from Inman Connect 2009

Tue, 11 Aug 2009 06:42:45 GMT

As promised, I wanted to provide my slides from Inman Connect 2009

My favorite links for learning about Agile methodologies are:

There are many more, but those are some of my favorites.

Standing up to be counted

Thu, 30 Apr 2009 09:58:57 GMT

Nick Sieger asked that people stand and be counted and I too will join in. I’m disappointed with many in our community. While I think the presentation that sparked this discussion was in poor taste, it’s not the presentation that bothers me. I’m disappointed by how little empathy some people are showing.

Whether you agree with somebody or not, I think it’s important to try to see things from their perspective. It may not change your mind, but it will hopefully at least give you a better understanding of how your actions affect others.

Speeding up location based searches

Fri, 06 Mar 2009 06:35:51 GMT

With the popularity of Google Maps, more and more web applications are becoming location aware. Rails already has some great tools for working with geodata, such as GeoKit. We've been happily using these tools for years. While GeoKit has some basic optimizations, we recently found a case where lookups were taking close to 8 seconds. We needed a way to speed this up. Read on to learn how we did it.

First, let me explain our domain. We are working on a site that shows the 25 nearest amenities to a location. The application is focused on San Francisco and the surrounding areas. We store the amenities in a table with columns for latitude and longitude. All told, we search about 800,000 amenity records.

To find the nearest 25 amenities, our database has to do a lot of work. For each listing in the database, we need to calculate the distance from our origin. Performing that calculation would be unbearably slow, so geokit speeds up the query by including a square bounds. First, the list of locations is limited by a square 5 miles on a side (we use a big window to make sure we get amenities in more rural areas.) Once we've limited our dataset, the results are sorted by distance. Unfortunately, due to the amenity density in San Francisco, we still have to go through 10,000 records on average.

This calculation and sort is just slow. In version 5.1, MySQL includes some geospatial extensions. These extensions include datatypes for locations and spatial indexes to make spatial searches much faster. Unfortunately, ActiveRecord doesn't support these extensions. That means we'll have to do some things manually.

First, let's look at the three primitives we'll use. The first is a @Point@. Points represent a location in a 2D space. We'll use Points to represent the location of an amenity by it's latitude and longitude coordinates. In MySQL, you can create a point using the SQL: @GeomFromText('Point(38.567 -122.576)')@

The second primitive is a @Polygon@. Polygons include a list of coordinates that make up the vertices. Polygons must be closed. For example, we could create the polygon @GeomFromText('Polygon((0 0,0 1,1 1,1 0,0 0))')@. We'll use a Polygon to represent the boundary inside which we want to search.

The final primitive is a @Line@. We use a Line to calculate the distance between two points. According to the OpenGIS specification, we should be able to use the distance function for this. Unfortunately, MySQL doesn't implement Distance.

We'll also need to create a spatial index. A spatial index is an r-tree index that is optimized for fast distance calculations. Currently, only MyISAM tables support spatial indexes. They also require that the column be defined as NOT NULL. We'll have to keep that in mind as we go along.

Let's start by adding a Point column to our amenities table. Because ActiveRecord doesn't support the GIS extensions, we'll need to use the execute method inside our migration. This looks like:

    execute 'alter table amenities add lat_lng_point point not null'

Because we want to use Spatial indexes, we'll also need to convert our amenities table to MyISAM.

    execute 'alter table amenities engine myisam'

With that in place, we'll need to convert our existing data to include the @lat_lng_point@ column. From what I can tell, Points can only be created using the GeomFromText method. That means we'll need to generate the text representation of a Point during our conversion. This can be done with the following SQL"

update amenities set lat_lng_point = GeomFromText(concat('Point(',lat,' ',lng,')'))

At this point, we now have a table with the location data being stored as a Point. Since all of our records have a Point, we now meet the criteria for adding our spatial index:

    execute 'create spatial index amenities_spatial_idx on amenities(lat_lng_point)'

With that in place, we can need a method that does a few thing. First, we need to compute a bounds to limit our distance calculation. We'll do that using @GeoKit::Bounds@. Once we have that, we need to write a query that uses the @MBRWithin@ function (MBR stands for Minimum Bounding Rectangle) to find the list of points inside our Bounds. Once we have that, we can use a Line to get the distance between our origin and each point and order by that. The end result is a little ugly:

  
  def self.near(mappable)
    bound = GeoKit::Bounds.from_point_and_radius(mappable,5)
    sql = <<SQL
    select *,  GLength(LineStringFromWKB(LineString(AsBinary(lat_lng_point), 
    (AsBinary(GeomFromText('Point(#{mappable.lat} #{mappable.lng})')))))) * 69 as distance
    from amenities where MBRWithin(lat_lng_point,
     GeomFromText('Polygon((#{bound.ne.lat} #{bound.ne.lng}, #{bound.ne.lat} #{bound.sw.lng},#{bound.sw.lat} #{bound.sw.lng},#{bound.sw.lat} #{bound.ne.lng},#{bound.ne.lat} #{bound.ne.lng}))')) 
    order by distance asc limit 25;
SQL
    find_by_sql(sql)
  end

It's ugly, but it is blazingly fast! Our query went from 8s to 0.01s. That's definitely worth a little code ugliness.

While we've handled the inital data conversion and querying, we can't add data to our table. We need to provide a Point for each record, and there's no way to do that using ActiveRecord. Instead, we'll need to use a MySQL trigger.

    execute <<EOT
      create trigger set_lat_lng_point 
      before insert on amenities for each row 
      set  new.lat_lng_point = GeomFromText(concat('Point(',new.lat,' ',new.lng,')'));
EOT

That code will create a Point from the lat and lng when a record is inserted. If you need to be able to update locations, you could make the trigger fire on update as well. You'll want to install the Trigger Happy plugin so that your triggers are dumped and included in your test database.

After all that, can you believe we're still not quite done? Due to a bug in MySQL, inserting a row where @lat_lng_point@ is NULL will give an error. It turns out that MySQL checks the not null columns before triggers are run. Luckily, there is a workaround. If you don't explicitly set the column value to NULL, the trigger will be run before the value is checked. To get ActiveRecord to not explicitly set the column to NULL, we'll use a bit of a hack.

  # Remove lat_lng_point from the list of attributes
  # otherwise, AR writes it as NULL which keeps the trigger from firing
  # and setting the value correctly  
  def before_save
    @attributes.delete("lat_lng_point")
  end

That code will remove the column for AR's list of attributes.

Obviously, this is somewhat complicated. If your searches are running okay using just normal SQL, I wouldn't recommend changing. If you need high performance and are searching large amounts of data, this little bit of code can make a huge difference.

# Our entire migration
class ConvertAmenitiesToUseSpatials < ActiveRecord::Migration
  def self.up
    execute 'alter table amenities add lat_lng_point point not null'
    execute 'alter table amenities engine myisam'
    execute 'update amenities set lat_lng_point = GeomFromText(concat(\'Point(\',lat,\' \',lng,\')\'))'
    execute 'create spatial index amenities_spatial_idx on amenities(lat_lng_point)'
    execute "create trigger set_lat_lng_point before insert on amenities for each row set  new.lat_lng_point = GeomFromText(concat('Point(',new.lat,' ',new.lng,')'));"
  end

  def self.down
    execute 'alter table amenities drop lat_lng_point'
  end
end

Speeding up SOLR indexing

Wed, 14 Jan 2009 12:01:01 GMT

We're finishing up a project that uses SOLR and acts_as_solr extensively. Our dataset isn't large, but it's non-trivial as well. We're managing a search index with a little over 100,000 documents in it. Each document has indexes on 12 fields. In general, our search performance is outstanding. We get results almost instantly. Our indexing performance was another story. I'm not a big fan of premature optimization, but when a full reindex took 4 days, I decided something needed to change

There's some good advice on the web for pulling indexing out of a request flow. Unfortunately for us, this wasn't going to help. Our updates would take several days to process. This delay makes testing changes to search incredibly painful.

After much googling, I found a number of people that recommended allowing SOLR to manage its own commits. It took me a few tries to get this working. In the end, I was making it much harder than it had to be. Disabling autcommit really only takes two steps.

First, update your call to acts_as_solr to disable autcommit:

class MyIndexedModel < ActiveRecord::Base
  acts_as_solr :fields=>[:name,:body], :auto_commit=>false
end

With just that change our index performance went from 2 seconds per record to 50 records per second. A 100x speed up. Unfortunately, our changes weren't showing up in the index. SOLR doesn't add indexed records to the database until a commit is done. SOLR does provide an easy way to tell it to manage the commits itself. To enable this, edit vendor/plugins/acts_as_solr/solr/solr/conf/solrconfig.xml. There is commented out configuration for autoCommit. Remove the XML comments around this area. We use the maxTime parameter to have SOLR update the index every minute. Our configuration looks like:

...
  <updateHandler class="solr.DirectUpdateHandler2">

    <!-- A prefix of "solr." for class names is an alias that
         causes solr to search appropriate packages, including
         org.apache.solr.(search|update|request|core|analysis)
     -->

    <autoCommit>
      <maxDocs>10000</maxDocs>
      <maxTime>60000</maxTime>
    </autoCommit>
...

That's all there is to it! We get 100x faster indexing and our updates still show up within a minute. Suddenly I don't dread making changes to our indexed models!

Announcing Facebooker support for Facebook Connect

Fri, 02 Jan 2009 09:51:20 GMT

Over the last few months I've received quite a few requests for Facebook Connect support in Facebooker. Thanks to Kevin Lochner, I'm proud to announce Facebook Connect support in Facebooker. Read on to learn how to create a simple Facebook Connect application.

The setup of a Facebook Connect application is very similar to that of a Facebook Platform Application. You'll need to create an application using the Facebook Developer tool and then build a Rails application and install the Facebooker plugin. If you need help with this, check out the screencast or My Book. You'll need to set up your facebooker.yml and configure a callback URL on Facebook.

Once you've got a basic application set up and running, we'll need to generate an xd_receiver file. The xd_receiver file allows Facebook to perform cross domain javascript requests. You can generate this file by running:

  script/generate xd_receiver

A file named xd_receiver.html will be generated and placed in your public directory. Make sure this file is added to your source code control system. It must exist for Facebook Connect to work.

With that setup done, it's time to try out Facebook Connect. Before we can do anything else, we'll need to ask our user to login. Let's create a controller and an action for testing. I called my controller "connect" and just used the index action because I'm lazy. Inside the view for action, we'll need to include a javascript file and do a little setup. Add this to your view:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:fb="http://www.facebook.com/2008/fbml">
<head>
  <%= javascript_include_tag :defaults%>
</head>
<body>
<h1>My Great Application</h1>
<%= fb_connect_javascript_tag %>
<%= init_fb_connect "XFBML"%>
<%= fb_login_button%>

</body>
</html>

Facebook connect requires an XHTML doctype declaration along with a namespace on your HTML element. It also works most reliably if your HTML is valid. If you're getting strange results, it's worth validating your HTML. The three calls to helpers are the actual meat of the view. The first call includes the Facebook Connect javascript. The second line runs a little setup and the final line renders an <fb:login-button> tag.

If all goes well, you should see something that looks like:

If you don't see that, you'll need to verify that things are configured just right. There's a lot of magic going on and it tends to be brittle. Make sure that your callback URL is correctly set and that you are accessing your site using the callback URL. Make sure that your public/xd_receiver.html file exists. Finally, make sure that you have correctly set up your facebooker.yml file.

Take a look at the source of your page. Doesn't that <fb:login-button> tag look strange? To render FBML inside your page, Facebook is using into the HTML equivalents. The login button is just one of many tags we can use. For example, we can include a user's profile photo by calling fb_profile_pic. Add a call to show your own profile. Once you've done that, you should see something like: Notice that Facebook isn't showing your picture. That's because of privacy restrictions. Before we can see information about a Facebook user, we'll need to login. Click the "connect" button and you should see a popup prompting you to log in. After logging in, your profile picture will now be shown.

Getting access to FBML outside of the Facebook canvas is cool, but it's just a start to what we can do. When we login using Facebook Connect, Facebook sets several cookies that our application can use to access a Facebook session. Facebooker knows how to use these cookies to set up a facebook session. Add the following code to your controller:

  before_filter :set_facebook_session
  helper_method :facebook_session

That code sets up the users Facebook session and makes it available to the view. With it, you have access to the normal REST API as you can see in the following code.

<% if facebook_session %>
  <h2>You are logged in as <%= facebook_session.user.name %></h2>
<% else %>
  <h2>You are not logged in!</h2>
<% end %>

This means you can ask for data about a user like their name and their friends. You can also post stories to their feed! You have most of the power of a Facebook Platform application inside your existing code!

This support is still very new, so make sure you join the Facebooker mailing list on Rubyforge if you decide to use it. Let us know what you build with it!

Thanks again to Kevin Lochner who provided the patch to support authentication using cookies. Thanks also to Nate Kontny of Inkling Markets who provided a great overview of how Facebook Connect works! Happy New Year everyone!

Daemonizing Rails

Tue, 02 Dec 2008 12:02:54 GMT

No, this isn't a post about us converting from Linux to BSD. Instead, I'm talk about starting long running processes that use the Rails environment. If you've tried to do this, you've probably seen some obscure errors like:

closed stream 
activesupport-2.1.0/lib/active_support/buffered_logger.rb:105:in `write'
activesupport-2.1.0/lib/active_support/buffered_logger.rb:105:in `flush'
activesupport-2.1.0/lib/active_support/buffered_logger.rb:118:in `auto_flush'
activesupport-2.1.0/lib/active_support/buffered_logger.rb:70:in `add'
activesupport-2.1.0/lib/active_support/buffered_logger.rb:76: ...

Inside, I'll show you how to avoid this

Daemons in UNIX are a wonderful, magical thing. Even though it seems like starting up a background job should be simple, there are a lot of steps involved. You can see them in detail in Tammer Saleh's Angels and Daemons talk. The really interesting issue for us is that processes close all open IO streams when they daemonize. That may not cause a problem for a normal Ruby Application, but it violates some assumptions that Rails makes.

When Rails boots, it opens a few different file descriptors. First, it opens a log file. This is your production.log or similar. It also makes a connection to the database. Rails gets unhappy when these files are closed out from under it. You could write a lot of code to handle these conditions or you can take the simple way out: Delay booting Rails until after you daemonize. Here's a simple script that does that:

#!/usr/bin/env ruby
require 'rubygems'
require 'daemons'
ROOT = File.expand_path(File.dirname(__FILE__)+'/../')
Daemons.run_proc('watcher', :dir_mode => :normal, :dir=>"#{ROOT}/log/",:log_output =>true, :backtrace=>true) do
  require "#{ROOT}/config/environment"
  # Do work here
end

This code is simple, but there are a couple of subtleties. The first is that we need to use File.expand_path to turn the Rails Root path into an absolute path. When an application daemonizes, it changes its working directory to "/". If we use a relative path for our RAILS_ROOT it will no longer point to our Rails installation.

Once we have the absolute RAILS_ROOT path, we can daemonize and then boot Rails. Rails will now start as usual and all files will be opened. That's all there is to it. Now you can easily start long running processes using Rails. You can even use a cap task to easily restart them on deploy:

task :restart_watcher do
  run "#{current_path}/script/watcher restart"
end

It feels good to see your name in print

Wed, 22 Oct 2008 08:11:51 GMT

It was almost a full year ago that I started writing a book on Facebook development. This past Friday I got a very exciting box. Unboxing pictures inside.

A nice note on top of the box.

Carefully wrapped

There it is.

I really underestimated the amount of work it would take to write a book. I had read Dave Thomas’ So You Want to Write a Book so I should have realized what an undertaking it was. I had just completed my first marathon at that point and figured it couldn’t be harder than running 26.2 miles. Boy was I ever wrong. I’m proud of the book. I know it is something I would have loved to have when starting to develop Facebook applications.

Now I’m going to get back to training for the Philly Marathon on 11/23. Training seems much easier now.