This is going to be a long blog post. I've tried to cut it down, but I've been thinking a lot about this topic for over a year and it's hard to walk all the way through this vulnerability without explaining a lot.
I'm not going to talk about vanilla-HTTP-form-post-with-no-secret-token cross site request forgery here. That vulnerability has been about unchanged for a long time, and I'm going to pretty much ignore it- I don't have anything new to say on it at the moment.
In 1995, when I first had web access, websites were pretty basic in comparison to what we have now. To run, say, a search engine, you'd have a handler on a webserver that took user input and wrote out response HTML. It might branch off a new process to do a search, depending on the software, and then it would collect the responses and return them via HTTP to the browser.
Very shortly webpages started getting a little bit more complex. When these first dynamic web pages were made it was very difficult to decouple "backend" software from the pages that it generated. There might be a database behind the web site, and maybe that was on a different server. Most likely, though, the code that wrote to the database was wrapped up in the same binary that generated the HTML for the webpages and processed the inputs and all that was just on one webserver. Everything would be in, perhaps, 1 perl file and a few perl modules.
RPCs, Services, and tiers
So then, "services" started to appear. And front end code, that printed out HTML, was often split out into new binaries, and CSS arrived. So we got some decoupling of "code that writes HTML" from "code that inserts data into the DB" in larger websites. Soon, "code that inserts data into the DB" was moved off to another server entirely. It was the start of "multi tiered" websites.
This is a great way to build complicated websites, and there's so much written about multi tier website architecture that I'm not going to explain it here. We'll just note that the middle tier servers that, perhaps, performed various actions and returned data to the web front ends probably had a bunch of protections in place limiting who could talk to them and involving swanky authentication. And this was generally over internal network connections, entirely within a particular data center. More service layers might be written, caches added and so forth, but the general architecture was the same. So let's say that a site wants to display to a user "here are the groups you belong to".
Probably this was what happened:
1. the user clicks to a URL like "example.com/showmemygroups.cgi", and the browser sends her login cookie with the request
2. the webserver gets the request and determines it's user "annika" from the cookies
3. the webserver software assembles a request object for the groups that user "annika" belongs to and makes a request to a middle tier server over some networking protocol (over an internal network connection, remember)
4. the middle tier server works magic and assembles an object that holds annika's groups, then dispatches the result to the web server software
5. the web server parses the groups object and builds a HTML page to display the data. that is returned to the user's browser via HTTP
But now in the programmable web world, developers don't want to have to make the user go to a new page to see, perhaps, that they joined a cool group on a social networking site. They want to be able to show the user "here are your groups" on the page that they are on right now.
Remember those steps above? Here's what they are now:
2. the request goes not to the webserver software that wrote the HTML but maybe gets forwarded right to the middle tier (potentially via some URL rewriting on the server side or a some other methods)
4. the browser takes the returned JS and writes it to the page that annika is viewing
So what just happened? For one, we changed who can make the RPC calls and who can access the data returned by it. And we changed the format of the request and the format of the response.
Notice how we lean only on cookies for authentication now. This is where I tend to lose people- there are still all those "determine user" steps in there, so it might look more secure. It's not.
An Explanation of what's broken
What happens in that scenario is that calls like <script src="http://othersite.com/myutilities.js"> - what we made above- can be placed on www.example.com -and then calls to its getGroups function will work. And the site is sending back data in a format that can be used on any third party website making the RPC call.
Where most of the security vulnerabilities arise that I've seen is that people don't stop to think about disclosing private data. I've seen a few account change actions disclosed this way, but not as many.
Another clarification here- for private data disclosure bugs, we need to have the return data from othersite.com in a format that the rendering webpage on example.com can read. For account information changing bugs- where hitting a URL will, for example, mark that "I give this group 4 stars!", the return data does NOT need to be readable by the page on example.com.
Can this be fixed? Yes, absolutely. It's just that I've seen a lot of naive implementations of frameworks of this sort. And a lot of people try to fix this and fail.
What is a bad fix? Checking referral headers. There was a way to use flash to break that, and it was fixed (but who knows how many users upgraded their flash to the fixed version). And now there's a new way to do it again... I wouldn't rely on that staying fixed. Using POST only isn't going to cut it either.
What is a good fix? Make sure that you don't return a JS list or other eval-able JS code in step #4 above. Boobytrapping is good. I linked a few days ago to a site which shows good and bad ways to return data from RPCs, but here it is again:
This post is really a work in progress because I'm still trying to figure out how to explain all this in a way that's clear. Criticism and feedback welcomed, my email is on my domain's homepage.