Startup Security

Infrastructure security is one of those things that falls prey to the typical ad-hoc growth of startups as it relates to security. You may start out small, but once the project starts growing you need to put a development environment online, as well as a staging environment. These tend to grow organically, with the specific needs of an organization. In addition to that, it’s sometimes necessary to toss the odd server or site online for other testing purposes. What isn’t always considered here, is the availability and accessibility of those resources.

Let’s take the example of a staging site. Frequently made Internet accessible for testing purposes, staging sites are a perfect example of the gap between typical web users and more curious, perhaps malicious ones. While typical users will go to x.com and completely interact with the application there, it is not outside the realm of possibility for a curious individual to try staging.x.com, a well-known scheme for staging sites. If this exists, it could reveal information to a potential attacker through verbose error messages or debug information. Besides simply making this public available, here are a couple other flaws I have seen on live staging sites.

1. Having debug information available

   One startup allowed users browsing to their staging site to set a cookie that showed debug information. This debug information included exact SQL statements that were getting sent to the database. Having this information readily accessible could provide an attacker with the information needed to craft a custom attack.

2. Not locking down all avenues

   After it was publicly disclosed, this same startup restricted access to that staging site using basic authentication. Verifying this out of curiosity, I tried to access it via HTTPS. To my surprise, I was not prompted for authentication.

Another common problem is that of putting a site online in the early stages of a startup. Maybe you need to demo the site to a potential investor or client. Maybe you have outsourced some testing. In any case, the one thing to remember is that if you open a port to the Internet, it’s not private. Port scans, especially for web servers, are extremely common. Combine that with the re-use of IP addresses (*cough*The Cloud*cough*) and your “private” site may be crawled by search engines within the hour. I’m sure most of you are familiar with the statistic that an unpatched XP system only lasts 20 minutes on the net without being found and compromised. If you do restrict access to this, here are a few things to remember.

• Use strong passwords

  When encountered with an authentication prompt of an application that I’ve been hired to review, some of the first things I try are:

  1. admin/admin
  2. guest/guest
  3. demo/demo
  4. Names of, or relating to, the site or application
  5. And a few more…

  You may or may not be surprised with the number of “protected” sites I’ve been able to gain access to by applying this simple logic.

• Use multiple layers, if possible

  A combination of authentication techniques is much preferred to only one. For example, usernames and passwords can be shared and frequently are. IP filters will help limit where access is allowed from and is a very effective method of restricting access.

• Know your external attack surface

  Only by knowing what services or servers you have Internet accessible, can you adequately address the potential risk.

Maintaining infrastructure is not an easy task, but limiting access to exposed services can help limit the risk to an environment. Beyond that, keeping up-to-date on security patches, applying them in a reasonable time frame, and applying common sense security to system administration are all very important. When the environment becomes larger, the toolset necessary to manage these different aspects also becomes important.

Well yesterday’s post was certainly fun and exciting, as the topics of vulnerability and exploitation tend to be. But there’s definitely a lot more for a startup to worry about than Cross-Site Scripting, including various state and government regulation and compliance. This post is largely US-centric, so International readers may want to continue solely out of curiosity.

History

Back in 2003, California Senate Bill 1386 was put into effect. In brief, “it requires an agency, person or business that conducts business in California and owns or licenses computerized ‘personal information’ to disclose any breach of security (to any resident whose unencrypted data is believed to have been disclosed).” [source, sb-1386.com]

Effectively, this means that if you maintain “personal information”, as defined by the bill, of a California state resident in an unencrypted form and there is a breach of security that results in an unauthorized individual gaining access to said data … you must report the breach.

Now this is generally considered to be a good thing as if my social security number gets hijacked, I should be made aware of that.

Present Day

As of September 16, 2008, at least 44 states have breach notification laws. This means a couple things for you, the startup.

1. You need to identify what information you store about your customers
2. You need to identify what data falls under these laws in each state
3. You need to be aware of security breaches
4. You are legally required to report breaches that result in unauthorized access to unencrypted personal data

I’ve talked about it before, but this data identification and classification process should be part of a threat model and documented very clearly. So what is this data? Well it’s different in every state, but a few of the common ones are:

• Social security number
• Driver’s license number
• Passport information
• Credit card number

However, depending on your application, you may have sensitive personal data that you are not intentionally storing.

What Data Are You Storing?

One of the questions I have, and can’t answer completely as I Am Not A Lawyer, is what about services that store my data?

• If I DM somebody my credit card on Twitter and Twitter gets compromised, are they responsible for disclosing that to me?
• If I upload my passport into Evernote for safekeeping and it is compromised, are they responsible for disclosing that to me?

As I read the law, and if I interpret Evernote’s Privacy Policy and Twitter’s Privacy Policy properly, that is the case.

*** Please note that to my knowledge, neither Twitter nor Evernote have had breaches that revealed sensitive information. I’m simply using them as examples of data storage services where my data is not encrypted. ***

How Often Does This Happen?

Well if we look at a listing of data breaches since 2005, there have been a total of 245,201,693 records containing sensitive information involved in security breaches. That’s 81% of the US population, by the raw numbers and simple math.

Tell Me How To Fix It

If you are aware of sensitive information that you are storing as part of your application functionality or perhaps as an administrative function (credit cards for paying members), the best way to mitigate these risks is through the use of encryption. This can take many forms and it can be complex as utilizing the full-disk encryption of various vendors (PGP), or it can be as simple as creating an encrypted disk image on OS X to store the sensitive information.

If users upload data onto your systems, it is likely not feasible to encrypt all of that information. From that perspective, it’s about managing the risk. Performing regular server and network security audits to verify that the proper steps have been taken to secure your network and, in turn, your customer’s data.

A quick note for startups (or anybody) that may be deploying on Google App Engine:

As App Engine does not automatically escape output, you need to do this yourself.

As an example, here’s a very simple snippet:
Welcome, {{ firstname }}!

If “firstname” is not properly sanitized when stored in the database or escaped on output, I could easily make my first name the following:
damon<script>alert('hi!')</script>

And then we would have stored JavaScript code execution, aka Cross-Site Scripting, as the <script> tag would get interpreted by the browser when echoed out.

The solution?

Simple, just |escape your output when coding in Google App Engine:
Welcome, {{ firstname|escape }}!

You can also sanitize data prior to storing it in the database, but as an additional layer it’s good to escape it on output as well.

I’m not sure if this is a derivative of the fact that GAE utilizes Django 0.9.7, but I guess we’ll see when they upgrade to Django 1.0 as it autoescapes all output by default (thank you, Django!).

So as part of my recent Life Reboot, I’ve been getting pretty deeply embedded in the (Seattle) Startup world. One of the fun things is completely changing gears after being embedded in the security world for so long and digging in to a new group of people and technology. This means, of course, that I’ve been paying much more attention to the little startups that crop up all over the place.

And what I’m finding is a little disappointing.

Perhaps I’ve become a little jaded in the past few years as a security consultant. Working with some large organizations where people usually “get” security makes most (yes, there is definitely still a decent level of feedback) conversations surrounding security fairly straightforward. Not only that, but large organizations generally have the benefit of staff that understand security risks and can effectively merge them with the desires of the business to reach a reasonable solution. But let’s take a brief tour over what I’ve come across recently. (Names left out to protect the innocent.)

Exhibit 1 – A random startup (Company A) allows signups on its web page. As with any new service, I’m sure reporters would have liked to have seen who was signing up from what companies, particularly in order to validate Company A’s claims of rampant success and signups. Company A, unfortunately, would display a users’ email address on the confirmation signup page. That confirmation signup page had a parameter that could be incremented up or down to reveal email addresses of anybody that signed up.

Class: Basic design flaw, forced browsing.
Status: Fixed, within a half-hour of reporting.

Exhibit 2 – Another random startup (Company B) has a new service that claims to solve some of your email woes. One of the features of this service is that you can subscribe to RSS feeds of your Inbox. Unauthenticated RSS feeds. Yes, yes, there’s a SHA-256 hash in the URL, but both Google Reader and Bloglines are searchable, not to mention the potential for referral sniffing or just plain information leakage issues.

Class: Design flaw, lack of authentication.
Status: Removed, within a half-hour of reporting. Put back into place using HTTPS. Authentication still not used.

Exhibit 3 – Company C has a sweet iPhone app. They also have a web page that is somewhat integrated into the application. That web page is vulnerable to SQL Injection.

Class: Input validation, SQL Injection.
Status: Reported, twice. No response. Depression ensues.

Exhibit 4 – Company D releases an update to their product and is highly regarded as their service no longer requires the installation of an additional component. Unfortunately, the new feature that permits for that is Cross-Site Scriptable.

Class: Input validation, Cross-Site Scripting.
Status: Reported on October 13. Waiting.

I guess what depresses me is that some of these are pretty basic issues. With all of these problems, I came across them within 5 minutes of using the application. I suppose this post is a plea to the startup’s of the world…

Please, if you aren’t familiar with security, take the time to either talk to somebody that is or bring in a security person for even just half a day to take a look at your product. If nothing else, it’s one less fire that you’ll have to put out. You can even contact me directly if you’re not sure what questions to ask, although I’ll probably put a post up about that at some point.

Granted, the statement above applies to everybody as even Russ McRee frequently points out on his HolisticInfoSec blog, many other organizations struggle with basic security flaws regardless of their size or status.

Many social networking sites these days (most of which I could consider in the startup phase) have the concept of private and public accounts. In an attempt to appease those that may not want their daily activities indexed by the world, they set a simple flag that indicates the status of the account.

This is inherently broken.

Why do I make that statement? Because when there is no enforcement of that “private” data other than a database flag, it should be considered public. Let’s take one of our favorite examples, Twitter. A vulnerability in Twitter was recently announced (and subsequently patched) that revealed protected account’s messages. The flaw here is simply that Twitter didn’t really consider this use case when writing the SQL query and forgot to account for protected accounts with a small SQL statement that probably looked like “AND protected = 1“.

So why is this a problem?

Because the application server always connects to the database server as the same user and it is the responsibility of the developers to not forget to account for protected accounts. Unfortunately, we as humans are fallible and while this practice continues, mistakes will be made.

How can we fix this?

There are several approaches, but I’ll talk about a couple. One relies on process, the other on technology.

1. Threat modeling, data flow and data classification

   This really should be done anyway, but there’s a process known as threat modeling where application developers, business owners and security personnel review an application and identify the data that flows through it, potential avenues of attack and the classification of said data, among various other things. Every application should have this concept of data flow and classification (public vs. private) and verify new code against this model during the development process.

2. Build identity into the data layer

   More mature databases such as Microsoft SQL and Oracle have various concepts of row-level security where options exist to further limit access to data based on a combination of data classification labels and restrictive views. Combined with impersonation, this approach actually relies on the database to enforce privileges, rather than the developer. While I know views and the potential for row-level security exist on MySQL, I don’t think they’re considered or used very often.

The moral of the story is that your data may only be as “private” as the security awareness of the developer coding your SQL queries.

Oh, another topic I should talk about sometime is security through obscurity or other ineffective means. That link I posted to sqlpass.org? That “login” box is simply a CSS overlay that can be removed in about three steps using one of my favorite tools, Firebug.

Update: Somebody asked where they could find more resources about Threat Modeling. Microsoft is actually one of the best places for this and the Application Threat Modeling page and Threat Modeling blog provide hours of endless entertainment. They also have a fantastic Threat Modeling Tool that greatly helps the process. Free, no less! And they are apparently releasing an update soon. I’ve had the pleasure to work with Microsoft in the past and they are very dedicated to the Secure Development Lifecycle. (*I’ll ignore the peanut gallery’s comments about the recent out-of-band patch, but that actually is a good example of how well their process works*)

In light of Michael Santarcangelo (@catalyst) coming to town, I’m putting together a little meet-up in Seattle this coming Monday, October 27. It’s going to be at 7pm at the Owl & Thistle, Seattle location. That’s at 808 Post Ave, Seattle, WA.

Any folk in the security community or interested in chatting about security over a pint are more than welcome to join. Michael will also be bringing signed books.

People often ask me if their web site is secure. Or perhaps what things they should be looking for that may be indicative of security problems. While I can’t answer that first question completely without performing a web application review (and associated legal docs), there are definitely conclusions I can reach based on a few minutes on the site and even a short discussion with the developer(s). Listed below is what I do to get a feel for an application when I’m engaged on a security review.

1. Look and feel

   Honestly, if a web application looks like it was coded using a GeoCities or Frontpage template, chances are pretty good that it was coded by a developer with little experience. What that means, in turn, is that there are likely to be common security issues throughout the site.

2. Error handling

   Somewhat related to the point above, how does the site handle application errors? Does it spit out a verbose error message with a stack trace? Or does it gracefully handle the error message and display a generic message or redirect the user appropriately. Error messages are an attacker’s friend.

3. Parameter Fields

   What kind of parameter fields does the site use, particularly in the URL? If numeric, are the parameters relative to the user or global to the application? Do file names get passed in as parameters? If so, those might susceptible to path traversal issues. Is there a routing protocol on the front-end that limits the input, or do parameters get passed to server-side code to verify format?

4. Invalid user input

   How does the application react when I enter input that is invalid? By invalid, I mean everything from attempting to access data that isn’t mine to putting in a name in a telephone field. Depending on this, I can gauge how much effort was put into validation and sanitization.

5. Reflected user input

   Related to the point above, how does the application display user output. If I type HTML characters in a search that are then displayed on the results page, are they HTML encoded? Django 1.0 automatically takes care of this by autoescaping all output, for example, but other frameworks don’t.

If the developers haven’t thought about these few things, it will be painfully obvious. If they have, I know that it’s going to be more difficult to find problems and they will usually be *logic errors* or *infrastructure issues*.

Welcome, to the Startup Security blog.

This site is intended to become a resource for developers and others involved in a startup organization for information about security. As I’ll post in the coming days, there are a number of concerns I have about integrating security into the development lifecycle. While this is a common trend in large organizations, regardless of the development model used, it is less common in the startup world where development teams may only be composed of a couple people.

In this blog, I’ll post information about problems I’ve come across on live sites, information on secure coding, and talk about tools that can be used to not only help make security easier, but sometimes aid in the debugging of applications as well.

In addition to the technical nature of security, I’ll also discuss some of the regulatory and compliance issues a startup should be aware of. Software as a Service is popular model, but it comes with its own set of risks as well.

So welcome, startup folk. I hope you find this site useful and please let me know if there is anything in particular you would like to see discussed. In the coming days I’ll be updating various things, but it’ll settle down soon.