Photosynthesis

We’re hiring all across our engineering team. The descriptions below should give you a feel for what we’re looking for, but are not limited. If you feel you have skills to contribute and are interested in our team, please apply to grow@sproutsocial.com!

Web Developer

Sprout Social’s Web Team builds and maintains our front-end web software, used 24/7/365 globally by names like Yammer, Twilio, Groupon and Fender. We’re looking for smart, engaging, supportive people who have a passion for social media and technology to join our Chicago team. If you are the best at what you do and ready to immerse yourself in one of Chicago’s hottest startups, apply now with your best code samples.

Skills & Requirements

• You consider yourself a “Javascript programmer” before a “web developer”.
• You proactively stay at the forefront of HTML5/CSS3 and want to use every Modernizr test you can.
• You think about performance and design implications with every line of code you write.
• Unit tests keep you sane (the more the better!).
• You have solid software design skills.
• You want to work in a small, agile team with fast projects and tight deadlines.
• You are very hard working and self-sustaining.
• You like variety in your projects.
• You deploy bulletproof code with time to spare.

Platform Software Engineer

Big Data. Distributed Systems. NoSQL. Do we have your attention? The Platform Team at Sprout Social is responsible for building and maintaining back-end software that powers our products and shapes our industry. Weekly we serve and process millions of requests, 10’s of millions of records and 100’s of GBs of data; and we’re growing fast! We’re looking for smart, engaging, supportive people who have a passion for social media and technology, to join our Chicago team. If you are the best at what you do and ready to immerse yourself in one of Chicago’s hottest startups, please apply with code samples.

Skills & Requirements

• You have a long history with complex languages (Java, C, C++ or other low-level languages). 
• You are a passionate systems programmer (threading, I/O, distributed systems, etc.). 
• You’re excited to work with tech like: Cassandra, MySQL, Redis, Kafka, Solr, Elasticsearch, Java, Python. 
• You have experience or interest in scaling techniques. 
• You communicate collaboratively, clearly and frequently. 
• You are self-sufficient and comfortable in a rapidly changing environment with short project cycles. 
• You love creating elegant solutions to complex problems.

Android Engineer

The Mobile Team at Sprout Social is responsible for building and maintaining awesome tools that allow businesses to manage their social media on the go and to work in concert with our industry-leading web app products.

Problems we work on:

• Design: Implement pixel perfect designs that exceed use expectations for Android.
• Engineer: Develop reliable software that makes people’s lives easier.
• Test: Ensure our code works flawlessly across the Android platform.
• Innovate: Push our platform forward by being involved in product discussions and decisions.
• Standardize: Work with other mobile and web engineers to create a consistent environment for our users.

Skills & Requirements

• You love Android.  You have an Android phone, and you love telling people why it’s better than their iPhone.
• You love pixels.  Poor Android device scaling makes you cry, and you make sure it doesn’t happen to your apps.
• You love Java.  Well, maybe because you have to.  Either way, you know it inside and out.
• Your code works.  You know all the use cases and make sure they work how they’re supposed to work.
• You ship.  Your apps go out the door when they’re supposed to.
• This isn’t your first rodeo.  You have apps in the app store.

Overview: This post focuses on our experiences building on Postmark’s Inbound feature.

Postmark is our trusted mail carrier

Here at Sprout we use Postmark to deliver the emails that come out of our application. An example is the welcome email we send after someone signs up. Rather than build and maintain our own infrastructure to deliver emails, we gladly delegate that to them!

Most engineers think initially, “why wouldn’t you just run your own Postfix?” The answer is that deliverability is hard. Getting a scalable email infrastructure up and running isn’t too daunting. That’s especially true for us as one of our Dev Ops engineers built the Postfix infrastructure that underlies Rackspace Cloud Sites. However, ensuring deliverability of millions of emails is scary as hell, on our own. Thankfully, Postmark knows the tricks and they give really clear reporting to prove it. Our results have been outstanding, hovering at around 99.7% of our emails reaching their destination. That’s even more than 37signals gets using their home-grown infrastructure. Besides, the “cloud” is hot for a reason: specialization of labor. We’re small and every minute we spend not working on our core competency, our product, is not time well spent. 

Now they deliver to our house too

So we love Postmark and were stoked when they announced support for inbound emails. The timing couldn’t have been better. We had just started spec’ing out an enhancement to our Tasks feature (now released), which would bring email notifications. The gist is that users can task social media content for a teammate to handle; for example, responding to a customer question. Team members may also communicate with each other, in Sprout, on a thread related to the task at hand. Given the complexities of receiving emails, we expected to start with just a one-way, no reply email notification when some task activity happened. That would allow users to get quick visibility, but would leave them helpless until they got back to a computer. Postmark’s feature made it so damned easy, building the reply feature now was a no brainer.

Here is the gist of how the inbound feature works from a high level, if you are not familiar. A user receives an email from Sprout (via Postmark). The user then replies to that email, which is sent back to Postmark using a Postmark-generated From address. That From has a unique hash that lets Postmark know the response is intended for you. Postmark then converts that reply into JSON which they POST to a public HTTP endpoint that you specified ahead of time. Your endpoint then receives the HTTP POST with the Postmark-generated JSON representation of the email the user sent. In our case, we use the inbound JSON to create comments on the given task within Sprout. You can get more details on how this process works at Postmark’s site.

The one gotcha

Our experience with Postmark’s inbound feature was relatively painless besides one medium-sized caveat: Postmark does not parse out custom reply header text. Here’s what we mean. Open your email client and hit ‘reply’ on a message. Above the original message text you will notice a string that is pre-pended. Here is an example of what gmail prepends:

On Fri, Feb 17, 2012 at 1:49 PM, Sprout Social  wrote:

We love user experience at Sprout, and so stripping this line and all of the text that will appear below it, is ideal. Remember, the text above that line is the content we want to capture, not what’s below. Who wants to see a messy email thread at the bottom of their one line comment? From a programming perspective, this seems like no big deal, but when you delve into the variations across email clients, you find there are many cases to handle: Gmail, Hotmail, Yahoo! Mail, several Outlook versions, Mac Mail, Thunderbird, etc. Online stats reveal just how many clients are popular and some are really surprising for 2012. As one source, the fantastic Litmus published their take on the breakdown of email client usage.

While they don’t parse it out for you, Postmark at least offers some guidance, by passing the X-Mailer header in their JSON representation. That could direct what regex to use, but it’s not comprehensively provided. For example, although X-Mailer is very prevalent, Gmail doesn’t send it. Jerks. For us, what it all boiled down to is a class that searches the available headers in the inbound payload and then uses regular expressions to parse out the particular version of the line we want to strip. Here is an example of a regular expression that matches the above prepended reply text:

gmail_regex = ".*\"?Sprout Social\"?\s+\s*wrote:.*"

This isn’t an ideal solution as we now have to maintain which email client sends what headers and how their pre-pended reply string appears. Presumably, this can change over time too. It would be great if Postmark handled this mess once and for all, else each developer will have to reinvent the wheel. 

In closing, we’re big fans of the feature, and indirectly, our users will be too.

- Rob (@rmadd3n) & Aaron (@aaronrankin)

Today we decided that we needed to have autocompletion of usernames in the compose view for the mobile apps.  There are a lot of ways we could do it, and trust me we talked through a lot of them.  Our solution though, I think is awesome.  I’ll let you in on a little secret: it’s not complicated.  And in that lies the beauty.

We talked about setting up an endpoint to our API to poll for usernames on a user’s social graph in real time, but this would be expensive to develop the backend, and slow to to the user.  At Sprout Social we’re all about the user experience so that proved not to be our best option.

We did some thinking and, hat tip to Aaron for coming up with this, found an awesome solution.  The people you are most likely to be talking with are people that appear in your inbox and in your Twitter stream.  So, every time we see a Twitter screenname in your stream or inbox, we add them to a list.  Then when someone hits that ‘@’ symbol in compose we poll that local list for people matching the first few characters.  Then every time you select a user using autocomplete we increment their count.  We use that count value to break ties between queries that are equal. 

As a fallback, if you have no matches locally, we will hit the API to look for users.  We made this API call cheaper by searching the full Twitter namespace, instead of having tailor results for each user based on their followers and following lists.  The selected user is then added locally so this API call doesn’t need to be repeated in the future.

We were skeptical about how useful this would be immediately, but after about 5 hours of using it, the local cache has everyone I want to reply to.  And as a bonus, the people I’ve responded to most are the people that show up at the top of the list.

Simplicity is beautiful.

-@andylavoy

Intro

I will try to keep this short, sweet, and to the point. There’s a lot that goes on behind the pretty face that is sproutsocial.com. A web of systems work together to gather, enrich and store gobs of data quickly for our users. Messages are passed between these systems to coordinate work. This entry will focus on my recent experience with designing for high availability with Apache’s ActiveMQ.

After successfully setting up two instances of the ActiveMQ server pointing towards a MySQL database sitting on a separate external server (which was a difficult task in itself), the need arose for us to test the failover/redundancy of the entire system. For those not familiar with ActiveMQ, there is a specific nomenclature used as there is with many CS related topics. Here are a couple basic terms:

• Producer - Generates the jobs that are submitted to the ActiveMQ server.
• Listener - Listens for jobs on the ActiveMQ server and consumes them.

The Code

My goal was the create a producer that sent unique jobs every n seconds while a simple Listener listened for these jobs and printed debugging information. Here’s how I set this up. First I created a simple producer in python that attempted to connect to one of the ActiveMQ servers; if that failed, it attempted to connect to the second ActiveMQ server; if neither time was successful, it waited for a period and then retried.

SERVER1 = 'X.X.X.X' 
SERVER2 = 'X.X.X.X' 
PORT = XXXXX 

client = PublishClient(SERVER1, ACTIVEMQ_PORT)
ip_dest = ''
activemq_ip = SERVER1

try: response = client.connect(USER, PASSWORD) 
     activemq_ip = SERVER1 
except Exception as e: 
    client = PublishClient(SERVER2, PORT) 
    try: 
        response = client.connect(USER, PASSWORD) 
        activemq_ip = SERVER2 
    except Exception as e: 
        if (mq_params['count'] == 0): 
            print "Could not connect to activemq." 
            time.sleep(3) 
            mq_params['count'] += 1 
            self.send(mq_params) 
            return 
        else: 
            print "Could not connect after retry, return 500."
            sys.exit(0) 

client.send(queue, json_str) 
client.disconnect()

Perhaps not the cleanest python script, but for testing purposes it did the trick. On the other side of the queue, I created a Java listener to consume the jobs and log debugging information. Here’s a snippet of the Listener, particularly the overridden JMS ‘OnMessage()’ method in my FailoverListener class:

@Override
public void onMessage(Message msg) {
    
    if (msg instanceof BytesMessage) {
        BytesMessage bMsg = (BytesMessage) msg;
	StringBuilder buffer = new StringBuilder();

	try {
            for (int i = 0; i < (int)bMsg.getBodyLength(); i++) {
	        buffer.append((char) bMsg.readByte());
            }
	} catch (JMSException e) {
	    System.out.println("Failed to convert byte msg to string");
	}

	String text = buffer.toString().trim();
		
	FailoverPojo pojo = new FailoverPojo();
        Gson gson = new Gson();	
	pojo = gson.fromJson(text, FailoverPojo.class);

	System.out.println("Received message: " + pojo.asJson()); 
			
	// Must call acknowledge on Message object specifically.
	try {
		msg.acknowledge();
	} catch (JMSException e) {
		e.printStackTrace();
	}
    }
}

Most importantly, the connection for the listener needs to use the failover String as per the ActiveMQ Java library:

/* IMPORTANT PART */
private static final String connStr = 
    "failover://(tcp://IP1:PORT,tcp://IP2:PORT)?randomize=false"; 
cf = new ActiveMQConnectionFactory(connStr);

try {
    conn = cf.createConnection();
} catch (JMSException e) {
    System.out.println(e);
}

Destination failoverDest = new ActiveMQQueue("TestFailover");
Session failoverSession = 
     conn.createSession(false, Session.CLIENT_ACKNOWLEDGE);
MessageConsumer failoverConsumer = 
     failoverSession.createConsumer(failoverDest);
failoverConsumer.setMessageListener(new FailoverListener());   
conn.start();
assertNotNull(conn.getClientID());

That ensures that the Java listener will dynamically failover if the current server it’s listening to goes down.

Now, here’s the fun part. Hop on to each of your ActiveMQ servers using ssh and start each instance. Whichever grabs the database lock first will be the actual server in use. Start your Java listener and make sure that it connects to your ActiveMQ server. Next, run the producer that will send jobs to the ActiveMQ server every n seconds (specified in the script). Lastly, kill the currently running ActiveMQ server and watch the other server grab the lock and pick up where the previous server left off without a hitch.

Here are some screenshots showing the live failover. The first is the python producer sending jobs to the queue and the second is the Java listener consuming said jobs:

Insights

It’s possible that one of your jobs will fail during the downtime, depending on your retry threshold (the amount of time that your python script sleeps before trying to resend). You will see the job being retried and successfully submitted to the new ActiveMQ server. On the other side, your listener will have caught the switch and seamlessly grabbed the job from the new server.

That’s it! Try experimenting with how low you can set the “retry threshold”. I was able to get it down to roughly four seconds before the job fails.

- Rob (@rmadd3n)

“Testing is fun.” - No one has ever said this

I’m not sure that I could find a whole lot of developers who agree with this statement, but the iOS community has been given hope. 

Before I tell you about my reason for hope, let me enlighten you on the cluster@#^# painful process that used to be iOS testing. 

The Pain

Feel free to skip this section if you only care about how to do it the cool way, but for the sake of sensationalization, let me be verbose.

1) Send an email to a prospective tester asking them to send me their UDID.
2) Add the UDID to our iTunes Connect account. (Apple’s hub for regulating test devices)
3) Generate and download an updated provisioning profile and build the app using an Ad Hoc configuration (with this provisioning profile). 
4) Send the app (an .ipa file) along with the provisioning profile (a .mobileprovision) file to the tester with ridiculously detailed instructions on how to twist the phone’s arm into running the app.
5) The tester gets the email, gets confused by the instructions, messes up the process a few times, and finally drags the 2 files received from the developer into their iTunes Library on the computer they sync the device with, and then syncs the iOS device.

99% of iOS users don’t know what a UDID is, nor should they need to.  Folks, it’s the device’s ID number.  The process for finding this ID is in no way straight forward and often requires additional tester instruction.  Eventually the tester does find the ID and sends it to me.  Later, after I perform some magic to get the app ready to install, the same user receives a .zip file containing an .ipa and a .mobileprovision file.  Again, we have a confused tester.  They probably know how to sync their iPhone, but this whole drag and drop to install is frustrating and confusing.  It isn’t the normal process, nor is it as straightforward and easy.

Testers are hard to come by, and losing them because of technical misunderstanding is a sad story.

The Hope

When Apple released iOS 4.0 they dropped in an amazing feature for developers: Over the Air Distribution.  The biggest hurdle in Ad Hoc testing was getting testers to figure out how to install the app.  Now with 2 taps on an iPhone, a user is able to download my app OVER THE AIR.  No sending files, no syncing, no instructions, and no confusion.

So, I discovered OTA about 9 months ago when I read Jeffrey Sambells’ article outlining how to do it, but that was just the start.  Some suckers for simplicity decided that this wasn’t good enough.  They felt iOS developers’ pain and did something about it.

Icing on the Cake

Enter TestFlight.  Now, I’m doing my best to not make this sound like an advertisement, but I have found love.  As a developer, I created an account for Sprout Social and sent out a link (Step 1 from above) to perspective beta testers.  The testers signed up for the service (it was easy) and registered their iPhones from the comfort of their couches and Mobile Safari.  Unbeknownst to them, they were supplying me with everything I could ever ask for as a developer.  TestFlight harvested their UDIDs, their iOS versions, their hardware versions, and their email addresses, and all I did was send a single link. 

I now have a list of testers with all their relevant info.  All I need to do is add the newly harvested UDIDs to our iTunes Connect account (Step 2), build the app with the new profile from ITC (Step 3), and then upload the app to TestFlight (replacement for step 4).

The tester then gets an email telling them that I have the app ready for them to test, with a link to click on to install the app over the air (übereplacement for step 5).

Apple, with some awesome help from TestFlight has made managing my testing efforts a breeze.  I can now recruit new testers with almost no effort, and publish updates to existing testers without having to worry about whether they’ll be able to figure out how to install the app.

The bottom line is that I get to write my code and distribute it without too much pain.  TestFlight saves me time and lets me get back to building the best possible mobile experience for our customers.  TestFlight saves our testers time and headaches, making them more likely to actually test the app (this is important) and give us feedback to make it better.

Testing may never be truly fun, but now I can at least get excited to push new updates to our testers.  And as a developer, that’s as good as it gets.

- @andylavoy

Sprout Social is a cute nephew/niece of Groupon, sharing a common investor lineage. Despite the startup-hip points the big G has, our engineering team has turned many a head while grabbing coffee in the breakrooms of 600 W Chicago Ave. (yes, we don’t have our own machine yet). All it takes are phrases like “opcode caching”, “hiphop PHP optimization”, “HTML5 local storage”, “mustache templating” and “replication trees”. Sounds borderline hipster? Not quite. Groupon: “What the hell are you talking about?” Us: “Oh, we actually don’t work here.” Situation: [weird]

We played with a lot of random technology as kids, back to DOS 2.1; we’re not that old, we were just retro kids. Amortize this across 12 (and we’re hiring) engineers and you still can’t build the perfect solution for today’s world. We have a lot to pull from, but the industry couldn’t have given us a palette that’s more chaotic and more like a giant Ouija game:

• social media
• browser evolution into a premier app platform (speed, html5, css3)
• cloud infrastructure, services and SaaS
• mobile and tablets
• “web scale” data: affordable parallel computing and simple patterns (Map Reduce), noSQL, scaling relational for high reads, writes and storage simultaneously
• Apple altering consumer expectations
• the list can go on

Predicting where all of this will go is Nostradamus territory and we don’t want to step on old media’s expertise in dramatization. Sprout Social is straddling it all and adding our own momentum to the game. As we build and learn with our users, we will have plenty to share, especially as engineers. Stay tuned. This will be fun.

- Aaron  (@AaronRankin)