Ready…Set…Hack!

HackMIT: Hack to the Future

In the past couple of months, UnifyID has been busy attending university hackathons at MIT and UC Berkeley. What this means is hours and hours of non-stop hacking, but it also means unlimited snacks, mini midnight workouts, and lots of young, passionate coders working to create impactful projects.

John poses with a16z representative Nigel at HackMIT.
On September 16, John Whaley flew to Cambridge, Massachusetts to attend HackMIT: Hack to the Future where he had the opportunity to meet more than 1500 students from all different universities. Representing a16z, John participated in a fireside chat where he covered a variety of topics including what it’s like to work in a startup, choosing industry versus graduate school, and building a company on machine learning. He discussed the fundamentals of entrepreneurship, team-building, fundraising, and more, as students picked his brain about technical topics and career advice. Later, John was able to speak more in depth during his tech talk about UnifyID and identifying individuals based on gait. Students were deeply interested in the problem UnifyID is trying to solve as well as the impact and intellectual aspect of UnifyID’s approach to the issue.

Aside from his fireside chat and tech talk, John had the opportunity to mentor hackers in their own projects. His favorite part was meeting and interacting with all of the students, seeing their ambition, passion, and genuine interest in the projects that they were working on. He also enjoyed the intense energy in the arena, choosing to stay and mentor hackers until 3am.

After 24 hours of hard work and non stop hacking at MIT, many groups of students presented their projects. Projects covered a wide range of topics from virtual reality games to homework-help mobile applications. Even though John had been to plenty of hackathons in the past, he was still amazed by the caliber and level of innovation that the students were able to reach in their projects. The first place prize ended up going to a group of students who created Pixelator, “a simple product that sharpens blurry images without a lot of code.”

Cal Hacks 4.0

A few weeks later, on October 6, Andres Castaneda crossed the Bay to attend Cal Hacks 4.0 at the UC Berkeley Stadium. With nearly 1500 students listening, he gave a presentation about UnifyID’s Android SDK and API, receiving a positive response from students who believed it was a revolutionary idea. Similar to John at MIT, Andres also had the opportunity to mentor up-and-coming hackers. For 36 hours, he helped students solve technical challenges as they competed for over $100,000 in prizes, including UnifyID’s contribution: a $300 Amazon giftcard and a Rick and Morty card game.

Based on the level of positive impact, innovation, and technical difficulty, the winning hack for UnifyID’s prize was Safescape, a mobile application that analyzes real-time news articles and alerts people in areas of “non-safe” events. It uses UnifyID’s Android SDK to validate individuals on the application. Inspired by the recent natural and terror crises occurring globally, Safescape also provides those in danger with potential escape routes, allows them to alert others around them, and contains a simple way to contact loved ones.

Andres’ favorite part about participating in Cal Hacks was “seeing people build a product from 0 to 1 in 36 hours.” He also found it hilarious that many students brought sleeping bags and threw them on the floor for intermittent opportunities to take naps.

Andres poses with mentees and previous UnifyID interns Aditya and Michael.

UnifyID is a strong supporter of hackathons because they provide great opportunities to connect with university students. Witnessing the high caliber of work accomplished at these events, UnifyID is inspired by young hackers who are truly passionate about making an impact in the world. These students represent a large diversity of talent from all different schools and backgrounds and are able to demonstrate what students are interested in nowadays. Additionally, hackathons allow UnifyID the chance to give back to the community. They are not only learning opportunities for up-and-coming hackers, but they also help UnifyID to understand how to cater to students’ interests and needs. After 2 hackathons in the span of one month, UnifyID is channeling its focus back to the day-to-day for now; however, we cannot wait for the next one!

A load balancer that learns, WebTorch

In my previous blog post “How I stopped worrying and embraced docker microservices” I talked about why Microservices are the bees knees for scaling Machine Learning in production. A fair amount of time has passed (almost a year ago, whoa) and it proved that building Deep Learning pipelines in production is a more complex, multi-aspect problem. Yes, microservices are an amazing tool, both for software reuse, distributed systems design, quick failure and recovery, yada yada. But what seems very obvious now, is that Machine Learning services are very stateful, and statefulness is a problem for horizontal scaling.

Context switching latency

An easy way to deal with this issue is understand that ML models are large, and thus should not be context switched. If a model is started on instance A, you should try to keep it on instance A as long as possible. Nginx Plus comes with support for sticky sessions, which means that requests can always be load balanced on the same upstream a super useful feature. That was 30% of the message of my Nginxconf 2017 talk.

The other 70% of my message was urging people to move AWAY from microservices for Machine Learning. In an extreme example, we announced WebTorch, a full-on Deep Learning stack on top of an HTTP server, running as a single program. For your reference, a Deep Learning stack looks like this.

Pipeline required for Deep Learning in production.
What is this data, why is it so dirty, alright now it’s clean but my Neural net still doesn’t get it, finally it gets it!

Now consider the two extremes in implementing this pipeline;

  1. Every stage is a microservice.
  2. The whole thing is one service.

Both seem equally terrible for different reasons and here I will explain why designing an ML pipeline is a zero-sum problem.

Communication latency

If every stage of the pipeline is a microservice this introduces a huge communication overhead between microservices. This is because very large dataframes which need to be passed between services also need to be

  1. Serialized
  2. Compressed (+ Encrypted)
  3. Queued
  4. Transfered
  5. Dequeued
  6. Decompressed (+ Decrypted)
  7. Deserialized

What a pain, what a terrible thing to spend cycles on. All of these actions need to be repeated every time the microservice limit is crossed. The horror, the terrible end-to-end performance horror!

In the opposite case, you’re writing a monolith which is hard to maintain, probably you’re either using uncomfortable semantics either for writing the HTTP server or the ML part, can’t monitor the in between stages etc. Like I said, writing a ML pipeline for production is a zero-sum problem.

An extreme example; All-in-one deep learning

Venn diagram of torch, nginx
Torch and Nginx have one thing in common, the amazing LuaJIT
That’s right, you’ll need to look at your use case and decide where you draw the line. Where does the HTTP server stop and where does the ML back-end start. If only there was a tool that made this decision easy and allowed you to even go to the extreme case of writing a monolith, without sacrificing either HTTP performance (and pretty HTTP server semantics) or ML performance and relevance in the rapid growing Deep Learning market. Now such a tool is here (in alpha) and it’s called WebTorch.

WebTorch is the freak child of the fastest, most stable HTTP server, nginx and the fastest, most relevant Deep Learning framework Torch.

Now of course that doesn’t mean WebTorch is either the best performance HTTP server and/or the best performing Deep Learning framework, but it’s at least worth a look right? So I run some benchmarks, loaded the XOR neural network found at the torch training page. I used another popular Lua tool, wrk to benchmark my server. I’m sending serialized Torch 2D DoubleTensor tensors to my server using POST requests to train. Here’s the results:

Huzha! Over 1000 req/sec on my Macbook air, with no Cuda support and 2 Intel cores!

So there, plug that into a CUDA machine and see how much performance you squeeze out of that bad baby. I hope I have convinced you that sometimes, mixing two great things CAN lead to something great and that WebTorch is an ambitious and interesting open source project! Check out the Github repo and give it a star if you like the idea.

https://github.com/UnifyID/WebTorch

And hopefully, in due time it will become a fast, production level server which makes it easy for Data Scientists to deploy their models in the cloud (do people still say cloud?) and devOps people to deploy and scale.

Possible applications of such a tool include, but not limited to:

  • Classification of streaming data
  • Adaptive load balancing
  • DDoS attack/intrusion detection
  • Detect and adapt to upstream failures
  • Train and serve NNs
  • Use cuDNN, cuNN and cuTorch inside NGINX
  • Write GPGPU code on NGINX
  • Machine learning NGINX plugins
  • Easily serve GPGPU code
  • Rapid prototyping Deep Learning solutions

Maybe your own?

Distributed Testing with Selenium & Docker Containers (Part I)

As seen at TechCrunch Disrupt, the UnifyID Google Chrome extension plays a prominent place in the user’s first impression with the product. As such, it is critical that the experience be smooth, user-friendly, and most important, reliable. In minute 2:10 of the demo, the Amazon login fields are replaced with the UnifyID 1-click login. This requires DOM (document object model) manipulation on the web page which isn’t challenging on a single website; however, when scaling your code across sites in multiple languages and formats, finding the right login form and place to insert our logic is challenging.

How to find the right elements in here?
How to find the right elements in here and generalize well?

Parsing through the noise of variations on HTML structures, CSS naming conventions, and general web development is like the wild west–but despite this, there are ways to gain sanity, namely in testing.

As most UIs are tested, we utilize Selenium and its Javascript binding. For developers who have had some experience with Selenium, it becomes painfully obvious very early on that speed is not its forte. The testing environment must handle testing functionality across dozens, hundreds, and thousands of sites while maintaining the fast, iterative nature of development. If every test case takes an hour to finish, then our continuous integration would lose its magical properties (you know, moving fast without breaking things).

Surfing at speed through the codebase.
Surfing at speed through the codebase, knowing everything will be ok.

The solution to distribute testing across multiple computers will require the following:

  1. Must run multiple instances of the tests with the same code but on different websites.
  2. Must run all of the tests in parallel.
  3. Must collect all results.

In order to simplify my work, I discovered that Selenium has this neat feature called Selenium Grid that allows you to control testing in a remote machine so that the local machine doesn’t have to run the tests itself. You just run the grid in another machine, expose the port, and remote connect to that port by using settings found in your chromedriver.

This was amazing! This meant that we could have machines entirely dedicated to running a bunch of Selenium webdrivers (which are considerably RAM consuming) without killing our CI server. Plus, we could easily scale by just spawning more of these machines and more of these web drivers.

Finally!
Finally! Scalability!

As a result, the design evolved to this:

  1. Set up a Selenium server to run a grid of browsers. Run Chrome.
    1. Package under a Docker image for easy deployment.
    2. Run under an Amazon AWS instance for easy scaling.
  2. The CI server has to be able to send requests to the Grid.
  3. The tests have to run in parallel.
    1. Create a Javascript library to make tests into chunks that can be sent over to the server.
  4. Finally, the results have to be merged together and sent back to the CI server.

For 1, I downloaded selenium-server-standalone-2.53.1.jar from “http://selenium-release.storage.googleapis.com/index.html?path=2.53/” and saved as selenium-server-standalone.jar.

Run hub with:

java -jar selenium-server-standalone.jar\
-role hub

Run node with:

java -jar selenium-server-standalone.jar -role node\
-hub http://localhost:4444/grid/register\
-browser browserName=chrome,version=52.0,\
          chrome_binary='/usr/bin/google-chrome',\
          platform=LINUX,\
          maxInstances=5

Now to connect to the grid, we can run from JavaScript:

Look out for my next post which will delve deeper into parts 2, 3, and 4 above.

Thanks for Hacking!

UnifyID @ HackMIT

I just got back from HackMIT, and what a crazy, intense experience it was. For those who don’t know, HackMIT is a 24-hour hackathon with over 1,000 students from all over the country and the world, all hacking on some very cool stuff. I was on the judging panel as well as acted as a mentor, helping students debug issues with a wide variety of technologies like node.js/Express, cocoapods and Swift 3, Ethereum smart contracts, Angular and Javascript, 502 errors on HTTP requests, and a bunch of other issues. A few students came up to me after they recognized UnifyID from our TechCrunch video and wanted to take photos together.

I met a lot of great students from all over the US, Europe, and South America. I also gave a tech talk where we demonstrated our implicit authentication technology in action with a volunteer from the audience. Being a technical crowd, I was able to dive deep into the technical aspects with some of the actual data in a Jupyter notebook. People were amazed by some of the unique aspects to human movement and how much information you can get from the accelerometer and gyroscope in your phone!

HackMIT had tons of free food/drinks/activities. They had no soft drinks because they were encouraged to avoid unhealthy drinks, but they had plenty of Red Bull (?). And unlimited Soylent, too. Plus food/snacks at all hours of the day and night, like fresh smoothies at midnight and hot waffles with chocolate in the morning. And crazy activities like the 2am shakedown and the 7 minute workout outside in the wee hours of the morning.

Many/most teams stayed up all night hacking. There was a wide variance in hacking ability but the top teams were truly astonishing in what they were able to build in 24 hours. All of the top ten were amazing and it was hard to choose.

a8d03f4d3a3c6932a895ff34069b12d3

The ultimate winner was “WindowShare”. They built an awesome cross-platform tool where you can drag any window between computers and it seamlessly copies the program’s file and opens it on the other machine. Like if you open a text file in TextEdit on a Mac, you can drag the window over and the contents appear in a Notepad window on the Windows machine. Likewise for images and Chrome windows/tabs. They also implemented remote mouse so you could move your mouse on the other screen as well and control it without messing up the original mouse. They implemented in Java with JNI and socket communication.

The runner-up was a book-reading bot that used the phone, OCR, and text-to-speech to read (physical) books aloud. It also used a motorized mechanism including a computer fan to reliably turn pages.

We also added a honorable mention: “Fretless”, an MIT team that built a Guitar Hero like contraption that hooks to your violin. It takes a MIDI file and lights up where you are supposed to press your fingers so you can learn how to play real songs.

All of the top ten projects were amazing and the teams got a ton done in 24 hours! To everyone who participated, I say “Hack on!”