Sign In

Christopher Meiklejohn

Software Engineer; Ph.D. in Microservice Testing

Christopher Meiklejohn, a Ph.D. graduate from Carnegie Mellon University, is renowned for his extensive research spanning fault-tolerant and reliable distributed systems.

An expert in the field, Meiklejohn created Filibuster, a groundbreaking microservice testing framework available at filibuster.cloud, developed based on empirical academic research and his profound doctoral studies.

His academic journey includes a Doctor of Philosophy (Ph.D.) in Software Engineering from Carnegie Mellon University, a Bachelor of Science (B.S.) in Information Technology from Northeastern University, and an Associate of Science (A.S.) in Computer Science from Community College of Rhode Island.

Having a rich and diverse work history, Meiklejohn has served in various roles, including Software Engineer at DoorDash, PHD Candidate at Carnegie Mellon University, Independent Contractor at DoorDash, Research Intern at Amazon Web Services (AWS) and Microsoft, Consulting Scientist at Macrometa Corp., Ph.D. Student at Carnegie Mellon University, Research Intern at Microsoft and Université catholique de Louvain, Independent Contractor at Mesosphere, Senior Software Engineer at Machine Zone, Inc. and Basho Technologies, Software Engineer at Swipely, Software Engineering Manager at Berklee College of Music, and Senior Systems Engineer at Conversent Communications.

Highlights

Aug 27 · linkedin

Many microservice applications classify their dependencies into different Service Tiers. Perhaps your developers are working on a Tier 1 service and want to make sure that they are tolerant to any failure of Tier 2+ failures. How can they do that?

Well, if they've written a functional test that can be run with Filibuster, you can use a new feature of Filibuster called "fault injection filters". Fault injection filters allow you to arbitrarily implement filter functions that are provided with the invoked RPC service, method, and arguments to conditionally prevent fault injection. For example, by implementing a filter that allows faults for any downstream services with a tier greater than your service.

Here, we have a simple purchase method executed by an Order Service. It issues 7 downstream RPCs:

  • one to look up the user, which when failed, causes the service to return error.
  • one to validate the user session, which when failed, causes the service to return error.
  • one to look up the user's cart, which when failed, causes the service to return error.
  • three to look up the possible discounts for the user's cart.
  • one to send an email to the user using a Tier 2 service.

Now, if I use Filibuster normally, I'll execute this test over and over and fail every RPC and every combination of RPCs. Any test that asserts the purchase is successful will fail: for example, by failing the downstream RPC to look up the user, the test will fail because I'll return error. In this case, I'm supposed to return error, and Filibuster provides an API that allows you to say this precisely: when that downstream RPC fails, I won't be able to complete the purchase and will get this error.

But, perhaps I want to verify if all of my Tier 2 dependencies fail, the system will always return success.

Since sending the user an email is done using a Tier 2 service, I can easily implement a filter that states that I do not want to inject faults on any Tier 2 services, and that the email service is a Tier 2 service. Then, when I run Filibuster, I will only inject faults on Tier 2 services and any failure of my test indicates that it is not tolerant to a Tier 2 service failure: it returned failure, when the system should have been tolerant to that failure.

...and of course, we can visualize all of this right inside the IDE.

#chaosengineering

Aug 13 · linkedin

The distance between research software and software that can be used by real software engineers is --- quite honestly --- enormous. It just doesn't get talked about enough, and it should.


There's a long lineage of work (short: papers) on finding concurrency bugs in systems like Zookeeper, with a lot of it done by the University of Chicago.

Why is that?

Well, there are multiple factors, but perhaps the most interesting to me is that some time in the past, someone found a way to instrument Zookeeper in such a way that they could control thread scheduling and each RPC sent. Therefore, when new Ph.D. students come along, they can build on this existing work and focus only on the algorithmic improvements.

For example, SAMC (OSDI '14) had semantics-aware dynamic reduction policies like local/crash-message independence; FlyMC (EuroSys '19) had parallel flips and state symmetry. Same system, more bugs found each paper. The bugs just keep getting more complicated too, as it's a battle of attrition. From FlyMC: "The second bug was reported to appear once every 500 unit test cycles [...] successfully discovered 5 new bugs [...] bug depths ranged from 9 to 30 events."

In short, Zookeeper bugs get more and more complicated, we come up with better tricks to find them; and Zookeeper keeps getting better every day. One might even ask: when will Zookeeper run out of bugs to be found, if ever?


In my own Ph.D. work, I've been building software for testing microservices applications where one of the key tenets is that services are developed by different teams. This means that each service can be written in a different style, using different versions of libraries, with different dependencies, using different testing frameworks, using different DI frameworks, etc., etc. This makes mechanization of a "one size fits all" approach to testing... very hard.

I think there are two implications when it comes to software testing:

  • From the industrial perspective, I feel like this is why developers often resort to infrastructure-level testing techniques, because utilizing an application-level testing technique across multiple services is hard, often less-valued, work.

  • From the research perspective, it makes developing software that can run on real services very difficult. In short, don't be too generic in that you sacrifice performance, but be generic enough, having just the right hooks, that you can wire yourself into any framework.

This is a skill that Ph.D. students just do not have. In fact, I often fail at it myself, and I was a professional engineer for 15 years in senior roles before starting my Ph.D.

Tech transfer isn't a goal for academics and doesn't seem to be really valued. As a community, we often rely on someone in industry to create a derivitive work that works on real things.

Perhaps this is how research is supposed to be in the field of Computer Science.

Is that how it should be in the field of Software Engineering?

Christopher Meiklejohn
Add to my network

Location

Pittsburgh, Pennsylvania, United States