Benchmarking the Top AutoML Platforms: An In-Depth Analysis

The rapid rise of AutoML has been a boon to organizations seeking to make the most of their data. Machine learning used to be an extremely complex, time-intensive, and expensive process. Automated machine learning decreases time-to-value, complexity, and cost - but not all AutoML platforms are made equal, and many still require expert data scientist users.

No-code AutoML platforms are designed to make machine learning accessible to non-technical users, freeing up time to focus on more creative, business-oriented tasks. While these platforms are intended for business users, they also have the potential to enhance the capabilities of data scientists.

In this article, we’ll compare the top AutoML platforms.

Finding the Best AutoML Platform

If you’re in the market for an AutoML platform, you’re faced with a deluge of options, from the Fortune 500 to startups. It’s important to make sense of the pros and cons of each because there’s no one-size-fits-all solution.

Traditional AutoML solutions can be surprisingly difficult to set up, resulting in a distinct lack of performance benchmarking. In short, we couldn’t find any end-to-end AutoML comparison articles, so we decided to do the research ourselves.

We tested the following platforms:

• Google AI Platform (Cloud AutoML)
• Microsoft Azure AutoML
• Amazon SageMaker Autopilot
• Akkio

Conducting Tests

We tested each of these platforms on over a dozen, open-source, real-world datasets.

The resulting automated machine learning models were benchmarked on a variety of factors, including accuracy, F1 scores, training time, and cost. We’ll first cover these objective metrics, before diving into ease-of-use.

Accuracy and F1 Scores

Accuracy is simply how often a machine learning algorithm correctly predicts outcomes. We measured accuracy on 14 popular datasets for each platform.

For example, we uploaded the Australian credit approval dataset, where the goal is to automate credit card eligibility outcomes, to each AutoML platform.

Akkio’s machine learning model had an accuracy of 0.9135, meaning that it correctly predicted credit card eligibility in 91.35% of data points. Microsoft’s accuracy was 0.8985, Amazon’s accuracy was 0.913, and Google wasn’t able to process the data at all.

It’s important to consider “F1 score” as well, which considers the number of false positives and false negatives, providing a metric that’s especially useful for “imbalanced” datasets, like credit card fraud or cancer detection.

For instance, if you built a model to predict credit card fraud, and the model said “no fraud” every time, you would have a very high accuracy of, say, 99.99%. However, a model that always predicts “no” is useless, and this would be reflected in the very low F1 score of such a model.

Of the 14 datasets tested across four platforms, Akkio had both the highest accuracy and the highest F1 score on 6 datasets; Google came in second place, with the highest accuracy metrics on 3 datasets; Microsoft came in third place, with the highest accuracy metrics on 2 datasets; Amazon SageMaker came last, with the highest accuracy metrics on 1 dataset.

Here’s a visual summary.

Note that the above adds to 12 datasets, as multiple platforms tied on 2 datasets.

Training Time

The major selling-point of AutoML is saving time. In fact, a Dotscience survey reported 80% of respondents’ companies take more than six months to deploy an artificial intelligence model.

That’s an extremely long time to devote to something that may not pay off.

AutoML comes with the promise of making the AI journey exponentially faster, but as we said, not all AutoML platforms are made equal, so we compared the training times on 14 datasets across the aforementioned platforms.

Akkio was significantly faster on every single dataset - typically one to two orders of magnitude faster. Akkio’s average training time across the 14 datasets came out to just over three minutes, compared to over an hour on Microsoft, nearly 2 hours on Google, and over 2 hours on Amazon.

Cost

Cost is another critical metric to compare. After all, the low price of AutoML is another big selling point, given the sky-high salaries of data science and AI engineer teams.

Most AutoML platforms - including those by Amazon, Google, and Microsoft - charge hourly, which means that you have to commit to an up-front cost, without knowing whether a model will work.

As we’ve seen, traditional AutoML platforms also have long training times and generally lower accuracy than Akkio, resulting in the need to go back-and-forth to make adjustments, adding to your total cost.

Akkio has no hourly costs, and instead uses a highly affordable usage-based subscription pricing model.

Ease-of-use

Lastly, let’s talk about ease-of-use, as it’s more subjective than the other metrics, but is an important factor to consider.

Many of the major AutoML platforms today do speed up the AI journey, but only to the extent that they make it easier for technical professionals. There’s still considerable complexity and effort required for non-technical folk, which makes them largely inaccessible.

We went through the on-boarding processes for Google AutoML, Azure AutoML, and SageMaker to show you what we mean.

To sign-up for Azure AutoML, you need to verify your phone number, add a credit card, add your personal address information, and sign a complex agreement.

You then need to create a machine learning workspace (with details like a workspace name, subscription, and resource group), create a new Automated Machine Learning model, create an experiment, create a new compute (specifying compute name, virtual machine size, minimum and maximum number of nodes), select the compute, upload a dataset, select the prediction task, run the model, and more. Clearly, the extent of "automation" is limited, and the machine learning process is still lengthy.

That’s only covering the gist of taking Azure AutoML for a test drive, and we haven’t even covered deployment. In short, Azure AutoML has a very poor ease-of-use.

The process for Amazon SageMaker Autopilot is similarly complex. You’d need to create an AWS account, set up Amazon SageMaker Studio with an IAM role that has the required permissions, run code to extract data from Amazon S3, create an experiment specifying details like “S3 location of input data” and “S3 location for output data,” and more. This is all from their hands-on tutorial. Clearly, SageMaker Autopilot also ranks poorly for ease-of-use.

Now, let’s take a look at Google AutoML, which was recently rebranded to Google AI Platform. Their AI platform includes AutoML Vision and other solutions - we'll focus on their tabular data AutoML. Frankly, Google AutoML is quite difficult to use, which explains why Google AutoML is a sought-after skill - for enterprises that are locked-in to using Google AutoML, they’ll pay a handsome price to figure out how to actually use the software.

Google’s quickstart guide explains how you need to create a data “bucket,” enable Cloud AutoML and Storage APIs, set a “training budget,” and fiddle with many other features just to get things up and running. After building a machine learning model, you also need to be careful to undeploy your model, delete the model, and delete the dataset, lest you incur unnecessary Cloud AutoML charges.

Ultimately, the AutoML offerings by Amazon, Google, and Microsoft are complex and difficult to use, and are best left to technical experts.

Akkio is worlds apart, allowing you to sign-up in seconds, without needing to put in credit card details, your phone number, or any extraneous information.

From there, it’s literally one click to view a demo machine learning model, or to create your own AI flow, where you can effortlessly upload data, select a column to predict, and build a model, all in under a minute. All the heavy lifting, from feature engineering to hyperparameter tuning to model training, is done in the background.

True automated ML shouldn't require a data science team, or even a data science process - it should be usable by anyone.

Conclusion

There are many different options to choose from when considering an AutoML solution. At Akkio, we believe that the best way to determine the right platform for you is to think about your goal and needs. Machine learning is a business tool, so choosing the right AutoML tool for you will ultimately come down to what matters most to your company’s success.

Here’s a visual summary of the three main metrics we tested: Speed (of both training time and time-to-value), accuracy, and price.

Akkio was the fastest solution, followed by Microsoft, Google, and Amazon. Akkio was also the most affordable solution, followed by Microsoft, Amazon, a large gap, and then Google. Finally, Akkio was most accurate, followed by Google, Microsoft, and then Amazon.

If you want to build AutoML algorithms as a true citizen data scientist, without worrying about complex topics like model selection, model performance, hyperparameter optimization, and the like, then Akkio is for you. Try Akkio to see how easy it can be to use AutoML in your business.