The risk in following ML models is they could be based on false assumptions and skewed by noise and outliers. That could lead to making bad predictions. That is why ML cannot be a black box. The user must understand the data and algorithms if the models are to be trusted. So here we look at some more measures of trustworthiness. As in the … [Read more...]

Here we introduce some terms important to machine learning; variance, r2 score, and mean square error. We illustrate with these concepts using scikit-learn. It is important to understand these metrics to determine whether regression models are accurate or misleading. Following a flawed model is a bad idea. So it is important that you can … [Read more...]

Here we explore another machine learning framework, scikit-learn, as well as show how to use matplotlib, to draw graphs. The scikit-learn python ML API predates Apache Spark and TensorFlow, which is to say it has been around longer than big data. It has long been used by those who see themselves as pure data scientists, as opposed to data … [Read more...]

Here we explain what is a Spark machine learning pipeline. We will do this by converting existing code that we wrote, which is done in stages, to pipeline format. This will run all the data transformation and model fit operations under the pipeline mechanism. The existing Apache Spark ML code is explained in two blog posts: part one and part … [Read more...]

In part one we explained how to create a training model. In this part we show how to make predictions to show which machines in our dataset should be taken out of service for maintenance. First, here is how to submit the job to Spark with spark-submit: jar file that contains com.bmc.lr.makePrediction what file to read, i.e., the one you … [Read more...]

Here we explain a use case of how to use Apache Spark and machine learning. This is the classic preventive maintenance problem, one of the most common business use cases of machine learning and IoT too. We take the data for this analysis from the Kaggle website, a site dedicated to data science. This is sensor data from machines, specifically … [Read more...]

Here is the last part of our analysis of the Tripadvisor data. Part one is here. In order to understand this, you will need to know Python and Numpy Arrays and the basics behind tensorflow and neural networks. If you do not, you can read an introduction to tensorflow here. The code from this example is here and input data here. We create a … [Read more...]

  Code refactoring means to take a working program and change it to make some improvements. It changes the code but not the outcome. These improvements can: make it easier for other programmers to read make aesthetic improvements such as implement a clever idea make the program run faster or use less resources adhere to … [Read more...]

The Google Cloud Machine Learning Engine is almost exactly the same as Amazon Sagemaker. It is not a SaaS program that you can just upload data to and start using like the Google Natural Language API. Instead, you have to program Google Cloud ML using any of the ML frameworks such as TensorFlow, scikit-learn, XGBoost, or Keras. Then Google spins up … [Read more...]

In the last blog post we showed you how to use Amazon SageMaker. So read that one before you read this one because there we show screen prints and explain how to use the graphical interface of the product, including its hosted Jupyter Notebooks feature. We also introduced the SageMaker API, which is a front end for Google TensorFlow and other … [Read more...]