Prerequisites and Environment setup<\/h2>\n
This tutorial is carried out in Anaconda Navigator (Python version – 3.8.3) on Windows Operating System. The following packages need to be installed before you continue with the tutorial –<\/p>\n
- \n
- Pandas<\/li>\n
- NumPy<\/li>\n
- Scikit-learn<\/li>\n
- Flask<\/li>\n
- Joblib<\/li>\n<\/ol>\n
You can install these packages in Conda’s virtual environment using
conda install package-name<\/code>. In case you are using Python directly via terminal\/command prompt,pip install package-name<\/code> will do the work.<\/p>\nNote that to access GridDB’s database through Python<\/a>, the following packages will be required –<\/p>\n
- \n
- GridDB C-client<\/li>\n
- SWIG (Simplified Wrapper and Interface Generator)<\/li>\n
- GridDB Python-client<\/li>\n<\/ol>\n
Great! Our environment is all set up and ready to use. Let’s create a Machine Learning model.<\/p>\n
Creating a Machine Learning Model<\/h2>\n
We will be using the Linear Regression model from one of our previous posts on Machine Learning using GridDB<\/a>. The full source code is available on Github<\/a>. We will modify the code as we move forward to inculcate the new changes. You can either download the source code to follow along or we will be attaching the python file at the end of this tutorial as well.<\/p>\n
The Linear Regression Model for CalCOFI dataset looks like –<\/p>\n
\nimport numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LinearRegression\nimport joblib\n\ndataset = pd.read_csv(\"bottle.csv\")\ndataset=dataset[[\"Salnty\",\"T_degC\"]]\ndataset = dataset[:500]\ndataset=dataset.dropna(axis=0)\ndataset.reset_index(drop=True,inplace=True)\n\nx_label=np.array(dataset['Salnty']).reshape(493,1)\ny_label=np.array(dataset['T_degC']).reshape(493,1)\nx_train, x_test, y_train, y_test = train_test_split(x_label, y_label, test_size = 0.2, random_state = 100)\nregression_model=LinearRegression()\nregression_model.fit(x_train,y_train)<\/code><\/pre>\n<\/div>\nNote that we have removed the last few lines where we’re making predictions on the test dataset as we don’t need them here. The model accuracy is
87%<\/code> as we saw in the previous post. If you plot the resulting model, it will look like –<\/p>\n\nplt.figure(figsize=(12,10))\nplt.scatter(x_label, y_label, color='aqua')\nplt.plot(x_train, regression_model.predict(x_train),linewidth=\"4\")\nplt.xlabel(\"Temperature\",fontsize=22)\nplt.ylabel(\"Salinity\",fontsize=22)\nplt.title(\"Linear Regression\",fontsize=22)<\/code><\/pre>\n<\/div>\n![\"\"](\"linear_regression_model.png\")
<\/p>\nSerialization and Deserialization of the Machine Learning Model<\/h2>\n
Our model is now ready to make predictions, but we will need to store the trained model so that we don’t have to execute the same steps every time we receive a request from the server. This process of storing a Python object into a byte stream so that it can be utilized later is called Serialization. Deserialization, as the name suggests, is the reverse of Serialization, i.e. converting a byte stream back into a Python object. Serialization and Deserialization in Python can be done via packages like
joblib, pickle,<\/code> etc.<\/p>\nLet’s go ahead and save our model.<\/p>\n
\njoblib.dump(regression_model, 'regression_model.pkl')\nprint('Model dumped')\nregression_model = joblib.load('regression_model.pkl')\nregression_model_columns = list(x_train)\njoblib.dump(regression_model_columns, 'regression_model_columns.pkl')<\/code><\/pre>\n<\/div>\nOur file after adding those code snippets will finally look like –<\/p>\n
\nimport numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LinearRegression\nimport joblib\n\ndataset = pd.read_csv(\"bottle.csv\")\ndataset=dataset[[\"Salnty\",\"T_degC\"]]\ndataset = dataset[:500]\ndataset=dataset.dropna(axis=0)\ndataset.reset_index(drop=True,inplace=True)\n\nx_label=np.array(dataset['Salnty']).reshape(493,1)\ny_label=np.array(dataset['T_degC']).reshape(493,1)\nx_train, x_test, y_train, y_test = train_test_split(x_label, y_label, test_size = 0.2, random_state = 100)\nregression_model=LinearRegression()\nregression_model.fit(x_train,y_train)\n\njoblib.dump(regression_model, 'regression_model.pkl')\nprint('Model dumped')\nregression_model = joblib.load('regression_model.pkl')\nregression_model_columns = list(x_train)\njoblib.dump(regression_model_columns, 'regression_model_columns.pkl')<\/code><\/pre>\n<\/div>\nLet’s go ahead and save our
regression_model.py<\/code> in the same directory. To execute the code in the command line, typepython regression_model.py<\/code>. After execution, our directory structure will look like this –<\/p>\n![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2021\/07\/directory-structure.png\")
<\/a><\/p>\nThe
ipynb<\/code> file is optional if you are running your files in the terminal\/command prompt.<\/p>\nGreat! Now that our model is saved, we are all set to create an API for making real-time predictions.<\/p>\n
Developing an API using Python’s Flask<\/h2>\n
We first need to import our model and convert it into a python object when our application starts. We can do that with the following lines of code –<\/p>\n
\nlr = joblib.load(\"regression_model.pkl\")\nprint ('Model loaded')\nmodel_columns = joblib.load(\"regression_model_columns.pkl\")\nprint ('Model columns loaded')<\/code><\/pre>\n<\/div>\nNow we need to create an API endpoint that actively listens to requests, processes them in a Python object and pass them to the model for making predictions. We also need to take care of runtime exceptions in case the input is not in the desired JSON format. The following script should do the work –<\/p>\n
\n@app.route('\/predict', methods=['POST'])\ndef predict():\n if lr:\n try:\n json_ = request.json\n query = pd.DataFrame(json_)\n print(query)\n\n prediction = list(lr.predict(query))\n\n return jsonify({'prediction': str(prediction)})\n\n except:\n\n return jsonify({'trace': traceback.format_exc()})\n else:\n print ('Train the model first')\n return ('No model here to use')<\/code><\/pre>\n<\/div>\nBy default, a flask application runs on http:\/\/127.0.0.1:5000. Let’s customize this setting if the user wants to provide a specific port number during execution. The compiled file now looks like –<\/p>\n
\nfrom flask import Flask, request, jsonify\nimport joblib\nimport traceback\nimport pandas as pd\nimport numpy as np\nimport sklearn\n\n# Your API definition\napp = Flask(__name__)\n\n@app.route('\/predict', methods=['POST'])\ndef predict():\n if lr:\n try:\n json_ = request.json\n query = pd.DataFrame(json_)\n print(query)\n\n prediction = list(lr.predict(query))\n\n return jsonify({'prediction': str(prediction)})\n\n except:\n\n return jsonify({'trace': traceback.format_exc()})\n else:\n print ('Train the model first')\n return ('No model here to use')\n\nif __name__ == '__main__':\n try:\n port = int(sys.argv[1])\n except:\n port = 12345\n\n lr = joblib.load(\"regression_model.pkl\")\n print ('Model loaded')\n model_columns = joblib.load(\"regression_model_columns.pkl\")\n print ('Model columns loaded')\n\n app.run(port=port, debug=True)<\/code><\/pre>\n<\/div>\nNote that in the
load()<\/code> function, we have provided just the file name because we’re saving everything in the same directory. In case you encounter aFileNotFoundError<\/code> during compilation, provide a full path instead.<\/p>\nLet’s go ahead and compile the file to see if everything’s working correctly. We get the following output –<\/p>\n
![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2021\/07\/api-file-compile.png\")
<\/a><\/p>\nThis means that our server is now active at http:\/\/127.0.0.1:12345. We now need a third-party application to send it a request and see what output are we getting.<\/p>\n
Making real-time predictions<\/h2>\n
To send a request to our server, we will be using Postman’s Desktop application<\/a>. There are several tools available for API testing, you can use whichever you prefer. The console will typically look like –<\/p>\n
![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2021\/07\/console_postman.png\")
<\/a><\/p>\nLet’s make the following changes –<\/p>\n
- \n
- Change the method to
POST<\/code><\/li>\n- Within the
Body<\/code> tab, selectraw<\/code> andJSON<\/code> as the Request format.<\/li>\n- Pass input as a list such as
[{\"Salnty\":34}, {...}]<\/code><\/li>\n- Hit
Send<\/code><\/li>\n<\/ol>\nFew things to note<\/h3>\n
- \n
- Your
api.py<\/code> needs to run in the background, otherwise, the server will not be active and you’ll receive an error in the console.<\/li>\n- The input we’re sending is in a JSON format which uses a combination of lists and dictionaries. Therefore, it is important to pass your input in a
{key: value}<\/code> pair. In case of multiple attributes, your input will look like{\"attribute1\": value1, \"attribute2\": value2, ...}<\/code><\/li>\n<\/ol>\nWe get the following output in the console –<\/p>\n
![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2021\/07\/prediction.png\")
<\/a><\/p>\nTo double-check the input we got from the Postman console, check your terminal. It should have printed out the JSON query like –<\/p>\n
- The input we’re sending is in a JSON format which uses a combination of lists and dictionaries. Therefore, it is important to pass your input in a
- Within the
- Change the method to
![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2021\/07\/console.png\")
<\/a><\/p>\n