{"id":46637,"date":"2021-03-18T00:00:00","date_gmt":"2021-03-18T07:00:00","guid":{"rendered":"https:\/\/griddb-linux-hte8hndjf8cka8ht.westus-01.azurewebsites.net\/blog\/use-machine-learning-and-griddb-to-build-a-production-ready-stock-market-anomaly-detector\/"},"modified":"2025-11-13T12:55:14","modified_gmt":"2025-11-13T20:55:14","slug":"use-machine-learning-and-griddb-to-build-a-production-ready-stock-market-anomaly-detector","status":"publish","type":"post","link":"https:\/\/griddb-linux-hte8hndjf8cka8ht.westus-01.azurewebsites.net\/en\/blog\/use-machine-learning-and-griddb-to-build-a-production-ready-stock-market-anomaly-detector\/","title":{"rendered":"Use Machine Learning and GridDB to build a Production-Ready Stock Market Anomaly Detector"},"content":{"rendered":"

In this project, we use GridDB to create a Machine Learning platform where Kafka is used to import stock market data from Alphavantage<\/a>, a market data provider. Tensorflow<\/a> and Keras<\/a> train a model that is then stored in GridDB, and then finally uses LSTM prediction to find anomalies in daily intraday trading history. The last piece is that the data is visualized in Grafana and then we configure GridDB to send notifications via its REST Trigger function to Twilio’s Sendgrid.<\/p>\n

The actual machine learning portion of this project was inspired by posts on Towards Data Science<\/a> and Curiously<\/a>. This model and the data flow is also applicable to many other datasets such as predictive maintenance or machine failure prediction or wherever you want to find anomalies in time series data. An anomaly as it relates to machine learning is when the predicted value is significantly different than the actual value, in this case stock price anomalies may mean there are good trading opportunities but anomalies in sensor data from an engine may mean failure is imminent.<\/p>\n

The following GridDB features are showcased: – Key-Container data model – kafka-connect-jdbc & JDBC – Pandas Data Frames – Grafana Connector – Triggers<\/p>\n

GridDB’s remarkable write performance is necessary as hundreds of thousands of rows of data for just a few stocks are loaded daily after the market close. It’s Key-Container architecture means queries for individual stocks are efficient, it is not necessary to scan GOOG’s intraday data to find the price anomalies in AAPL’s stocks. If you do not already have GridDB set up, follow the Getting Started<\/a> manual to first install GridDB.<\/p>\n

Kafka<\/h1>\n

Kafka<\/a> is a data streaming platform with many different possible inputs and outputs that are easy to create. Kafka can be downloaded from their downloads page<\/a>; we’re using version 2.12-2.5.0. You will also need to have a Java 1.8 development environment installed on your system. After downloading, we simply untar and start the Zookeeper and Kafka Servers.<\/p>\n

$ tar xzvf kafka_2.12-2.5.0.tgz\n$ cd kafka_2.12-2.5.0\n$ export PATH=$PATH:\/path\/to\/kafka_2.12-2.5.0\/bin\n$ zookeeper-server-start.sh  --daemon config\/zookeeper.properties\n$ kafka-server-start.sh --daemon config\/server.properties\n<\/code><\/pre>\n
We will use Kafka Connect JDBC, a connector that allows Kafka to write to many different JDBC databases. We’ve already added GridDB support to the connector, but it is only available from our own Github repository. Download the JAR<\/a><\/p>\n
We will also need to use the gridstore-jdbc.jar<\/a> from GridDB.net’s Github repository. It has implemented additional JDBC functions requiured by the Kafka Connect JDBC connector.<\/p>\n
Once you’ve downloaded both JARs, place them in \/path\/to\/kafka_2.12-2.5.0\/libs\/.<\/p>\n
Now we’ll create a config file for kafka-connect-jdbc, configs\/jdbc-connect.json. The config files tells the connector what JDBC URL to connect to, the topics to listen to, and how to transform the string time field to an actual timestamp used by the database.<\/p>\n