Integrating Apache Storm with Splice Machine

This topic walks you through building and running two different examples of integrating Storm with Splice Machine:

Inserting Random Values in Splice Machine with Storm

This example iterates through a list of random words and numbers, inserting the values into a Splice Machine database; follow along in these steps:

  1. Download the Sample Code:

    Pull the code from our git repository:

    https://github.com/splicemachine/splice-community-sample-code/tree/master/tutorial-storm
    
  2. Check the prerequisites:

    You must be running Splice Machine 2.0 or later on the same machine on which you will run this example code. If Splice Machine is running on a different machine, you’ll need to modify the server variable in the SpliceDumperTopology.java file; change it to the name of the server that is running Splice Machine.

    You also must have maven v.3.3.9 or later installed.

  3. Create the test table in Splice Machine:

    This example inserts data into a Splice Machine table named testTable. You need to create that table by entering this command at the splice> prompt:

    splice> CREATE TABLE testTable( word VARCHAR(100), number INT );
    
  4. Compile the sample code:

    Compile the sample code with this command

    % mvn clean compile dependency:copy-dependencies
    
  5. Run the sample code:

    Follow these steps:

    1. Make sure that Splice Machine is running and that you have created the testTable table.
    2. Execute this script to run the program:

      % run-storm.sh
      
    3. Query testTable in Splice Machine to verify that it has been populated with random words and numbers:

      splice> select * from testTable;
      

About the Sample Code Classes

The random insertion example contains the following java classes, each of which is described below:

Class Description
SpliceCommunicator.java Contains methods for communicating with Splice Machine.
SpliceConnector.java Establishes a JDBC connection with Splice Machine.
SpliceDumperBolt.java Dumps data into Splice Machine.
SpliceDumperTopology.java Defines the Storm topology for this example.
SpliceIntegerSpout.java Emits tuples that are inserted into the Splice Machine table.

Inserting Data into Splice Machine from MySQL

This example uses Storm to read data from a MySQL database, and insert that data into a table in Splice Machine.

  1. Download the Sample Code:

    Pull the code from our git repository:

    https://github.com/splicemachine/splice-community-sample-code/tree/master/tutorial-storm
    
  2. Check the prerequisites:

    You must be running Splice Machine 2.0 or later on the same machine on which you will run this example code. If Splice Machine is running on a different machine, you’ll need to modify the server variable in the MySqlToSpliceTopology.java file; change it to the name of the server that is running Splice Machine.

    You also must have maven v.3.3.9 or later installed.

    This example assumes that your MySQL database instance is running on the same machine on which you’re running Splice Machine, and that the root user does not have a password. If either of these is not true, then you need to modify the call to the seedBufferQueue method in the MySqlSpout.java file. This method takes four parameters that you may need to change:

    seedBufferQueue( MySqlServer, MySqlDatabase, mySqlUserName, mySqlPassword );
    

    The default settings used in this example are:

    seedBufferQueue( "localhost", "test", "root", "" );
    
  3. Create the students table in Splice Machine:

    This example inserts data into a Splice Machine table named students. You need to create that table by entering this command at the splice> prompt:

    splice> CREATE TABLE students( name VARCHAR(100) );
    
  4. Create the students table in your MySQL database:

    This example read data from a MySQL table named students. You need to create that table in MySQL:

    $$ CREATE TABLE students( id INTEGER, name VARCHAR(100) );
    

    If your MySQL instance is on a different machine

  5. Compile the sample code:

    Compile the sample code with this command

    % mvn clean compile dependency:copy-dependencies
    
  6. Run the sample code:

    Follow these steps:

    1. Make sure that Splice Machine is running and that you have created the testTable table.
    2. Execute this script to run the program:

      % run-mysql-storm.sh
      
    3. Query the students table in Splice Machine to verify that it has been populated with data from the MySQL table:

      splice> select * from students;
      

About the Sample Code Classes

This example contains the following java classes:

Class Description
MySqlCommunicator.java Contains methods for communicating with MySQL.
MySqlConnector.java Establishes a JDBC connection with MySQL.
MySqlSpliceBolt.java Dumps data from MySQL into Splice Machine.
MySqlSpout.java Emits tuples from MySQL that are inserted into the Splice Machine table.
MySqlToSpliceTopology.java Defines the Storm topology for this example.
SpliceCommunicator.java Contains methods for communicating with Splice Machine.
SpliceConnector.java Establishes a JDBC connection with Splice Machine.