Using Spring-XD to Load Files into GemFire XD

Spring XD Scripts

My General Setup Script (I save it in setup.xd and load it via  script –file setup.xd)

had config fs –namenode hdfs://pivhdsne:8020
admin config server http://localhost:9393
hadoop fs ls /
stream list

The Script for Loading a File into GemFireXD via Spring-XD

stream create --name fileload --definition "file --dir=/tmp/xd/input/load --outputType=text/plain |  jdbc --tableName=APP.filetest --columns=id,name" --deploy

 

Spring XD Configuration for GemFire XD

Copy the GemFire XD JDBC Driver to Spring-XD (might need tools.jar as well)

cp /usr/lib/gphd/Pivotal_GemFireXD_10/lib/gemfirexd-client.jar /opt/pivotal/spring-xd/xd/lib/

Modify the Sink’s JDBC properties to point to your Gemfire XD, if you are using the Pivotal HD VM and install Spring-XD with Yum (sudo yum update spring-xd), this is the location:

/opt/pivotal/spring-xd/xd/config/modules/sink/jdbc/jdbc.properties
url = jdbc:gemfirexd://localhost:1527
username = gfxd
password = gfxd
driverClassName = com.pivotal.gemfirexd.jdbc.ClientDriver

For Peer Client Driver you need more files from GemFireXD Lib (the .so binaries), linking is probably a good idea.

 

GemFire XD Setup

gfxd
connect client 'localhost:1527';

create table filetest (id int, name varchar(100)) REPLICATE PERSISTENT;
select id, kind, netservers from sys.members; 
select * from filetest;

Spring XD Commands

stream list

show your jobs

 

Reference:

 

Leave a Reply