Trying out Tachyon on Hadoop 2.2

Watch the Videos

Tachyon Resources:
Tachyon with Spark
Interview with Tachyon Lead
Reliable File Sharing in Memory
Haoyuan Li (Project Lead and Co-Creator)
Future Architecture of a Data Lake in Memory
Tachyon: Memory Throughput I/O for Cluster Computing Frameworks
Tachyon meetup
Github for Haoyuan Li
Tachyon Github
xPatterns on Spark/Shark/Tachyon/Mesos
Amplab
Article on Tachyon

Pivotal hosted a talk on Tachyon (github), so I wanted to try it out first.  Turns out it is really easy to install and use, so here’s how.  I am running using the local directions.

To Install and run Tachyon on PIVHDSNE_VMWARE_VM-2.0.0-52 (Pivotal Hadoop Single Node VM 64-bit RHEL 6)

Download it

wget https://github.com/amplab/tachyon/releases/download/v0.5.0/tachyon-0.5.0-bin.tar.gz

Build It

mvn -Dhadoop.version=2.2.0 clean package

make sure it’s built with Apache Hadoop 2.2 (PHD 2.0)

Format, Run it and Run Some Tests

[pivhdsne:tachyon-0.5.0]$ ./bin/tachyon format
Connection to localhost... Formatting Tachyon Worker @ pivhd.localdomain Removing local data under folder: /mnt/ramdisk/tachyonworker/ Connection to localhost closed. Formatting Tachyon Master @ localhost Formatting JOURNAL_FOLDER: /home/gpadmin/research/tachyon-0.5.0/libexec/../journal/ Formatting UNDERFS_DATA_FOLDER: /home/gpadmin/research/tachyon-0.5.0/libexec/../underfs/tmp/tachyon/data Formatting UNDERFS_WORKERS_FOLDER: /home/gpadmin/research/tachyon-0.5.0/libexec/../underfs/tmp/tachyon/workers

[pivhdsne:tachyon-0.5.0]$ ./bin/tachyon-start.sh local
Killed 1 processes
Killed 1 processes
Connection to localhost... Killed 0 processes
Connection to localhost closed.
Formatting RamFS: /mnt/ramdisk (1gb)
Starting master @ localhost
Starting worker @ pivhd.localdomain

./bin/tachyon runTests

make sure everything is working, takes some time

Results of One of the Tests

/home/gpadmin/research/tachyon-0.5.0/bin/tachyon runTest Basic THROUGH
/BasicFile_THROUGH has been removed
2014-10-15 16:41:03,987 INFO   (TachyonFS.java:connect) - Trying to connect master @ localhost/127.0.0.1:19998
2014-10-15 16:41:04,017 INFO   (MasterClient.java:getUserId) - User registered at the master localhost/127.0.0.1:19998 got UserId 14
2014-10-15 16:41:04,018 INFO   (TachyonFS.java:connect) - Trying to get local worker host : localhost
2014-10-15 16:41:04,025 INFO   (TachyonFS.java:connect) - Connecting local worker @ localhost/127.0.0.1:29998
2014-10-15 16:41:04,044 INFO   (CommonUtils.java:printTimeTakenMs) - createFile with fileId 26 took 58 ms.
2014-10-15 16:41:04,093 INFO   (TachyonFile.java:readRemoteByteBuffer) - Try to find and read from remote workers.
2014-10-15 16:41:04,093 INFO   (TachyonFile.java:readRemoteByteBuffer) - readByteBufferFromRemote() [NetAddress(mHost:localhost, mPort:-1)]
2014-10-15 16:41:04,107 INFO   (TachyonFS.java:createAndGetUserTempFolder) - Folder /mnt/ramdisk/tachyonworker/users/14 was created!
2014-10-15 16:41:04,110 INFO   (BlockOutStream.java:<init>) - /mnt/ramdisk/tachyonworker/users/14/27917287424 was created!
Passed the test!

Checks the logs directory

-rw-r--r-- 1 gpadmin gpadmin 44958 Oct 15 17:18 worker.log@127.0.0.1_10-15-2014
-rw-r--r-- 1 gpadmin gpadmin  9610 Oct 15 17:18 master.log@127.0.0.1_10-15-2014
-rw-r--r-- 1 gpadmin gpadmin   500 Oct 15 17:17 user.log@127.0.0.1_10-15-2014
-rw-r--r-- 1 gpadmin gpadmin   778 Oct 15 17:17 user.log@127.0.0.1_10-15-2014_9
-rw-r--r-- 1 gpadmin gpadmin   500 Oct 15 17:16 user.log@127.0.0.1_10-15-2014_8
-rw-r--r-- 1 gpadmin gpadmin   500 Oct 15 17:16 user.log@127.0.0.1_10-15-2014_7
-rw-r--r-- 1 gpadmin gpadmin   500 Oct 15 17:13 user.log@127.0.0.1_10-15-2014_6
-rw-r--r-- 1 gpadmin gpadmin   778 Oct 15 17:13 user.log@127.0.0.1_10-15-2014_5
-rw-r--r-- 1 gpadmin gpadmin   500 Oct 15 17:13 user.log@127.0.0.1_10-15-2014_4
-rw-r--r-- 1 gpadmin gpadmin   500 Oct 15 17:13 user.log@127.0.0.1_10-15-2014_3
-rw-r--r-- 1 gpadmin gpadmin   500 Oct 15 17:12 user.log@127.0.0.1_10-15-2014_2
-rw-r--r-- 1 gpadmin gpadmin   500 Oct 15 17:12 user.log@127.0.0.1_10-15-2014_1
-rw-r--r-- 1 gpadmin gpadmin  2337 Oct 15 16:00 master.log@127.0.0.1_10-15-2014_1

Test Via URL (http://localhost:19999/home)

Tachyon UI

Logs

image004

Create a Directory

 [pivhdsne:tachyon-0.5.0]$ ./bin/tachyon tfs mkdir /xd
Successfully created directory /xd

View File System

[pivhdsne:tachyon-0.5.0]$ ./bin/tachyon tfs ls /
80.00 B   10-15-2014 16:41:01:053  In Memory      /BasicFile_MUST_CACHE
0.00 B    10-15-2014 16:41:01:560                 /BasicRawTable_MUST_CACHE
80.00 B   10-15-2014 16:41:02:069  In Memory      /BasicFile_TRY_CACHE
0.00 B    10-15-2014 16:41:02:532                 /BasicRawTable_TRY_CACHE
80.00 B   10-15-2014 16:41:03:032  In Memory      /BasicFile_CACHE_THROUGH
0.00 B    10-15-2014 16:41:03:531                 /BasicRawTable_CACHE_THROUGH
80.00 B   10-15-2014 16:41:04:042  In Memory      /BasicFile_THROUGH
0.00 B    10-15-2014 16:41:04:528                 /BasicRawTable_THROUGH
80.00 B   10-15-2014 16:41:05:055  In Memory      /BasicFile_ASYNC_THROUGH
0.00 B    10-15-2014 16:41:05:525                 /BasicRawTable_ASYNC_THROUGH
0.00 B    10-15-2014 16:53:56:062                 /xd

Load a JSON File

[pivhdsne:tachyon-0.5.0]$ ./bin/tachyon tfs copyFromLocal /tmp/xd/test.json /xd/load/test.json
Copied /tmp/xd/test.json to /xd/load/test.json

[pivhdsne:tachyon-0.5.0]$ ./bin/tachyon tfs ls /xd/load
449.48 KB 10-15-2014 17:17:03:489  In Memory      /xd/load/test.json

image006

image008

Leave a Reply