Hadoop and s3a

Last Updated: May 02, 2019 07:49AM PDT

NooBaa supports S3A API, which is used by Hadoop and others. 
Example configuration for Hadoop can be found below. Please replace <NooBaa endpoint> <NooBaa bucket> <NooBaa access key> and <NooBaa secret key> with your NooBaa system. You can get it by clicking on Connect Application button in the main window. More information can be found in Connect Application article

 noobaa-s3a-config.xml:

<configuration>

    <property>
        <name>fs.defaultFS</name>
        <value>s3a://<NooBaa bucket></value>
    </property>
    <property>
        <name>fs.s3a.endpoint</name>
        <value><NooBaa endpoint></value>
    </property>
    <property>
        <name>fs.s3a.connection.ssl.enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>fs.s3a.path.style.access</name>
        <value>true</value>
    </property>

    <property>
        <name>fs.s3a.access.key</name>
        <value><NooBaa access key></value>
    </property>
    <property>
        <name>fs.s3a.secret.key</name>
        <value><NooBaa secret key></value>
    </property>

</configuration>



Quickly test it with docker and NooBaa. Simply upload a text file to your <NooBaa bucket> and run the following commands 

This command will list the objects in your <NooBaa bucket>

docker run -it --rm --name hadoop \
  -e HADOOP_OPTIONAL_TOOLS="hadoop-aws" \
  -v $PWD/noobaa-s3a-config.xml:/etc/hadoop/core-site.xml \
  bde2020/hadoop-namenode:2.0.0-hadoop3.1.1-java8 -- \
  hadoop fs -ls /



This command will count words of <file.txt> and output the results to a folder in your <NooBaa bucket>/wordcount

docker run -it --rm --name hadoop \
  -e HADOOP_OPTIONAL_TOOLS="hadoop-aws" \
  -v $PWD/noobaa-s3a-config.xml:/etc/hadoop/core-site.xml \
  bde2020/hadoop-namenode:2.0.0-hadoop3.1.1-java8 -- \
   hadoop jar /opt/hadoop-3.1.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.jar wordcount /<file.txt> /wordcount





 
Did you find this article helpful?
195e69fe2c7a9be6b2b2aa6b87f970548@noobaa.desk-mail.com
https://cdn.desk.com/
false
desk
Loading
seconds ago
a minute ago
minutes ago
an hour ago
hours ago
a day ago
days ago
about
false
Invalid characters found
/customer/en/portal/articles/autocomplete