Connect to SFTP server - Scala Solution

spark-shell --packages com.springml:spark-sftp_2.11:1.1.3

import org.apache.commons.net.ftp.FTPClient
import org.apache.commons.net.ftp.FTPReply
import org.apache.commons.net.ftp.FTPSClient
import com.jcraft.jsch._

val jsch = new JSch()
var session = jsch.getSession("test","securetransfertest.com",22)
session.setPassword("test123")

var config = new java.util.Properties()
config.put("StrictHostKeyChecking", "no")
session.setConfig(config)

session.connect()

var channel = session.openChannel("sftp").asInstanceOf[ChannelSftp]
channel.connect()
println("Directory:" + channel.pwd())  // This prints the present working directory

session.disconnect()

//jsch.addIdentity(privatekeyfile);

Further research need to be done to put/get files from sftp server.

Connect to Hive with Kerberos Set up from a Local PC

The below approach can be used to connect to a hive instance with kerberos authentication enabled.

One can use tools like DBVisualizer , DBeaver etc.

Check Java Version:-

C:\Program Files\DbVisualizer\jre\bin>.\java.exe -version

openjdk version "1.8.0_212"

OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_212-b03)

OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.212-b03, mixed mode)

Use "kinit" to generate kerberos ticket

C:\Program Files\DbVisualizer\jre\bin>.\kinit ksumanam@ABC.COM

Password for ksumanam@ABC.COM:

New ticket is stored in cache file C:\Users\ksumanam\krb5cc_ksumanam

Use "klist" to check the ticket validity

C:\Program Files\DbVisualizer\jre\bin>klist

Credentials cache: C:\Users\ksumanam\krb5cc_ksumanam

Default principal: ksumanam@ABC.COM, 1 entry found.

[1]  Service Principal:  krbtgt/ABC.COM@ABC.COM

     Valid starting:     May 11, 2020 09:53:43

     Expires:            May 11, 2020 19:53:43

If Using DbVisualizer - Add the below (If other tools add the below in the relevant .ini files)

Click on Tools - Tool Properties and add below details. In box Specify overridden JAVA VM properties here

-Dsun.security.krb5.debug=true

-Djavax.security.auth.useSubjectCredsOnly=false

-Djava.security.krb5.conf=C:\Users\ksumanam\krb5.ini  -- See below for file content.

Sample krb5.ini File (This is the same file krb5.conf copied from Unix)

[libdefaults]

 renew_lifetime = 7d

 forwardable = true

 default_realm = ABC.COM

 ticket_lifetime = 24h

 dns_lookup_realm = false

 dns_lookup_kdc = false

 udp_preference_limit = 1

#default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5

#default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5

[domain_realm]

abc.com = ABC.COM

.abc.com = ABC.COM

[logging]

default = FILE:/var/log/krb5kdc.log

admin_server = FILE:/var/log/kadmind.log

kdc = FILE:/var/log/krb5kdc.log

[realms]

ABC.COM = {

  admin_server = abc.com

  kdc = abc.com

}

Create a new connection on DbVisualizer(One might need the below details)

Database server = hn1-ahd701.abc.com

Database Port = 10001

Database = default;principal=hive/_HOST@ABC.COM; transportMode=http

JDBC String:

jdbc:hive2://hn1-ahd703.abc.com:10001/dl_explr_india;principal=hive/_HOST@ABC.COM;transportMode=http

Example beeline jdbc URL

beeline -u 'jdbc:hive2://hn1-ahd701.abc.com:10001/default;principal=hive/_HOST@ABC.COM;auth=kerberos;transportMode=http;httpPath=cliservice'