todo.txt - knox - Git at Google

 Security
 1) Migrate from Apache Shiro to SpringSecurity or decide to stick with Shiro.
 2) Make HTTPClient do SPNEGO with secure Hadoop gateway.
 3) Setup ApacheDS as a KDC in gateway functional tests.
 4) Figure out how to do REST SSO with ActiveDirectory+Hadoop.
 5) Figure out untrusted code isolation for windows and linux.
    5a) Is java security enough?
    5b) For linux ook into Linux Containers, VServer, SeLinux, etc.
    5c) For windows look into Job objects
 6) Figure out the general Hadoop user/group/role/priv model.  Must work with ActiveDirectory.

 Usability
 Make ** the default for path and queryParam shortcut patterns.
 This
   <source>/namenode/api/v1/{path=**}?{**}</source>
 should be
   <source>/namenode/api/v1/{path}?{**}</source>
 and

 Usability
 Eliminate the need for two webhdfs entries in the config XML.
 This will require making this
         <source>/namenode/api/v1?{**}</source>
         <target>http://{namenode.address}/webhdfs/v1/?{**}</target>
 and this
         <source>/namenode/api/v1/{path=**}?{**}</source>
         <target>http://{namenode.address}/webhdfs/v1/{path=**}?{**}</target>
 behave identically when the input URL is one of these
   http://host:port/gateway/gateway/namenode/api/v1
   http://host:port/gateway/gateway/namenode/api/v1/

 Bug
 Right now any parameter that wasn't used in an expansion by the time the query is expanded is added to the
 URL if there is a {**} in the template.  This should only expand unused query parameters.  It also doesn't
 take into account anything that might later be expanded in the fragment.  So
 (1) Need to know where parameters came from.
 (2) Need to do explicit expansion of query and fragment before implicit expansion.

 Design
 If all query templates are optional make sure there is no ? at the end of the

 Feature (IMPORTANT)
 Need to be able to populate the user.name query parameter with the authenticated LDAP user name.

 Demo
 Finish adding Jobtracker and Tasktracker to the demo.  Docment the same way as HDFS was done on gdocs.

 Limitation
 It looks like a fragment can contain name values pairs much like a query.

 Usability
 Need to be able to more easily create a template that matches a full URL but only cares about the path for example.
 Right now this will need to look something like this *://*:*@*:*/**?**#**
 I'm not even sure how this will handle the query and fragment sections.
 To match a real full URL would this be required or even currently work? *://*:*@*:*/**?**#*

 Usability
 Sort out the handling of * vs ** for match and expand.

 Security (IMPORTANT)
 A HTTP BasicAuth challenge must never be issued over an insecure transport.
 Probably need to put a filter in place before shiro/spring-security to enforce this.
 Might just want to put this in as a "global" filter in front of the org.apache.org.apache.hadoop.gateway servlet.

 Feature (IMPORTANT)
 Need to be able to use something from the authentication to map the authenticated use to a Hadoop user.

 Feature (IMPORTANT)
 Need to setup SSL for the org.apache.org.apache.hadoop.gateway.
 This will require some way to do porotocl mediation from HTTPS -> HTTP.

 Configuration
 Figure out how to generate the shiro.ini file instead of getting it from the classpath.
 Most important so that we don't hard code any network addressed into classpath resources.

 Commit
 Move System.out to debug messages.

 Testing
 Write unit tests and verify via Cobertura coverage reports.

 Testing
 Resolve all FindBugs issue.

 Demo
 Get the file browsing stuff to work in the demo.

 Usability
 Need to handle the <source> value starting with/without a slash better.

 Usability
 Consider changing <service> to <resource>.

 Usability
 Really need to get rig of the <rewrite.n> stuff.  It might make more sense to have either <rule> or <url>.
 In any case give some thought to having some form of "param rule" embedded within a "url rule" for
 manipulating the query parameters.

 Security
 For nn_browsedfscontent.jsp need to provide a delegation parameter.
 Q: What is the relationship between the org.apache.hadoop.auth cookie and the delegation parameter?
 In general delegation parameters are provided by the server as a result of a redirect from a service protected via SPNEGO.
 This might be useful in creating a delegation parameter.
 //      Subject s = Subject.getSubject( AccessController.getContext() );
 //      String u = s.getPublicCredentials(UsernamePasswordCredentials.class).iterator().next().getUserName();
 //      String p = s.getPrincipals(KerberosPrincipal.class).iterator().next().getName();
 //      AuthenticationToken authToken = new AuthenticationToken(
 //        u, p, "kerberos" );
 //      authToken.setExpires( Long.MAX_VALUE );
 //      String raw = authToken.toString();
 //      String sig = signer.sign( raw );
 //      signer.verifyAndExtract( sig );
 //
 //      token = new AuthenticatedURL.Token( sig );
 //      conn = new AuthenticatedURL().openConnection( changesUrl, token );

 Usability
 This needs to automatically add the org.apache.hadoop.auth cookie if there is an authenticated subject.
 The test for this is that you should be able to browse the data node file system.

 Feature
 Should be able to redirect to a HTTPS login form to allow the rest of the console to be HTTP.
 For REST APIs this shouldn't happen.  If the transport isn't secure they should just fail.

 Feature
 Will need to mediate between HTTPS on the inbound side and HTTP on the outbound side.
 The hardest part might just be the URL translation.

 Security
 Make sure that the challenge filter will only challenge over a secure transport.

 Configuration
 Build into the org.apache.org.apache.hadoop.gateway wrapper the ability to take either a URL or a config string as an init parameter to create
 the chain chain.

 Feature
 Name for logging stuff.  Use terms "contract based", "resource", "message".

 Hygiene
 Convert all System.outs to messages.

 Design
 Change HttpUrlRewriter to just UrlRewriterFilter and have it handle cookies, headers and parameters.
 On the input and output side create a registry of mime types to stream url rewriters

 Design
 Modify stream rewriter by passing an UrlRewriter interface down to the chain.

 Design
 For logging create separate modules for each logging framework integration.

 Usability
 Enhance the config digester to uses object factories so that instead of <filter> a more meaningful thing like
 <trace> could be used.  This would just set the name and class params automatically.  There would need to be
 a registry of factories based on tag name.

 keytool -keystore client-keystore.jks -alias client -genkey -keyalg RSA
 keytool -keystore server-keystore.jks -alias server -genkey -keyalg RSA
 keytool -export -alias client -keystore client-keystore.jks -rfc -file client.cer -storepass horton
 keytool -export -alias server -keystore server-keystore.jks -rfc -file server.cer -storepass horton
 keytool -import -alias client -file client.cer -keystore server-truststore.jks –storepass horton
 keytool -import -alias server -file server.cer -keystore client-truststore.jks
	Security
	1) Migrate from Apache Shiro to SpringSecurity or decide to stick with Shiro.
	2) Make HTTPClient do SPNEGO with secure Hadoop gateway.
	3) Setup ApacheDS as a KDC in gateway functional tests.
	4) Figure out how to do REST SSO with ActiveDirectory+Hadoop.
	5) Figure out untrusted code isolation for windows and linux.
	5a) Is java security enough?
	5b) For linux ook into Linux Containers, VServer, SeLinux, etc.
	5c) For windows look into Job objects
	6) Figure out the general Hadoop user/group/role/priv model. Must work with ActiveDirectory.

	Usability
	Make ** the default for path and queryParam shortcut patterns.
	This
	<source>/namenode/api/v1/{path=}?{}</source>
	should be
	<source>/namenode/api/v1/{path}?{**}</source>
	and

	Usability
	Eliminate the need for two webhdfs entries in the config XML.
	This will require making this
	<source>/namenode/api/v1?{**}</source>
	<target>http://{namenode.address}/webhdfs/v1/?{**}</target>
	and this
	<source>/namenode/api/v1/{path=}?{}</source>
	<target>http://{namenode.address}/webhdfs/v1/{path=}?{}</target>
	behave identically when the input URL is one of these
	http://host:port/gateway/gateway/namenode/api/v1
	http://host:port/gateway/gateway/namenode/api/v1/

	Bug
	Right now any parameter that wasn't used in an expansion by the time the query is expanded is added to the
	URL if there is a {**} in the template. This should only expand unused query parameters. It also doesn't
	take into account anything that might later be expanded in the fragment. So
	(1) Need to know where parameters came from.
	(2) Need to do explicit expansion of query and fragment before implicit expansion.

	Design
	If all query templates are optional make sure there is no ? at the end of the

	Feature (IMPORTANT)
	Need to be able to populate the user.name query parameter with the authenticated LDAP user name.

	Demo
	Finish adding Jobtracker and Tasktracker to the demo. Docment the same way as HDFS was done on gdocs.

	Limitation
	It looks like a fragment can contain name values pairs much like a query.

	Usability
	Need to be able to more easily create a template that matches a full URL but only cares about the path for example.
	Right now this will need to look something like this ://:@:/?#*
	I'm not even sure how this will handle the query and fragment sections.
	To match a real full URL would this be required or even currently work? ://:@:/?#

	Usability
	Sort out the handling of * vs ** for match and expand.

	Security (IMPORTANT)
	A HTTP BasicAuth challenge must never be issued over an insecure transport.
	Probably need to put a filter in place before shiro/spring-security to enforce this.
	Might just want to put this in as a "global" filter in front of the org.apache.org.apache.hadoop.gateway servlet.

	Feature (IMPORTANT)
	Need to be able to use something from the authentication to map the authenticated use to a Hadoop user.

	Feature (IMPORTANT)
	Need to setup SSL for the org.apache.org.apache.hadoop.gateway.
	This will require some way to do porotocl mediation from HTTPS -> HTTP.

	Configuration
	Figure out how to generate the shiro.ini file instead of getting it from the classpath.
	Most important so that we don't hard code any network addressed into classpath resources.

	Commit
	Move System.out to debug messages.

	Testing
	Write unit tests and verify via Cobertura coverage reports.

	Testing
	Resolve all FindBugs issue.

	Demo
	Get the file browsing stuff to work in the demo.

	Usability
	Need to handle the <source> value starting with/without a slash better.

	Usability
	Consider changing <service> to <resource>.

	Usability
	Really need to get rig of the <rewrite.n> stuff. It might make more sense to have either <rule> or <url>.
	In any case give some thought to having some form of "param rule" embedded within a "url rule" for
	manipulating the query parameters.

	Security
	For nn_browsedfscontent.jsp need to provide a delegation parameter.
	Q: What is the relationship between the org.apache.hadoop.auth cookie and the delegation parameter?
	In general delegation parameters are provided by the server as a result of a redirect from a service protected via SPNEGO.
	This might be useful in creating a delegation parameter.
	// Subject s = Subject.getSubject( AccessController.getContext() );
	// String u = s.getPublicCredentials(UsernamePasswordCredentials.class).iterator().next().getUserName();
	// String p = s.getPrincipals(KerberosPrincipal.class).iterator().next().getName();
	// AuthenticationToken authToken = new AuthenticationToken(
	// u, p, "kerberos" );
	// authToken.setExpires( Long.MAX_VALUE );
	// String raw = authToken.toString();
	// String sig = signer.sign( raw );
	// signer.verifyAndExtract( sig );
	//
	// token = new AuthenticatedURL.Token( sig );
	// conn = new AuthenticatedURL().openConnection( changesUrl, token );

	Usability
	This needs to automatically add the org.apache.hadoop.auth cookie if there is an authenticated subject.
	The test for this is that you should be able to browse the data node file system.

	Feature
	Should be able to redirect to a HTTPS login form to allow the rest of the console to be HTTP.
	For REST APIs this shouldn't happen. If the transport isn't secure they should just fail.

	Feature
	Will need to mediate between HTTPS on the inbound side and HTTP on the outbound side.
	The hardest part might just be the URL translation.

	Security
	Make sure that the challenge filter will only challenge over a secure transport.

	Configuration
	Build into the org.apache.org.apache.hadoop.gateway wrapper the ability to take either a URL or a config string as an init parameter to create
	the chain chain.

	Feature
	Name for logging stuff. Use terms "contract based", "resource", "message".

	Hygiene
	Convert all System.outs to messages.

	Design
	Change HttpUrlRewriter to just UrlRewriterFilter and have it handle cookies, headers and parameters.
	On the input and output side create a registry of mime types to stream url rewriters

	Design
	Modify stream rewriter by passing an UrlRewriter interface down to the chain.

	Design
	For logging create separate modules for each logging framework integration.

	Usability
	Enhance the config digester to uses object factories so that instead of <filter> a more meaningful thing like
	<trace> could be used. This would just set the name and class params automatically. There would need to be
	a registry of factories based on tag name.

	keytool -keystore client-keystore.jks -alias client -genkey -keyalg RSA
	keytool -keystore server-keystore.jks -alias server -genkey -keyalg RSA
	keytool -export -alias client -keystore client-keystore.jks -rfc -file client.cer -storepass horton
	keytool -export -alias server -keystore server-keystore.jks -rfc -file server.cer -storepass horton
	keytool -import -alias client -file client.cer -keystore server-truststore.jks –storepass horton
	keytool -import -alias server -file server.cer -keystore client-truststore.jks