musings between the lines

there's more to life than code

development diary - Jan 2014

| Comments

January 2014

I have no idea how short or long these things will be, but I thought it’ll be a nice attempt to keep myself honest and maintain a development diary so that I’ll be forced to account for my time and actions over the course of the year, one month at a time.

Some items may be a bit cryptic since there will be some projects I’m working on that I’m not quite ready to talk about in specifics, but at least it’ll get some sort of mention. Either way, a progress report is due, so let’s start with January of 2014.

  • core : I’ll cheat a little and bleed into December, but basically I’ve wrapped up some of the major features of the system that I wanted to have in place. It’s working, but as usual, not quite as well as I like. The basics are there, but it’s far from intuitively usable, hence no real announcement. I still need to properly build out a new user module separate from the core module that will stack on top to give the foundation for genuinely starting a bootstrap site.

  • authentication : Not too much here. Just a little cleanup to fix some code that broke the Persona system from working right. Actually, it was the sample that wasn’t working. The backend code catching the Persona requests were just fine.

  • project “ab” : We sat down for an initial chat about what we wanted to get out of the project. Got some servers and logins out of the way and wrote up a quick milestone chart to see when we’ll get to some interesting publicizable code. Let’s see how it goes. Lots to do here.

  • misc : Getting my thoughts in order for what type of projects I want to tackle. I have quite a few in mind, but making myself narrow it down to a handful that I can work on. Some will be just casual tinkering on the side and others will require a little bit more effort. Will have to be careful not to have them bleed out too much time from each other.

I have no idea how long this will last. Image sourced from Red Bonzai

user profiling

| Comments

User Profiling

Nearly every system that has any type of persistent user identification needs a profiling system. I’m in the process of writing a framework (more on that at another time) that will let me kickstart future Java projects with a sound base like data storage and retrieval, json based REST APIs, and agnostic web framework containment. So naturally, a starter base for a user system would be something nice to have.

The question then becomes how to write it in a generic manner that wont be too specific to a particular instance of an application but also not too generic that it starts to become too nebulous and not tight enough for use. So choice have to be made.

Part of that will be that the user subsystem will probably be an abstract implementation. It’ll contain the basics that every profiling system should have, and then leave the details of the rest to a higher level implementation. This should achieve the goal of saving time getting the structure in place and the rest becomes filling in the implementation details. The reason for even wanting a user subsystem in first place is to start to enable some basics editing authorizations for the other systems in the core framework. I’m hoping that having, at least in abstract form, a more concrete user system will allow me to start to issue these authorizations at a lower framework level that can then float up higher as more pieces are implemented. Less work later on if a foundation is laid for the basics. Or something like that.

Pieces

I just wanted to start to review the general pieces of a usable user subsystem and what it should have in order to act as a reasonable base for most projects. I’m willing to have it be a little opinionated in order to satisfy most projects that might need this type of thing, but also willingly exclude some the edge case projects. I figured with those, I can actually fork and refactor the code as needed instead of extending and implementing it.

I’ve designed and built profiling systems in the past, but they were rebuilt each time for each project as needed. Also, being internal to a company, there were vastly different data association requirements (no need for name/email verification, tertiary data store for person information, etc).

At the moment, this subsystem is just getting off the ground, so I haven’t decided yet how granular it will be. Do I store some very generic preferences with the main object? Do I split those out to a separate named object? A generic preference association system? How detailed should it become - track login times? Track it with location information? Keep a history of all login timestamps and duration?

It’s going to be useless to accommodate all possible combinations between projects that want that type of granular information and others that don’t, hence I’ll probably just decide on what most of my projects may need and start from there. In the end, if I build into it a concept of a generic preference storage system, it may suffice for most cases. We’ll see.

Design

So, here are some of the data points about a user that would be convenient to have in a profiling system:

  • Unique System ID - Internal identifier
  • Unique Username - External identifier
  • Display Name - Common societal name
  • Email - More for contact/news delivery purposes
  • Email Verified - Just to make sure the user is who they say they are
  • Is Admin - Simple administrator flag for special access
  • Date Created - Date account was created
  • Date Updated - Date account was updated by the user
  • Date Touched - Date account was accessed by the user
  • Date Deleted - Date of deletion request (if deletion is delayed or if the account is simply removed from visibility)
  • Date Logged In - Date of last login.
  • Associated Accounts - 3rd party accounts used for login. From the Authentication Module
  • Preferences - Simple key/value preference pair storage system

Kill it with fire

One normally salient piece of information missing is “Password”. I’m not going to use one, nor an option for a traditional login. Everything is going to be dependent on some form of 3rd party login that the user should have access to. I do not want to be in the business of storing and securing password information, especially since it raises a lot of security concerns, and it feels like a disservice to store it, even if salted, because it forces to the user to have to create and maintain yet another password, which will undoubtedly be a copy of another password from another site.

By eliminating it, it protects this system from compromises elsewhere, and protects other systems from compromises to this one. So instead, I’ll rely on login via sites like Twitter, Google+, Facebook and even Mozilla’s Persona system in lieu of having a direct login. They can offer the ability to remove access to the project should a compromise occur. It’s a tradeoff with a reliance on a 3rd party system and better (perhaps falsely optimistic) security. It’s the day and age of the interwebs, we’re all going to be connected and networking is almost ubiquitous, so it’s a good time to start taking advantage of it.

Will this cause some issues further down the road should these systems be offline or meet their hype demise? Possibly, but I think some of that can be mitigated by enabling the user to tie in several systems together to offer a variety of ways to get into their account should their preferred one go the way of the dodo.

At any rate, this will be one of the opinionated ways in which I’ll be designing this system to see if it’ll be something that can be sustainable.

Feedback

Let me know of any additional thoughts on what should be here. I’m sure there’s a lot to add or perhaps some things that just are plain wrong that I’m blind to.

It’s been eons since I’ve written a profiling system… Image sourced from somewhere completely random

maven workflow

| Comments

Maven Workflow

I stumbled upon Maven much much later in my Java career than I should have. But now that I’m here, I’d like to leverage it for my needs. It’s a pretty versatile system that can do quite a lot. I’m sure much more than I’d care to explore. I’m not going to really introduce it as there’s enough on the web you can browse to basically get the hang of what Maven is about and what it can do for you.

For the purposes of this post, I’m going to catalog some of the things I did to configure Maven so that it can accomplish my workflow goals. Namely:

  1. Create a Java project that can be managed by Maven and developed in Eclipse.
  2. Commit to a source control management system.
  3. Install to the local Maven repository so other projects can use it.
  4. Run the project in a web container.
  5. Sync the content to an external repository.
  6. Deploy releases of the project as a zip and to a maven repository.

If you need an introduction to what Maven is or the exact inner workings of a pom.xml, this post wont be that. But if you want to take a peek at some of the things I’ve clobbered together in the pom.xml, and perhaps have any other neat suggestions, feel free to take a look and contribute some of your own solutions.

Commands

But first, a little bit of a cheatsheet for the commands I most commonly use in Maven:

  • mvn clean : Return the project to base state. Basically removes the “target” directory.
  • mvn package : Create the jars and packaged items (like wars) from your code
  • mvn install : Install the project to your local repository. Keeps any -SNAPSHOT monikers used.
  • mvn deploy : Pushed the installed project to remote maven repositories.
  • mvn release:prepare : Prepares the project for release. Be sure to check in all your code first else this will faile. Also the release process will automatically remove any -SNAPSHOT monikers from your version and allow you to specify the next version to update to. Pretty convenient.
  • mvn release:perform : Once prepared, actually performs the release. This means doing any install and deploy calls.
  • mvn release:clean : Cleans up the release details like the property file and so on. Especially useful if you forgot to checkin prior to doing the prepare stage. Although the plugin is smart enough to continue where you left off, I usually just like to have a clean start all the way through.
  • mvn release:rollback : Rollsback a release to the previous version. I haven’t quite used this enough to really make use of it yet.
  • mvn war:war : Explicitly creates the packaged war file for a project. Even if the project is specifying a packaging of jar.
  • mvn tomcat7:run : Runs the project in a tomcat environment. No need to explicitly create a war via the pom.xml or to execute war:war. It’ll just do it.

pom.xml

And the related pom.xml I’ve crafted to get the steps above done. It took a while to cobble it together and it’s still not quite fully there, but close enough. I’ll use the authentication module’s pom.xml as an example. See the version this post is based on: pom.xml and perhaps to what it has currently evolved: pom.xml.

1. Create

I’ve listed the creation step as a formality since I need to remind myself how it gets done. However, I already did that with the maven setup post, so go there, get setup, and come back.

2. Commit

Once your project is there, all you basically need to do is to git init in the directory and you’re all set. Most of the useful commands I use git, including some setup tips, I’ve cataloged on the git cheatsheet post.

3. Install

I usually write components and frameworks where one project will rely on another. So it’s important to have a way to nicely package and install these projects to a central location from where it can be synced. Maven offers a local repository to do just that. All you need to do once you’re ready is to do a:

mvn install

and the code should get packaged and installed to your local maven repository. Then other projects can simply add their dependency for your project based on your group and artifact ids (and version). Now why bother doing this if Eclipse can simply link the 2 projects together? It’s just more universal. You wont be tied down to Eclipse to manage the interconnection and if you want to reuse your component in something like a Play Framework project, it’s a trivial addition to add your local project just like you would any other public project. It’s one of those that’s simple to do and just better in the long run to get used to.

4. Run

Often times, even if I’m creating a component piece, I may want to include a simple showcase just to help the end user/developer visualize what the project is about. Normally, by Maven convention, you would have a base <packaging>jar</packaging> project and a related <packaging>war</packaging> project inked to it for the simple application. But I find it to be rather cumbersome to have 2 projects just for a simple showcase.

So instead, you can actually get away with telling maven to create your project as a web app, change the packaging to “jar” in the pom.xml, attach the tomcat plugin to it, and just tell it to ignore the packaging and run:

<plugin>
  <groupId>org.apache.tomcat.maven</groupId>
  <artifactId>tomcat7-maven-plugin</artifactId>
  <version>${tomcat7-maven-plugin.version}</version>
  <configuration>
     <ignorePackaging>true</ignorePackaging>
    <url>http://localhost:8080/manager</url>
  </configuration>
</plugin>

Magically it treats your jar project as a war (provided you created it as a web app) and just makes it run in the tomcat environment via this command:

mvn tomcat7:run

I had actually experimented with trying to keep the packaging as a war and telling maven to export just the jar into the local maven repository for shared use, but for some reason, it would always copy the .war over to the local maven repo as the .jar, which essentially broke all the downstream projects. I’m not sure why this happens, and a similar issue seems to simply have been closed.

So now I keep it packaged as a jar and if, for some reason, I actually want the war, I’ll use the mvn war:war command. Otherwise, the mvn tomcat7:run command works just fine.

5. Sync

For public projects, I, like quite a lot of developers, decided that GitHub can get that honor. For a simple sync of the code, it’s nothing more than your usual git push to push your code to GitHub. But a release deploy is a little different.

6. Deploy

Not only do I want to sync my code to GitHub, I want to use GitHub as a release hub for my code. This means that I would like to have my code packaged as a zip file, version tagged with the current version number I’m working on, upload it to the “Releases” section in GitHub, and increment my local version to the next iteration so that I can continue to work on the next version right away.

Not only that, I’d like to be able to offer my code to a Maven repository so that others can access it via Maven (or sbt) instead of downloading it. While officially, you’ll probably want to formally release your code to somewhere like Sonatype’s system so that your project will hit the main Maven repository, I just haven’t gotten that far yet. In the meantime, you can apparently just create a branch in GitHub to host your code. Sounds like a good compromise solution for the time being.

Luckily, GitHub offers a plugin for all this. You will have to modify a few things to make it work:

  • Add your GitHub credentials to settings.xml
  • Add a distributionManagement section to the pom.xml
  • Configure the maven-release-plugin to use the version format you want (or if you don’t care, no need).
  • Configure the site-maven-plugin with your desired details for release and maven repository publication.

Once all that is complete:

mvn clean
mvn release:prepare
mvn release:perform

should get you rolling. It’s going to ask you versioning information and so on about your release and incremental version. It’s not too bad, but be sure to commit your work first, else the process will fail and you’ll probably want to mvn release:clean before trying again just to make sure you get a proper clean start. When all is said and done, you should see your releases section populated and the named repository branch should have your latest release ready for public use. The repository for people to point to will be something like this:

<repository>
  <id>com.subdigit.authentication</id>
  <url>https://raw.github.com/subdigit/authentication/mvn-repo/</url>
  <snapshots>
    <enabled>true</enabled>
    <updatePolicy>always</updatePolicy>
  </snapshots>
</repository>

The one catch though is the plugin cant seem to do a nicely formated native release. So be sure to click on the version number of the release and edit it manually. Add whatever description you want and any other details, save, and it should look a whole lot better.

Suggestions?

I hope this helps. It’s nothing special, just something I want to use to help automate my project maintenance/publication workflow. I’m sure there are a lot of improvements that could be made and other neat tips and tricks to shove into the pom.xml, so if you have any, I’m more than happy to entertain suggestions.

Thanks!

I hope this takes care of a lot of the administrative stuff I dont really care to think about too often. Maven image sourced from ideyatech

google+ and youtube

| Comments

While in general, I do approve of the whole commenting system overhaul of Youtube, I thought I’d go check it out first hand. I’ve never commented on any Youtube videos in the past and only read the comments out of sheer amusement at the internet trolls.

So from that perspective, getting rid of the trolls is certainly a plus (and a minus for amusement value, but I’ll deal with that).

The issues I have though are with execution.

I like that the videos I share from Youtube show up [a bit incorrectly] as a “comment” to the video on Youtube and all subsequent replies here also show up there. (Can we please do that with public Communities?)

But for some reason, I have to unblock 3rd party cookies in order to get commenting to work. Took a while to figure that one out as blocking those seemed to be the default in my browser (Chromium).

And the source of most amusement, being able to click on “in reply to” links are now all broken for me. They simply open the video up in a new tab and loads it up fresh. I can’t see the parent reply inline at all anymore. Not sure why it was decided not to thread native Youtube comments as it threads Google+ based comments. (simple answer, structure and depth is not the same.)

Clicking on my name takes me to my empty empty profile on Youtube. If it knows that I’m also on Google+, it would be nice if it either prepopulates my Youtube information with G+ info, or at least prominently displays where to go to see more about me (preferred).

Reshares in G+ show up rather awkwardly in Youtube. It makes no visual sense and just looks like a random item or someone reposted a comment I made. It just makes no sense to the user upon quick inspection.

Also… I have no clue how to reply to comments native to Youtube. I can only assume that I cant because these were made before the Google+ update and any comment from now on can only be replied to if it came from an “approved” Google+ account? If that’s the case, I can totally understand the absolute outrage Youtubers are feeling about the comment system. Legacy is legacy, those still need to be functional. I’m assuming those are blocked for replies simply because they were not able to [or purposefully] uniformly integrate the legacy youtube comments with the new style of comments.

It’s a harsh way to expect the huge youtube population to essentially move overnight to G+ in order to continue being able to comment and reply. Outrage noted and approved.

As for execution, there is certainly much to be desired to clean it up and make it a bit more uniform and have it work smoothly. I’m a bit disappointed that it feels really hacked on instead of built as a well integrated system with Google+ with a unified threading system. You can’t have 2 different style of threading in one commenting system. That’s a complete user experience disaster and +Yonatan Zunger and company should really know better.

It would have been so much better to do a full legacy integration and start phasing out the old Youtube profile in phases to allow the transition to happen a little more gradually as people get used to (and eventually matriculate properly with the new account). Time pretty much makes people forget everything, and for a big change like this, rushing it with a mixed system wasn’t the most pleasant thing to do.

That being said, I live on G+, so I’m pretty comfortable with the controls it offers. I don’t need to share new comments to G+, and if I do, I can limit what circles see it.

My profile can show or hide most information I want, so in that sense, it’s sort of anonymous, but certainly not as anonymous as youtube accounts were. But that’s also part of the point. The whole trolling aspect is completely out of hand to make the site usable for the majority. This is a double edged sword of punishing the majority for the actions of the minority, in both situations. A relatively small number of trolls prompt a revamp away from anonymity, a smaller G+ population forces Youtubers to have to migrate to the new network or basically be banned.

If Facebook owned Youtube and forced me to log in only using Facebook just to post or comment on a video, I’d certainly have doubts. If I was a pre-migration user, I would certainly expect to be allowed a legacy account. But seeing that the controls offered over G+ based accounts are not transferred to the “regular” Youtube comments, it’s going to be a moot point to allow continued existence and use of those legacy accounts since there’s no way to prevent the already existing trolls from still being trolls.

If only the system was better integrated instead of slapped on, we could be using Google+ controls to filter out what we would consider the older youtube accounts to see only the new, more relevant content. Old users can stay where they are, but by moving, they get far better control and identification benefits. It could have been a very tempting optional migration path.

The process will indeed have been slower, but it’ll be much more of a transition as the desire to want to participate can move legitimate users into a system they have more time to get familiar with and will simply leave the trolls where they are. Or if the trolls also migrate, it makes them much easier to identify and manage via controls.

Overall, the concept of better accounts and the controls that come with it is undoubtedly a good one, the execution was poor. The rough dump of users from one vastly different system to another is pretty woefully done. This is where it’s the responsibility of the system owners to transition, not convert. To take heed how to entice users to want to move rather than just forcing them to. Force implies the lack of confidence in providing a service that will be transitioned to and that implication is probably far worse than what any troll could have done to both Youtube and Google+.

media rights

| Comments

tl;dr: Wouldn’t it me nice if we could actually own the rights to watch and engage with media in ways we see fit instead of being told what’s best (for them)?

Physical or Streaming

This concept makes the rounds once in a while: Whether it’s better to have something physical to keep and use, or to rely on a streaming service to provide for you so that you have the convenience of anytime anywhere. Both ways have their pluses and minuses. I probably wont exhaustively cover every aspect of it, as something like this has been written about all over the place. I’m just going to focus on the portions of these issues that I find interesting and relevant to my needs.

First off, my bias is that I’m a physical disc person, not a streamer. Oddly, it’s a contradiction, but I find at times more tedious to get things right with a streaming service than to actually have physical media in hand to leverage. I can certainly understand viewpoints like that of +Sarah Price who sees value in streaming where you get [almost] ubiquitous viewing on any device anywhere. But maybe because I’m more of a home based person, I tend to usually watch things on desktop or TV. I don’t have as strong a need for mobility as whatever is around the house is hooked up to play physical media or disk drive media.

Rights and Ownership

So what’s the real issue here? It’s not really about disc vs streaming, it’s far more about ownership and rights and content availability.

I like the “extras” that are available on hard media. I turn on subtitles for everything (they just help me grasp the story better in times of heavy dialog). I enjoy the extra features, cut scenes, even trailers for upcoming titles. These are all usually sacrificed on the streaming version. I just can’t understand in this day and age why the concept of a complete package isn’t present with how streaming media is rendered. Subtitles certainly should be an easy thing to get done. The packaging of the extras, or lack there of, really seems like a form of punishment for wanting to “cheap” out by getting the streaming version instead of the hard media.

But back to the core issue. Technically, in neither case do you actually own the movie (or software). What you do have is the right to view it, but that’s about it. If I wanted to watch a DVD on my Linux machine, legally, I really don’t have too many choice. Blu-ray? Forget it.

Streaming

I like that with streaming, you get that ubiquity and omnipresence of your content… to almost everything. Well, except Linux machines cause they’re so insecure… If new resolutions are available, not a problem, upgrading should be automatic. You get the convenience, but you sacrifice control. You lack things like accessibility, subtitles, special features. And the company you’re renting your streaming rights from may go broke. Then you’re left with nothing. A reality check that you were indeed just renting the material.

Physical

With hard media, you’re still “renting”, but with a little circumvention, you can own your content. Convert it to a portable media format and you have your content where ever you want. You can take it with you on your devices, put it in the cloud and do whatever with it. Want to preserve the whole DVD experience? Sure. Need to minify it for the phone? Sure. But it’s work. You need to be prepared and sync your content and/or you need to host it somewhere accessible, then download it to the device. It’s a pain for most, and an unnecessary use of disk space, but you have the control.

“Owning”

I strongly believe in the right to be able to watch media you’ve purchased in a way that you want, when you want, on what device you want. With hard media, you can achieve that. Unfortunately you break the law for that convenience since apparently it’s illegal to “have it your way”, but at least it’s possible.

Hence, here is my gripe with the media industry in general.

We’ve never bought anything since the advent of the tape and vinyl days (and probably earlier). Not a song, not a video, not an ebook. Nothing. We’ve always been renting it. Or rather, renting the rights to engage with the medium in the format you purchased the rights to. And as such, those rights can be revoked without warning, at any time, for reasons beyond your control or even comprehension. Look at what happened to Amazon’s ebook “recall” from a while back. One day you had “1984”, the next day it was removed from users’ kindles despite having paid for the right to read it.

You can’t do that with a physical book or any physical media (ignore a Fahrenheit 451 scenario). But I’m sure they would love to be able to enforce that.

Which brings me back to my long winded point. The media companies want control. Total and absolute control. Which is why streaming is the best solution. They have total and utter control over every aspect, and you get none. As in absolute zero. Yes, maybe you can rip the stream, but they do make that harder than copying a DVD.

If one day, they decide you no longer can stream to your iPhone because of a dispute with Apple, they can do that. If they decide your Android phone has a screen size that’s too large to consider it a phone, they can charge you more to stream to it instead. New 4K streaming capabilities == A new charging tier. $4.99 not enough for 1080p, they can simply serve you 720p instead.

Your rights and the concept of “fairness” is of no concern. Which is why in some sense you need to be able to protect what should have been immutable. You paid to watch something. In common sense terms, that implies you paid to be able to watch it when you want and how you want. I think we deserve that much when paying for a season of TV or a movie or an album. Technically, I think you certainly deserve that in addition to the BD, DVD, Streaming option you paid $24.99 for. You really should have the right to reformat and repackage as your needs dictate.

And in some sense, I don’t think the media companies actually worry too much about the legitimate use of their media for your use. They’re worried you’ll copy it then distribute it. But that’s a discussion for another day.

Common Sense

I think that’s what’s missing here. Common sense control abilities for my rights to view media. I want to buy the rights to view my media in whatever way I want. I want to be able to purchase my “rights” that allow me to view a movie how I want, when I want, where I want, using whatever I need to make it happen. If I want subtitles, if I want special features, if I want it in 1080p or 4K or 10bit or mkv or with director’s commentary. I simply want the ability to buy my right to do so.

In that case, I don’t mind paying a bit more. But please, let’s be reasonable. $24.99 for a BD/DVD/Streaming combo package is nice, but that really should be the upper limit for the concept. Just give me the rights to store it on a computer, or give me the same content in streaming I have on physical media. Just one way or the other, let me buy my rights. I’m not saying I should have the right to distribute it to anyone else, I just want the right to legally burn a spare copy, or to rip it to my HD, or to encode a version with soft subtitle and load it onto my tablet to watch on the plane.

That, for now, is the reason why I still prefer my hard media. It still offers the choices and ways I want to watch things. Yes, it sacrifices the ubiquitous convenience of streaming, and if you’re a traveler or need to have things on the road a lot, this may just not be convenient enough for you. But for me it is. And for me, streaming is not that altruistic solution that’s future proofing my media, it’s simply a way to leash control onto you and relinquish you of any rights you thought you had (which you never really had in the first place).

Something needs to change. We need a bit more fairness in this public vs big media battle.

fbi warning image from here, which I guess in it of itself is questionable if I am using it in fair use

authentication module

| Comments

tl;dr: Run it yourself in 10 seconds flat via GitHub and Maven: Authentication Module

The introduction post sort of served as a primer about why I wanted to create this module. This post is going to talk about technical side of the module and for what it should (and shouldn’t) be used. Or something like that. We’ll see where it went once I’m done writing it.

Authentication Module

Every site that wants to deliver customized content needs a way to recognize the user. The more customized and personal, the more likely you’ll want actual user identification rather than just plain browser recognition.

I figured that most sites I want to build will have some element of user identification, I may as well write a module that I can drop in and get going without having to worry too much about recreating it from scratch each time. Thus here we are, a Java servlet based module that will allow the developer to drop in a few servlets, modify a configuration file or two, and have a bare bones site that handles user logins from a variety of 3rd party systems.

Java

Yeah, it’s what I like, and what I still think is pretty relevant when it comes to server side content. I debated if I wanted to try something new, but I think for the backend, I’m comfortable with Java and it provides well in terms of support and being able to find and use libraries. I’ll save my experimenting for front end work.

Concept

The idea is to have a system that will allow the developer to add 3rd party login support by trivially calling some servlets and inserting provided Javascript. I want to get away from having to rely on some deep configuration of the host platform for user and password storage. It just makes things too complicated since most of the time, all you really want is to know who the user is and if they have been validated so that you can get/set their preferences and create a customized experience for them.

I also want to have this divorced from any particular system or infrastructure so that it can just run. No need for anything WebSphere or Tomcat or Glassfish specific or any deep configuration of LDAP or JAAS or some security module. Just alter a property file and you’re good to go. Now, is this the right way to do it? Well, that can be debatable especially since these servlet systems do offer a lot of robust options, but I also just want to give creating this project a shot.

In the end, it’s meant to serve needs that I’m identifying during site creation, so that’s my driving force. Along the way, I hope that it may be of help to others who want a bootstrap base to explore and jump from. We’ll see how well that gets accomplished.

Flow

The basic idea is to:

  • Have a screen with a bunch of login buttons.
  • Hook those buttons onto a servlet or javascript as needed.
  • Upon user click, initiate the needed redirection and prompting defined by the flow requirements of the 3rd party systems.
  • Upon proper authorization, grab the basic data returned from the 3rd party and present that information to the backend system to allow it to use it for a customized user experience.

Implementation

A collection of 4 servlets, 2 of which are more core, and the other 2 slightly more optional (read: yet to be fully implemented).

  • LoginServlet - Triggers sending the browser to the official authentication page of the 3rd party system.
  • CallbackServlet - Location the browser will be sent to after the user has authenticated with the 3rd party. This is usually a registered address with the 3rd party system or passed to it as part of the initial login (automatic).
  • LogoutServlet - A way to remove the current user information from the browser session.
  • DisconnectServlet - A way to decouple the 3rd party system from the user. ie. Remove authorization to leverage that 3rd party platform.

Looking aside the last 2 for now (since their implementations are on the lesser scope of interest), the system really only requires 2 solid servlets to function.

Login

The com.subdigit.auth.servlet.LoginServlet instantiates the com.subdigit.auth.AuthenticationHelper class which is what actually handles the gritty details. I did this with the hope that I can easily decouple the system from a specific servlet framework instance. For now the AuthenticationHelper requires the servlet’s request and response to be passed in so that it has access to the necessary information, but I can see this evolving so that you can extract the information from any source into the AuthenticationHelper to have it behave accordingly with 0 requirement to be in a servlet environment at all.

The AuthenticationHelper is issued a authenticationHelper.connect() call to start the process. It figures out which 3rd party system you’re calling, finds the proper class that can handle that system’s requirements (which needs to extend the com.subdigit.auth.AuthentcationService interface), dynamically loads it, and calls that service’s authenticationService.connect() method. That method creates the URL to the 3rd party system with all the required parameters (like the application id and key) and redirects the browser to it to prompt the user to authenticate.

Callback

Once the user authenticates, the 3rd party system is told to callback to the com.subdigit.auth.servlet.CallbackServlet which instantiates another AuthenticationHelper that chains the same AuthenticationService to call the authenticationService.validate() method. At this point, the system usually has an abstract token for the user, which then needs to be looked up with the 3rd party system to get the relevant details (like name, email, etc). Each service varies how this is done, which is why we need a per-service class (See the com.subdigit.auth.service package) that can handle these issues per 3rd party you want to connect to the Authentication Module.

Once the user’s information is retrieved, the service then packages all the data it has into an com.subdigit.auth.AuthenticationResults object which is floated back up to the CallbackSservlet. At the CallbackServlet level, the developer can probe the AuthenticationResults object’s various methods to access the stored information about the user. This will then allow the developer to correlate the data in the AuthenticationResults object to an existing user in the local datastore or to create a new user.

Installation

Download the source from GitHub:

github clone https://github.com/subdigit/authentication.git

That should get you an authentication/ directory. You will need to copy:

src/main/resources/authenticationserviceconfiguration.properties.sample -> src/main/resources/authenticationserviceconfiguration.properties

And fill in the appropriate .appid and .appsecret values for the services you want to use. It’s a hassle, but you’ll need to figure out the developer pages for the services and do the whole registration of your app there. Some URL hints for that are on the project’s README.md file. Once inserted, make sure the .enabled is set to true for the ones you will use.

Test Instance

To run a test instance, be sure you have Maven installed and run:

mvn tomcat7:run

That should fire up a bare bones system on localhost:8080. The systems you’ve enabled should show up there and you should be able to click and log into the test page with them. Should :).

Integration

The servlets provided are really bare bone samples. Take the flow in each of the goGet/doPost of the servlets and implement your own desired customizations in them. You can probably copy the LoginServlet as is and write your own LogoutServlet for however you want to get the user logged out. The only one where you should modify the code would be in the CallbackServlet. Once the AuthenticationResults object is returned from the authenticationHelper.validate(), this would be where you need to probe the results to figure out who the user is and how they relate to your existing system (new user, existing user, etc). At this point you will need to decide where to redirect the user to and if a new account needs to be created and so on.

The authenticationResults.getServiceUserID() will return back the primary user identifier from the 3rd party system and authenticationResults.getService() will let you know the 3rd party service in question. That should be enough information to find that same user in your back end and load their information. If they are new, you can get back some information from authenticationResults.getVariable(<String>) or get back all the parsed data via HashMap<String,Object> authenticationResults.getDataStore(). And if you still need more information, you can get the actual object that was returned back from Object authenticationResults.getReturnData(). You will need to look into the code for each service to find out what type of object is being stored.

Questions

If you have any questions about how to get this working, feel free to leave them here or to just bug me on Google+. Twitter is ok too, but the 140 character limit will just end up being frustrating.

Oh, and be more than free to do what you want with the code, just that if you make any changes, share them!

yeah, it’s still got a while to go, but this can be a start.

authentication introduction

| Comments

tl;dr: Logins suck. I’m making it modular. Plug it in here: Authentication Module

Authentication

I come from a corporate world of internal application development where the concept of a single, unified, user identification is handled through the official company email and corresponding backend password validation system. So when I create an application, there’s no question or issue about how and what identity provisioning mechanism to use. The corporate email and password ruled everything.

But the real world is a different beast.


Reality

Out here, the use of a user email implies the need to register it, and therefore to create a unique password to associate with it, and I, as the application developer, now have to store that password. It would be great if everyone picked a unique password for each site so that if there is a breach and the password becomes known, all that has to be done is to nuke that password and no one can get into the site. However, that’s not how people work. Passwords get reused… a lot. One password to rule them all is all too common. Combine that with a known email, and that of course spells trouble.

But there are certainly ways to actually store passwords securely. You can hash and salt them to obfuscate the actual password and only care about what the password represents. There are also numerous services out there that allow you to leverage them in order to validate a user. You can do things like log in via Facebook, Google+, Twitter and even Mozilla’s Persona initiative.

All of these put the validation, verification and identity check of the user on other systems so that you can leverage their user validation infrastructure and just validate who is accessing your site without becoming the actual provisioner of the security credentials. As a developer, less is more in this case. Doing this completely removes the onus of password security storage back to the “other guys” and it’s one less thing to worry about. All I need to worry about is who the user is and keep track of that information in my locally.

Now, are there problems this sort offloading entail? Sure. Your potential users need to be a member of one of the services that you’re coding against (though something like Persona is trying to eliminate that as much as possible). It’s also hard to correlate one person’s fecebook account with their google+ account unless you get something definitive to connect them together (like the underlying email address or an explicit login to associate one with another). This may happen intentionally by people to create a new account, though that’s something already present with email based logins anyway so not much of a new issue. It should actually be less of an issue since you can hope it isn’t as easy to create duplicate accounts on these other system (but that would be a naive assumption).

What’s actually worse though is the unintentional aspect duplication: poor memory. Many a times I’ve forgotten if I’ve signed into a site via facebook, twitter, google+ or something else. So the developer will need to take that into account as well.

But despite that, I think removing the onus of password storage and security is a good thing. You know, just in case, not that anyone would want to breach a site I create :).

In the end, it’s more about simplicity. I just don’t want to hassle with it. The less I deal with sensitive material, the less likely it can be breached.

New Project

Hence this project: Authentication Module

The goal is to create an easily usable Java servlet base module that can be leveraged as the login mechanism for anyone developing a site. Well, for the time being, that anyone is me, and I just didn’t want to rewrite this part for each site I wanted to create. I’ll have a little more inner details about the project in the future, but for now, you should be able to run it via a git clone and a maven call.

The purpose of this module is to provide the developer with all the necessary code and routine to allow for the end user to log in via the provided 3rd party services and return back the identifying information for the user so that their information can then be stored in the application as the local user. No more having to deal with passwords, just a return of the critical information about the user the system needs to allow a customized experience.

What the developer does with that information is beyond the scope of this module (I’m planning on creating a new module for that), but this will be a good jumping off point to just get going quickly.

I hope it’ll be useful to people. You can check it out on GitHub and feel free to suggest any requests, fork or pull or spoon or whatever you like with it.

security and “doing things right” are always such tough issues. personally, it’s a thorn in my side, hence the need to modularlize and just make it easy and reusable

maven setup

| Comments

More: My Maven based project workflow: Maven Workflow

Maven

I only recently stumbled upon Maven after seeing how Ruby and node.js have their respective package management systems. Well, actually, I’ve been stumbling upon pom.xml files for a while now but it never really clicked with me what they were for. After a bit of exploring during a new project setup, I figured it’s time to include a bit more package management into my Java projects so that I can be a little more versed with how things are done in modern Java development.

So, enter the Maven setup guide.

I’m writing this as a companion piece to the Java Development post since the concept is going to be related. This guide will assume some setup from that post for the basic Eclipse bootstrap, and the rest will diverge from there to help you get up and running with Maven and Eclipse and a simple project setup. Or something like that. We’ll see how it fares by the end of the post.

Setup

So here’s the plan:

  • Get Maven installed on the OS and in Eclipse
  • Create a new Java project with Maven support
  • Setup the various configuration files
  • Integrate the project into Eclipse
  • Make sure the project can be run independent of Eclipse

Preparation

You’ll want to get Maven installed on your OS, since part of the point of Maven is to be able to compile and run things independent of a platform (like Eclipse). For something like Ubuntu, you can easily do:

sudo apt-get install maven

and that should do the trick. For other OSes, you’ll have to do your due diligence to get it installed :). For Eclipse, you’ll need these repositories. Though technically, I think you just need the second one:

“Maven” : http://download.eclipse.org/technology/m2e/releases/
“Maven Integration for WTP” : http://download.eclipse.org/m2e-wtp/releases/
[Source]

With that, your system should be ready for Maven use at the OS and Eclipse level.

Project Creation

This is where the bizarre Maven commands come into play. For my case, I develop web applications aimed to be run on servlets, so this is the Maven command to create a project as such:

mvn archetype:generate -DarchetypeArtifactId=maven-archetype-webapp

Just run that from your project parent directory and it will eventually create a project with the specified artifactId you will provide during the prompted walk through. In a nutshell:

  • groupId: Your Java package. For me, I use something like com.subdigit
  • artifactId: The application package. So I may do something like testapp if my project package is com.subdigit.testapp
  • version: I usually downgrade my version to something like ‘0.0.1-SNAPSHOT’ just for starters. Keep the -SNAPSHOT bit as it actually as semantic meaning.
  • package: should be fine to leave as is, as it will just mirror the groupId.

And that’s it, you should now have a “testapp” directory populated with an outdated servlet web.xml. Congrats. Which of course means…

Fix Defaults: web.xml

Why the default archetype for a webapp is not up to date with modern servlet standards, I don’t know. But it’s an easy enough fix.

You’ll want to update the src/main/webapp/web.xml from the dark ages to at least the current standard of the 3.0 framework. So, just replace everything in there with this:

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
  version="3.0">
<display-name>User Module</display-name>
  <welcome-file-list>
    <welcome-file>index.html</welcome-file>
    <welcome-file>index.jsp</welcome-file>
  </welcome-file-list>
</web-app>

That should give you the basics to start with. And if you do it now, Eclipse will be much happier giving you the proper settings during the import.

Fix Defaults: pom.xml

For the most part, the pom file is ok. But it could use some version updating. The default junit version that’s hardcoded in is 3.8.1 so you might want to update it to something like 4.11.

You’ll also want to force Maven to use a modern JRE as the default is something rather ancient. To do so, you’ll need to explicitly specify the JRE version you want Maven to use in the pom file’s <build>/<plugins> section:

  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.0</version>
    <configuration>
      <source>1.7</source>
      <target>1.7</target>
      <encoding>UTF-8</encoding>
    </configuration>
  </plugin>

The rest you get to have fun modifying it to your needs adding in all sorts of dependencies and whatever you need. I wont cover that here, but you can see the pom.xml I use in this project.

Eclipse Import

Once all the above is set, you can import it into Eclipse. Some sites will tell you to use “mvn eclipse:eclipse -Dwtpversion=2.0” from the commandline to prepare the project, but the instructions that worked for me didnt like it, so I don’t use it for my projects. All you should need to do is:

File -> Import -> Existing Maven Projects
[Source]

And just follow the prompts. If you already updated your web.xml per the above, the Dynamic Web Module version should already be at 3.0. Which is what you want.

Fix Defaults: Eclipse

You’ll most likely have some of the directories you need, but not all of them. Odd isn’t it. Just go ahead and make sure you have at least these 4 directories:

  • /src/main/java: Java files go here
  • /src/main/resources: Property files go here
  • /src/main/webapp: jsps go here
  • /src/test/java: Unit tests go here

If not, you can just right click on the project and do a “New -> Folder” and create the missing ones. Again, why they’re not already all there, I have no clue.

Update Settings

One thing that didn’t catch for me at times were my Maven dependencies. Basically, with Maven, you’ll no longer be directly hosting the various dependency jars in “WEB-INF/libs”, instead they all go into a central repo on your machine that Eclipse needs to reference in its classpath. So you need to make sure your “Maven Dependencies” are linked properly, or else you’ll end up with a bunch of “ClassNotFound” exceptions.

Open up your project properties, and find “Deployment Assembly”. In that, there should be:

"Maven Dependencies" -> WEB-INF/lib
[Source]

If it’s not there, add it. If it’s there, golden. Just to make sure, you should also have your main java and resources directories pointing to “WEB-INF/classes” and the webapp directory pointing to “/”.

Eclipse Independence

At this point, you should be able to create your servlet, configure the web.xml or use the @WebServlet annotation and have it get registered properly (assuming you went through and configured a servlet container like Tomcat). You’re all golden.

Now we need to take advantage of the whole package and independence Maven is supposed to give you.

Personally, I never want to have to muddy my development environment just to test out a cool project I’ve see somewhere or to run the samples of a project without the hassles of setup. I love seeing all these node.js projects just be able to run everything from a lightweight commandline instance so that you don’t have to do any setup. Maven also lets you do the same.

For me, I’m currently using Tomcat, so I would love to just have Maven be able to run my project from the commandline so that people can easily test it out without hassle. To make that happen, you need to make sure you have a dependency in your pom.xml that points to the servlet API so that Maven can compile against it, and then you also need to have it call a server to run the compiled code on a localhost port.

So, for the servlet API dependency, make sure you have this as part of your pom’s <dependencies> section:

  <dependency>
    <groupId>javax.servlet</groupId>
    <artifactId>javax.servlet-api</artifactId>
    <version>3.0.1</version>
    <scope>provided</scope>
  </dependency>

And in your <build> section’s <plugins> area, you’ll need the Tomcat plugin installed:

  <plugin>
    <groupId>org.apache.tomcat.maven</groupId>
    <artifactId>tomcat7-maven-plugin</artifactId>
    <version>2.0</version>
    <configuration>
      <url>http://localhost:8080/manager</url>
    </configuration>
  </plugin>

Once that’s in place, you can then call:

mvn tomcat7:run

From the commandline and voila, you should now see your app running on http://localhost:8080 Congratulations. You’ve now achieved IDE independence.

So in theory, if you put your code up on GitHub, tell people to clone it locally, all they need to do to run it is to type the above. No setup, no IDE needed, no hassles. Quick and simple. You can try it with my project and see if it does indeed work.

Notes

There’s a lot I’m leaving out about Maven and pom.xml setup, but I’m just not versed enough to write about it, nor if I did, would this post be readable in length. It’s already pretty long and talking about the nuances of the pom.xml is just way beyond this post. Best bets: Go search for pom.xml files in GitHub and use them as examples. I’m sure there will be a few referential ones around :).

Next

I think I have most of my setup complete, so I want to start concentrating on creating some projects. You can get a peek at what I’m up to on my GitHub account if you’re curious. It’s going to be a series of modules that offer some good reuse and bootstrapping for starting a user login capable website… or so I think.

setup always takes so long to do. I hope this speeds it up for some people. At least it does for me. Also, maven image sourced from ideyatech

google+ shift

| Comments

tl;dr: Fix communities by allowing simultaneous share to public and filtering circles by communities to create a semantically relevant data pipe for Google’s knowledge graph. Or something like that. Just read it :).

So Google+ is an interesting beast. It’s currently my network of choice simply because there’s content here. I know, it’s supposed to be ghost town, but I’m just going to ignore that for another discussion. The bottom line for Google+ is that if you’re interested in having actual conversations with people about various topics, whether it be technical or social, this is a great place to be… provided, just like any other network, you spend a little time to find the right people, pages, and now, communities.

But this isn’t going to be a general post about Google+. That’s just covering way too much ground.

I want to focus on a particular shift I’ve noticed for Google+: We’re slowly moving away from a people centric system to a topic centric one.

People

Just to get this out of the way, we still follow people on Google+ because they’re the ones that actually post things that interest us. Just like we do in a variety of other networks and systems. But the nice thing about Google+ is that it’s also very topic oriented. And this can actually be a foreign thing for a lot of people, especially those coming from Facebook. That network is much more tightly woven together by people connections (family, friends IRL, etc). Hence they just can’t seem to leave it since it means leaving actual people behind and having to change the paradigm of operation when entering Google+.

Twitter users sort of have the same problem. Though freer than Facebook in terms of who you follow, you’re still basically following people. You may join a hashtag conversation once in a while, but in the end, you can only do so much to continue to be with people in the same mindset. Twitter an awesome system for quick, short, shoutouts, events, exchanges, promotions, and blurbs, but [for me at least] it’s an absolutely horrendous platform to actually converse on. Nothing is threaded, near impossible to see anyone else’s replies, the reply-to-id is woefully underused. No wonder they’re trying to ban a lot of 3rd party clients to try to enforce a common UX across everything. But I’m derailing, that’s a topic for another time.

Google+ started itself off as a happy medium between traditional threaded conversations [like in forums] and the faster paced people oriented messaging [like in Twitter]. Coming in late, it had the time and luxury to see what worked in Facebook (connections with people are important) and what worked in Twitter (fast updates), and probably more importantly, what was missing (poor conversation tracking, lack of privacy, lack of easily expanding beyond your “[1000’s of] friends”).

Topics

Now as it establishes itself (#2 social network), the next step is to try to push the system slightly away from people orientation to topical orientation. People are important still, but it’s the topics they write about that are what keep the people here. And Communities fills that niche very nicely.

It helps to centralize what would have otherwise been produced in separate stream by separate people amidst all the other topics people write about. With the introduction of Communities, people can now post their relevant content to a specific location and aggregate everything in one place. This helps to make sure if a topic is hot, or a community is populated well, you can always find and converse with people of like interests. That’s incredibly important for user retention. People need to feel like they belong, and now, even the newest of users can simply just jump into a community and be instantly connected.

Brilliant.

But… they got it wrong.

Not so fast

People are not topics. They still have a need to be known as an individual. From a Google perspective, I think it’s easy to harvest our output into topics for others to consume and hopefully reciprocate by contributing back. After all, what better way to extend the knowledge graph than to have great semantic data about what we post. And it is a great benefit for all the users. But we, as producers, still need our identity.

And our identity is defined by the output we produce, but the implementation of Communities fragments our outputs. We’re now forced to choose if we should post content to our main stream, or to post content to a community. I know that we can decide if we want our community posts to appear on our profile page, but quite honestly, that’s a secondary place people go to for content. The whole point of circles was to group people and listen to those circles. And with the current setup, your community posts do not show up on your main stream.

So you have to choose. Do you post to a community? Do you post to your main stream? Do you double post? It’s choice we shouldn’t have to make at all. And double posting just to make the post appear in your public stream is just a wrong waste of my time and effort.

Fix it

Easy fix? Allow us to post to a community and to our Public stream simultaneously in one posting action. I know the restrictions are in place to make sure the content publicity level and the community publicity level are accounted for, but that restriction was the easy way out (my guess is that a more robust method is being worked on, but it’s not a trivial issue). But there’s no reason to have that restriction in place for a fully public community. So perhaps the quick solution is to allow a “Public” share as well as a Community posting for those communities that are fully public.

I know there will be some strange UX issues, but quite frankly, this time, the benefits really out weighs the UX detriments of creating a special paradigm just for this one use case.

But you know what’s better than an easy fix? A proper fix. Ready? Take notes.

Go Multidimensional

People are not one dimensional as circles imply. I don’t fit into just one circle, I probably fit into several circles. So it’s of no use to have a Circle called “Java” and putting me in there expecting I just posts about Java since surely, as the internet is the internet, I’m going to post some Cat pictures. So with that one fell swoop, I’ve made your topical circle cluttered with junk and conceptually worthless.

You could follow a community instead, but all you’re given is an “Alert On/Off” system, which floods the crap out of your stream if I leave it on for large communities. So now, you can only either get alerted for everything or you have to remember to manually go to the community. Great, in that case communities are only theoretically useful for filtering.

We need to merge this concept of circles to follow people, and communities to follow topics. You need to be able to listen to what I post when I post to a specific community. In order for that to happen, I need to be able to make one post that works as a post to the community and a post to my public stream. This type of post now essentially semantically tags my post with a topic, ie. a community. Now all my posts are going to be semantically tagged. Isn’t that a dandy? And we didn’t even have to teach people to have to use hashtags or any gimmick, all part of the UX.

Awesomesauce.

Once my posts are tagged with a related community, let’s take advantage of that. Allow us to filter a circle based on a community (or even communities). Now you can put me in a circle called “Java”, filter it against the Java Community, and only those post I create that are also posted to the Java Community will show up in the circle stream. [Un]fortunately, you now no longer have to ever see my Cat posts (well, that is unless you put me in a circle associated with the Cat Pictures Community).

We’ve now taken the first step to balance the idea of people and topics into one UX friendly concept.

Progress

I’m now more willing to post to a community exclusively since I know if people want to just follow me as a person, my main stream has all my content. And for those that just want to follow one specific aspect of me, they can filter me by the communities I post to. People will be more willing to circle me, I’ll feel the love of gaining readers, and that will encourage me to post more, it’ll encourage others to just get what they want and therefore interact more, it’s a win-win for both sides.

Well, for Google also since interactions go up, and we’ve most importantly started to add genuine semantic data to our posting that can be programmatically mined instead of inferred via context and content.

Communities now are more useful. Circles are much more useful. Following people is much more comfortable. Less crud in my stream from people posting things I don’t want, but can still see the posts I do want from those same people. Egos can be satisfied. And we’re contributing to the I’m-sure-it’s-coming-soon Google knowledge graph search system. people’s egos will be satisfied.

It feels like a win-win for both sides, topical and people oriented, for Google and the users of Google+.

So, make it happen. Pretty please?

Well, this got a bit long. I can easily see what is out there now as the “quick” thing to put out. But there are also some directions the network is taking that may indicate that I’m not quite on the same page as the developers. Will need to dig deeper and see what’s actually there :).

java development

| Comments

Java

Look, I know there’s all these new paradigms out there for programming applications that provide useful features and new technologies, but sometimes you still want to return to your roots and what you’re used to, just to see what it’s like.

I’ve been a Java programmer for a good part of a decade and a half and a lot has changed in that time. I’ve been a bit stagnant in taking advantage of the new areas of Java, so I thought it would be time to get a fresh setup up and running so that I can tinker and take a look to see if Java can indeed compete with the rest of the pack. But first, I need to get setup…

Eclipse

Love it or hate it, it’s a pretty robust platform that cant be ignored. I lean more towards the love it side, but it may be simply because I haven’t really used many other integrated IDEs for Java development. Once you have the servers hooked in, I think Eclipse can offer a good experience in terms of “just getting things done”. But the one warning is that you really want to have enough RAM… I mean lots. I’m comfortable with the 16GB I have, but I can see why people feel like it’s bloated (it is) when you only have 2GB of RAM to share with other apps.

Oh, and sometimes, it can feel downright ancient, but you can easily work around that… I hope.

Setup

I’m currently on Ubuntu 12.10, and the default Eclipse is 3.8. I no longer know if this is really considered Indigo or Juno, but all I know is that the newer 4.2 branch is horribly unstable and prone to a lot of crashes on Ubuntu. I tried it, but had to revert back to the default, which I guess is the default for a reason.

Anyway, getting it installed and running is pretty much as easy as visiting the software center. The fun starts when trying to get all the plugins and dependencies resolved. For this round, here’s my goal, installed specifically in this order since I had trouble when things got out of order:

  • Install the Google Plugin for Android development and GWT (just to play around)
  • Install GlassFish
  • Install the “Web, XML, Java EE and OSGi Enterprise Development” tools
  • Install Tomcat’s TomEE variation
  • Install EclipseLink and JPA
  • Install Maven plugin
  • Install Git plugin

Now just getting them installed is a bit different from actually using them, but I just wanted to have some options setup now so that I can just try these out when I’m ready to play with them. So, onto the installation woes.

All these are installed via “Help -> Install New Software… : Add…” by adding a new plugin repository. You can actually add them all first then do each installation if you find that easier.

Preparation: Eclipse Update Site

Site: Eclipse Project Update Sites
Repository: “Indigo Update Site” : http://download.eclipse.org/releases/indigo/

For some strange reason, mine was missing the update side. So I added it in. Not sure if it was missing since Ubuntu is supposed to handle the updates or not, but I thought it’ll be good to have it there in case I wanted to browse it to add some extra plugins (like some of the ones below).

What tricked me up was that you can actually quickly see all the sites that are available to the Eclipse updater by using the dropdown arrow next to the filter box. For some reason I wasn’t seeing that and therefore wasn’t seeing the list of installable software populated. It would be nice to have it show everything by default, but understandably, they didn’t want to have an initial network hit fetching all that data, especially if you have a lot of sites.

First: Google Plugin

Site: Google Plugin
Repository: “Google Plugin” : http://dl.google.com/eclipse/plugin/4.2

This will install things like the ADT Plugin for Android development (you still need to have the SDK installed, which you can find instructions for here). It also adds support to be able to play with GWT which is a nice addition. Not sure if I will, but it’s on the list of things I want to take a look at.

After a bit of experimenting, this plugin has to go first. If I install it later, it complained about some ant dependencies that just wouldn’t resolve themselves. The joy of dependency hell. But install it first and no problems.

The plugin comes in a 3.7 flavor also, but use the 4.2 variety. I think 3.8 is really almost the same as 4.2, just without the UI makeover.

Second: Glassfish

Site: GlassFish
Repository: “GlassFish” : http://download.java.net/glassfish/eclipse/indigo

GlassFish is basically the reference Java EE server produced by Oracle to support and showcase the standard. I figured it should be a good one to have and use, especially since it came as a complete package with the Apache Derby DB. Why not.

Once you load up the repository, you should see a “GlassFish Application Server” entry. Go through all the hoops, restart Eclipse, and end up back at the Install New Software section. I figured it’s pretty heavy duty and the dependency chain it installs will get a lot I’ll need later on installed.

Note: I’ve had the repository fail on me from time to time, so alternatively, you can use the Eclipse Marketplace to get Glassfish installed. There’s a good instructions set on Tech Juice.

Third: Web/XML/Java EE Tools

Repository: “Indigo Update Site” : http://download.eclipse.org/releases/indigo/

I also loaded from the Indigo Update Site: “Web, XML, Java EE and OSGi Enterprise Development” tools. But without the “PHP Development Tools (PDT) SDK Feature” and “Rich Ajax Platform (RAP) Tooling”. Not 100% sure why, but just followed the instructions from a Tomcat installation post. I figured they know what they’re doing and I’m not a PHP person anyway so no loss there.

Fourth: TomEE

Site: TomEE [Tomcat]

Like GlassFish, TomEE is a full service stack. It’s basically Tomcat with the added modules to make it into a full Java EE stack.

This one you need to go download and install manually. We’re simply going to associate Eclipse with the freshly installed server. So, go download TomEE from Apache site (I got the Plus version). Then just “tar xfvz” it somewhere you like (I put it in /opt). It will need to be owned by the same user that will be initiating Eclipse in order to allow for file changes, so I just “chmod -R” to my user and group.

The instructions are nicely laid out on the site. If you already have a project, you can use the “Quick Start”, else it’s not too much of a hassle to do the “Advanced installation”. The only difference for me was that I couldn’t find the “Modules auto reload by default checkbox”. I changed the ports to +2 everything since I’m letting GlassFish run on port 8081 (a +1 from the default 8080). That way it wont interfere with my system’s default web server.

I opted to let Eclipse manage this instance of Tomcat so I set the preference to “Use Tomcat installation”.

When doing the server association, you can actually tell the system do go download Tomcat for you. I guess that way you can have a completely managed Tomcat instance via Eclipse, and all you would need to do is to add the additional libs/wars to turn it into a TomEE instance. I just opted to install my own separate instance and hook it in, but let me know if you’ve done otherwise.

Fourth.5: GlassFish configuration change

Just a note. I hate crappy directory layouts. And GlassFish creates a top level directory in your workspace for its server. However, once you install TomEE, it puts it’s server information in a “Servers” directory, like it should. So to fix this, I do a little cleanup to move the GlassFish server directory to the proper Server directory.

To do this, Open up the “Server” view, which should be showing a GlassFish server. Double click it and find the Domain Directory entry. Before hitting “Browse”, I copied the “glassfishdefaultserverlocalwhateverthislonguglynameis” directory to something like “Servers/GlassFish-3.1.2.2”. Once I have that, I hit “Browse” and simply selected that new directory and deleted the old one. All nice and clean.

Configuration Note: For some reason, I cant unlock the server port to edit it via the provided interface in Eclipse (when you double click on the GlassFish server). But you can do it manually by going to the GlassFish server directory (the one I changed to “Servers/GlassFish-3.1.2.2”) and editing the config/domain.xml file and finding the entry for the One note, anyone know how to change the default port on GlassFish? The Server and Admin Server Port entries are locked for me… /config/domain.xml change:

<network-listener port="8080" protocol="http-listener-1" transport="tcp" name="http-listener-1" thread-pool="http-thread-pool"></network-listener>

to

<network-listener port="8081" protocol="http-listener-1" transport="tcp" name="http-listener-1" thread-pool="http-thread-pool"></network-listener>

And that should do it. If you want to also run a stand alone version of GlassFish, you should probably go ahead and +1 all the other <network-listener> ports so that there wont be any conflicts.

Fifth: EclipseLink and JPA

Site: EclipseLink
Repository: “EclipseLink” : http://download.eclipse.org/rt/eclipselink/updates/

EclipseLink allows you to easily use the Java Persistence APIs (JPA) to connect and pass objects to and from a data store. Well, so they say. I have yet to try it so we’ll see how easy it really is :).

Sixth: Maven

Site: Maven
Repository: “Maven” : http://download.eclipse.org/technology/m2e/releases/
Repository: “Maven Integration for WTP” : http://download.eclipse.org/m2e-wtp/releases/

Maven’s the super ugly cousin of npm, gem, other modern dependency package management system. But I guess it works once you get the hang of it. I have yet to get the hang of it, but would like to, hence I’m installing it.

You can find more detailed installation and project setup instructions on this companion post: Maven Setup.

Seventh: Git

Site: eGit
Repository: “Git” : http://download.eclipse.org/egit/updates

And Git, or eGit, jGit, whatever. It’s basically git. Everyone seems to love git (and thus GitHub). I’m learning it. So I may as well jump on the band wagon.

Warning: Of course, the natural progression is to want to install the GitHub plugin. YMMV, but I couldn’t get it to work. In fact, it bricks my entire Eclipse installation and I had to clean out and reset my Eclipse back to the default and redo all the steps. I have no clue what went wrong, why, or how to fix it, but it’s borked me enough times to stay away from it for now.

At first when I tried to install the GitHub plugin (Site, Repository), it complained that I needed Mylyn installed. So I tried to install Mylyn (Site, Repository), but it complained that I needed Subclipse installed. So I installed Subclipse (Site, Repository), then Mylyn, then GitHub, then Eclipse would no longer restart. Had to nuke it all.

But do let me know if you can get it working…

Cleanup

So, speaking of nuking, sometimes, something will just go wrong and you need a fresh start. I’m not quite sure what to do about projects that got configured with important information, but I had a clean slate from the beginning, so I didn’t have to worry about relinking projects. If you need a clean Eclipse slate, because something like GitHub borked your installation, here’s some of the directories to clear out:

  • Remove the version of eclipse that you are using from ~/.eclipse
  • Remove the /.metadata directory.

Pretty easy, but again, be warned, if you already have established projects and you do this, the meta information about your project will be lost and you’ll have to re-import them back into Eclipse. I was on a clean slate so not too many worries for me.

Startup Errors

I’ve noticed that at times I’ll get a message during startup complaining “An error has occurred. See the log for more details”. And that’s it. No more eclipse. The log is equally unhelpful complaining about some sort of OutOfMemory error.

I found this article which alleviated the problem by calling eclipse with the -clean flag:

eclipse -clean

Which seems to allow start up of Eclipse with no detrimental effects. No clue why, and neither did the author of the post know. But so long as it works, may as well leave that tidbit here.

Notes

Even though both GlassFish and TomEE are installed and runnable simultaneously, unfortunately you have to associate your project during creation time to one or the other. Which means you cant just plop a Glassfish project into TomEE and visa versa. But perhaps with some manual XML hacking, it can be switchable once you create it. I just haven’t gotten around to that level yet, and probably wont. But it’s good to have either available to play around with and see what’s going on.

Next

Now that I’m all Java setup, I’ll probably start taking a look at some of the newer frameworks and see what they offer. Things like Spark, Britesnow, Play, Vaadin, Jetspeed, Jersey and even older ones like Hibernate and Spring… maybe. In the end, I want to see if modern Java development is up to snuff to compete with the newer frameworks.

As a preview, my next task would be to get a bit more of an understanding of JPA, which I’ll be experimenting with via this set of instructions. But experimenting early on, I’ve already run into a problem, so hopefully I can get around it (and I’m sure others) after a deeper dive.

Sometimes I cant tell Java is really the standout in terms of diversity and functionality or if I’m just used to it. Everything else feels pretty lacking in structure and organization. But I’ll see if there’s something out there that can compare. It is bloated, but there’s quite a lot out there for pretty much anything you want to do.