Railsconf 2007 Mapping Rails to Legacy Systems By Stephen Becker IV and Devon Jones Transcript of the talk Slide 1: Slide 2: Hello, My name is Devon Jones and I am the Architect of Web Applications at Vonage and this is Stephen Becker, one of our talented engineers. We are going to talk to you today about how we have used Rails with our legacy systems at Vonage. We'll cover some of the lessons that we have learned, the experiences we had, and share with you the strategies that have worked for us. We'll keep this talk shorter because it's hard to cover all of the possible problems people can encounter when working with Legacy systems, given that every legacy system is unique. Slide 3: We have adopted Rails in a big way at Vonage. We are using it for a number of customer facing systems such as: * Subscribe - our new subscription system, that replaces an existing J2EE app * VFax - A system that allows you to upload a document that will be transcoded to a PDF, and then faxed to a US fax number. * Others - Other systems that are in development that are as of yet unreleased We are also using it for a number of internal systems like: * Metrics Mole - A system that exposes statistical data from our legacy systems to various business people. * Restful Services - A wide variety of restful services that we have built to help get us off of our legacy. * Others - we have found that we can use rails to quickly build systems to solve small but persistent issues in the business When I arrived at Vonage a bit over a year ago, the company was facing a software crisis. Vonage had grown from 0 customers to over 2 million in less then 5 years. The development activity necessary to support this focused on expediency and not maintainability in order to support this massive level of growth. This lead to a set of legacy systems that have become resistant to change. In order to address this, we created a new Grand strategy of software development at Vonage. Slide 4: Our Grand Strategy is: "In order to achieve the company's vision of changing the way the world communicates, we must transform our legacy applications into agile systems composed of small programs with transparent interfaces." Slide 5: This strategy is composed of a few key strategic elements. Instead of replacing our systems with one or two giant projects, we have focused on changing our system gradually and consistently by using small projects. Further, in the grand unix tradition, we have focused on crafting small programs that encapsulated single business processes or business decisions instead of large apps that tie the logic of multiple processes together. Finally, we tie these programs together using transparent, textual interfaces. This strategy was arrived upon by observing numerous projects. In my experience a typical project to replace a legacy system would go something like this: The business would start off by coming to the conclusion that one of it's systems is no longer amenable to change, and can no longer support the business' needs. SO a project would be started to replace the existing system. Using conventional wisdom, this project will follow the waterfall model, and will start off by creating a set of large documents. Functional specifications and design documents will be written, and people will hope that they have found all of the processes and features that the legacy system is used for. These documents will charter a huge scope that will result in a multi-month coding effort. When actual code hits disk, the scope of the project will start to creep. The reason for this is that when the system is actually started, the programmers will start to run into numerous missed requirements that were really only documented in the code of the old system. A line will be (hopefully) drawn to stop further scope creep as the project slips from it's schedule. Finally the system will be tested and released. The resulting system will likely be a disappointment to management, because as a result of stopping the scope creep the new system cannot actually replace the existing legacy system. We recommend against this strategy. Slide 6: Working on a legacy system is like working on the engine of a car that is going 60. Every move you make must be made with care. You need to always have a working alternate for the piece you are touching, and if you make a mistake, you are screwed. Instead of tackling a task like this all at once, you want to attack it with small projects... very small projects, and lots of them. Slide 7: In 2004, the Standish Group surveyed over 13,000 projects and found that the larger a project is, the more likely it is to fail. This jives with my personal experience. Large projects are disasters waiting to happen. Small projects are consistently more successful. Slide 8: The people at your company don't necessarily oppose new technology in their system, what they fear is the risk that change introduces into their lives. Replacing the legacy system over the course of a long term migration of code allows people to acclimate to the new way of doing things. Replacing a legacy system is a long term thing, and you need political capital to be able to move forward with any strategy. By changing the system gradually and consistently you can build a portfolio of success. This is why you need to deliver single features and do so on a regular basis. This will sell your management and coworkers far more effectively then just about anything else. Attacking the problem as small projects also means that each step is in itself low risk. By breaking the project up into small chunks it gives you a much greater ability to react to changes in the business. For the large classical project, it may be months before the project reaches the end of a phase where it can react to a changing environment. For a long series of small projects, you can adjust course every few weeks. Ensuring that the projects are investing in things that align more closely with the business' needs today as opposed to its perceived needs months ago. Slide 9: Eric Raymond said in the Art of Unix Programming "Write a big program when it is clear by demonstration that nothing else will do", and I couldn't agree with him more. The problem with J2EE is that it transforms an entire program ecosystem into one giant application. By forcing it's data models to be flexible enough to work for every business process that needs access to them, it drives the programmer to create really heavyweight models, and thick layers of glue code to try and keep every bit of code reusable. Don't replicate this. Instead, write small programs that encapsulate single business processes or single chunks of business logic. Although this breaks the conventional wisdom, don't focus on universal code reuse across business processes. Focus instead on program reuse. In order to meet this goal, you of course need to be writing programs that are reusable. You can do this most effectively by setting expectations with the coders who will be calling your program. If you are able to use a textual return, such as XML, or just plain (but structured) text, you can create a development contract with your downstream consumers. They should look to the return text only for the values they care about, and ignore the rest. The classic Larry Wall quote of "Be strict in what you emit and liberal in what you accept" applies here. If your consumers reject your return value because you add more return data, you are stuck with that specific interface. If everyone agrees to use what they need and ignore the rest, you are free to alter the code to meet other use cases beyond the initial ones you designed for. This cuts straight to the YAGNI (You Ain't Gonna Need It) concept by allowing you to code only for what is needed today, while allowing for easy growth. Slide 10: This leads well to the discussion of interfaces. Keep your interfaces transparent and textual. By using interfaces that are easy to read, and can be called at any time (instead of only working in specific system states) you have a major debugging advantage. You no longer need to take say a large application server, spend 20 minutes clicking things to get to a specific state, and then run your test. You can call each step of the system directly with known inputs, and read what it outputs. You don't need tools that identify the state of things out of memory or a binary tool when you can look at a chunk of (simple) xml that a specific process generates. When trying to find where something is breaking down, it becomes trivial to log the call state between programs, because it's all text, and it's all decoupled. Functional and integration testing with this style of interface is trivial. You can wrap up as many or as few pieces of the process as you want in your test, while stubbing the calls to and from any other components of the system. Slide 11: Now that we have discussed strategies, lets look at some specifics. This example will look at attacking a legacy J2EE application. (Note: in presentation, asked how many people maintained legacy J2EE applications. Maybe 10% raised their hands. I then asked how many hated it. Same 10% of hands raised.) This strategy is not specific to J2EE, but it makes for a good example. Slide 12: In this example I will show how you can isolate and extract a specific feature from a legacy system. We'll do this by creating an interface into the code, writing automated tests and then transforming it into it's own small program. Finally we can shut off a slice of code from the Legacy System Slide 13: This is a pretty typical Legacy J2EE system. It suffers from numerous problems ranging from Half implemented re architecture attempts, design pattern poisoning, binary interfaces and chunks of the system that are not understood well (or at all). This system probably has lots of features that the business needs, and our first task is to try to chart some of them out. Spend some time using the legacy system as an actual user. Try to identify as meany features as you can at the interface level. The list you create will by no means be exhaustive, but it helps to start out with a menu of features to work from that you may want to consume elsewhere. After you have done this, you will want to identify a feature you need elsewhere. Slide 14: In this example we'll say that this feature is the ability to create an account. Now, you have no idea how this is actually implemented. This is fine, because realistically there are thousands of systems out there that literally no one at the company understands, so any strategy for dealing with these systems needs to accommodate this. Slide 15: So get out your debugger, and start at the top level of the code where this feature starts from, and follow it through the code. In chatty systems like java, this is relatively easy. You can watch the logs, find a good logging message to grep for, and the attach your debugger there. An aside, Java is somewhat easier to do this with, because any jvm gives you the ability to add flags on the command line to allow your chosen debugger to attach remotely. We have included links to a number of articles on how to do this on our del.icio.us page (http://del.icio.us/vonage_railsconf_2007). Now, follow the code through the system until you identify a choke point. Slide 16: This will be a place in the code where all of the data is in one place that then calls lower level functions that go off an actually implement the feature. By necessity this exists, even if it is all the way up in the JSP, there has to be a place where everything exists in memory at once and then runs off to start writing things out. Slide 17: What you have done here is identified where in the code the actual feature is. Now you will want to slap a restful servlet on top of it. Slide 18: This doesn't have to be anything complicated. It can be as simple as a servlet that responds to GET or POST and takes or returns some simple xml. This will work with any environment, because there are effectively no languages in use today that can't speak HTTP or write out XML. Slide 19: You are now able to slap your shiny new rails app on top of this service. Realistically you will probably need more then one service to implement anything interesting for which you can follow the same process that is outlined here. You could also read directly form the database, but it's important to note that at this point you have not yet teased out the actual logic of the old code yet, so it's not safe at *all* to write to the database. From here on in, you need some discipline, because although you now have access to your feature, you have not really made the legacy system any better. All that has been done is you have added another layer (even if it is a simple one) to the old system. Now it's time to shut some code off. Slide 20: First step in eliminating code is to port the old interface to the new restful service. Instead of calling in to the old code path, have it make the http request to the restful service. Slide 21: You now know a fair amount about this feature. You have implemented a service on top of it, and are using that service from two separate applications. Now is the time to write a set of functional tests against the service. This is another good place for ruby. Just write some simple tests that call the service via http, and verify the xml against some that you have persisted to disk. Slide 22: You can acquire jdbc drivers that are out there that will log the sql that passes through them. We've included some in our presentation links. Install such a driver, and then start running your functional tests. If your tests are broad enough, you should see a pretty representative set of data getting written out. Your next step is to add active record to your functional tests, and use it to query the database after making restful calls to ensure that the data is properly written out. Slide 23: You now have the tools you need to implement this feature cleanly outside the old system. You can re implement the service, comfortable with the knowledge that you have a functional test suite that can ensure that the service behaves exactly as the old system did. At this point, you have the choice of any language and any environment that you want for implementing the new service. After the new service is complete, and passes the tests, you can point the apps at this service instead of the one build on top of the legacy system. You are new free to actually remove a slice of code from the old system. Slide 24: All you have left to do is pick your next target. This method is simple, low risk and cheap to implement. It keeps your old apps working, while giving you the ability to start innovating on parts of your system that were previously deeply cemented in. Slide 25: Now I'm going to hand this over to Stephen who will talk to you about the tactics we have used when implementing rails on top of legacy systems. He will discuss how to deal with bad database designs, bad data, and how to integrate rails applications into your existing application suites. Slide 26: Now that you have started to tame the shrew that was the old system, you can use rails to access the database with some confidence. Along with the growing system there are growing databases. The database will be abused, become filled with last minute work a rounds, ½ implemented ideas, fat fingered data mistakes and other cruft.. The database documentation will most likely be lacking. When trying to configure active record to work with the old database, it would be nice to have a document that answers the whys. Why things are replicated, why this table exists. Along with a lack of documentation there might not be any original staff to question. Then you have a non-standard database schema to deal with. This is not a horribly difficult task, but it will just require a lot of custom code in the models. Some of the data can be represent in redundant ways. For example; 8 ways to represent false or 6 ways to spell terminated. Saving date times as varchars, and using columns to store comma separated values. Although there will be a lot of custom code, you can still keep your models dry. Slide 27: In this example we implement a module that can deal with CSV and Date columns. LegacyAr is nothing but a wrapper for declaring methods. Slide 28: Here is the code for converting a Comma Separated Value column in to an array, and joining the array back together in the setter. Slide 29: Here is the code snippet for converting a date stored in a varchar. The formatting you see is from the postgre's date format. These are just handy methods for converting some database mishaps in to logical ruby values. You will most likely create this code as you go. So if dealing with bad database values is not difficult, what is the hard part? Most of the model will be configuration. Rails works great with its own conventions, and convention over configuration is a key concept, but it feels like the fact that it can be configured is over shadowed by convention. Slide 30: This is 1/3 of the configuration for a model. This model has a lot of relationships and each one needed its own rules. There is a need to exclude columns that are using reserved words, such as call back, and there is a need to alias columns (http://wiki.rubyonrails.org/rails/pages/HowToAliasColumnNames). The code for both can be done to look like simple configuration. The configuration might consume a fair amount of developer time but once completed can be used across many applications. Slide 31: Llamas are dangerous, so if you see one where people are swimming, you shout... Slide 32: Just like the back end systems, you will want to change the user interface one work flow at a time. To the end user the transition from one interface to the new rails application should be seamless. One way to achieve this effect is with transparent proxies. This is embedding the proxy to the new rails application so that users never directly interface with it. This proxy can pass only the data that is needed for the work flow. With proper network configuration the only application that can interface with the new rails application making it more secure. There will be no direct access to the application and the only inputs will come from your old application. Some examples of proxies are noodle and transproxy with squid. Another solution is a single sign on key. This generates a key and passes between applications for authentication also achieving a seamless effect. Choosing between these 2 solutions will depend on how much of the legacy code needs to change. Its a choice between this link loads this page from another server, and implementing key generation and authentication. Slide 33: Did it work? Some things did. We now can produce new services quickly. More services, more functionality, more features. The new code (hopefully) is documented, clean, and clear, not clever and hacky and difficult to maintain. There is a strong push for testing at all points during the rewrite. Tested code helps reduce the time to market by catching problems earlier in the software life cycle. Reduce lines of code. Fewer lines of code means less code to have problems with. Slide 34: We also use rails for new projects that allow us to represent our data in new ways. Creating maps of customer data, Creating metric graphs, Project management, and exercising our legacy systems with test harnesses. These projects were all rapidly prototyped with rails and have become very common tools. Slide 35: Is it ready for the enterprise, Yes.. kinda.. for some tasks, but not all. Rails is good for rapid prototyping new applications and services. Non extreme performance services. What does extreme performance mean? You can prototype the service in rails and do load testing. Does the service perform well enough? What is the ratio of cost (developer time or hardware) versus performance increase? How critical is the service to the business? These are just some of the questions to consider. Slide 36: Some things we are working would like to see from rails and the community are. A greater focus on apache. Rails works with apache not in apache. This issue might be solved with jruby and deploying with war files. A ruby VM. After all a faster ruby means a faster rails. User commented documentation. Although reading blogs about o problems and solutions is fun, A centralized location for comments would be much easier. generic in support for cookie free sessions. We have read the work a rounds for but it would be nice if it was easily configurable or a mix in. Default composite keys. We are currently converting to the rubyforge project. Slide 26: Now that you have started to tame the shrew that was the old system, you can use rails to access the database with some confidence. Along with growing legacy code, databases can grow out of control. Likely the legacy database will have suffered years of abuse, been filled up with last minute work a rounds, half implemented ideas, fat fingered data mistakes and other cruft. The database documentation will most likely be lacking. When trying to configure active record to work with the old database, it would be nice to have a document that answers the whys. Why things are replicated, why this table exists. Likely you don't have this. Along with a lack of documentation there might not be any original staff to question. So you have a non-standard database schema to deal with. This is not a horribly difficult task, but it will just require a lot of custom code in the models. Some of the data can be represented in redundant ways. For example; 8 ways to represent false or 6 ways to spell terminated. Saving date times as varchars, and using columns to store comma separated values. Although there will be a lot of custom code, you can still keep your models dry. Slide 27: In this example we implement a module that can deal with CSV and Date columns. LegacyAr is nothing but a wrapper for declaring methods. Slide 28: Here is the code for converting a Comma Separated Value column in to an array, and joining the array back together in the setter. Slide 29: Here is the code snippet for converting a date stored in a varchar. The formatting you see is from the postgres' date format. These are just handy methods for converting some database mishaps in to logical ruby values. You will most likely create this code as you go. So if dealing with bad database values is not difficult, what is the hard part? Most of the model will be configuration. Rails works great with its own conventions, and convention over configuration is a key concept, but the fact that it can be configured is overshadowed by those conventions. Slide 30: This is 1/3 of the configuration for one of our models. This model has a lot of relationships and each one needed its own rules. There is a need to exclude columns that are using reserved words, such as callback, and there is a need to alias columns (http://wiki.rubyonrails.org/rails/pages/HowToAliasColumnNames). The code for both can be done to look like simple configuration. The configuration might consume a fair amount of developer time but once completed can be used across many applications. Slide 31: Llamas are dangerous, so if you see one where people are swimming, you shout... Slide 32: Just like the back end systems, you will want to change the user interface one work flow at a time. To the end user the transition from one interface to the new rails application should be seamless. One way to achieve this effect is with transparent proxies. This is embedding the proxy inside the legacy application so that your users never directly interface with with the rails app. This proxy can pass only the data that is needed for the workflow. With proper network configuration, only your legacy application will be talking directly to the rails application, which has a definite security benefit. Some examples of this kind of proxy are noodle and transproxy with squid. Another solution is a single sign on key. At login you generate a key and then pass this between applications for authentication, alowing the site to feel more seamless. Choosing between these 2 solutions will depend on how much of the legacy code needs to change. It's a choice between this link loads this page from another server, and implementing key generation and authentication. Slide 33: Did it work? Some things did. We now can produce new services quickly. More services, more functionality, more features. The new code (hopefully) is documented, clean, and clear, not clever and hacky and difficult to maintain. There is a strong push for testing at all points during the rewrite. Tested code helps reduce the time to market by catching problems earlier in the software life cycle. Reduce lines of code. Fewer lines of code means less code to have problems with. Slide 34: We also use rails for new projects that allow us to represent our data in new ways. Creating maps of customer data, Creating metric graphs, Project management, and exercising our legacy systems with test harnesses. These projects were all rapidly prototyped with rails and are finding heavy usage. Slide 35: Is rails ready for the enterprise? Yes.. kinda.. for some tasks, but not all. Rails is good for rapid prototyping new applications and services. Rails is probably not the choice for services that need extreme performance. * What does extreme performance mean? * Does the service perform well enough? * What is the ratio of cost (developer time or hardware) versus performance increase? * How critical is the service to the business? These are just some of the questions to consider. Slide 36: Some things we would like to see from rails and the community are. * A greater focus on apache. - Rails works with apache not in apache. Apache is a gateway to a lot of enterprise features, and if rails were to integrate a little moer tigelty with it, it would not need direct code for things like Kerberos, LDAP, etc. This issue may be solved with jruby, so that ruby can instead leverage java's enterprise features. * A ruby VM. - After all a faster ruby means a faster rails. * User commented documentation. - Although reading blogs about problems and solutions is fun, A centralized location for comments would be much easier. * Generic support for cookie free sessions. - We have read the work a rounds for but it would be nice if it was easily configurable or a mix in. * Default composite keys. - We are currently converting to the rubyforge project.