Slaying the behemoth, Extension Design Principles

Fifteen minutes reading time.

Once upon a time my ancestors roamed the icy plains of Northern Europe. They hunted behemoths, a single mammoth that could feed and clothe a tribe for a long while. But even though there was a lot of food on a single mammoth it was a bit of a pain to catch one. And once it is dead it is dead, and you need to eat it sharpish. So slowly they became smarter and started raising cows. Cows are standalone animals. Kill one, eat it, and the rest of the herd will continue to munch grass, produce milk, and be generally useful. So, we became cowboys. And survived.

Cool though the new job title of cowboy is it is nowhere near as cool as Slayer of Behemoths. That is why I offer a free Slayer of Behemoths t-shirt to the first 10 developers to successfully migrate a Dynamics NAV codebase to extensions. Details at the end of this post.

If you have been following me at all for the past months you will already know that at The Learning Network, we have been busy creating our first extensions. We have had our first extension in production since June and the grand total is now at eight, and several more in the works. It has not always gone easily though, but you probably read about that as well.

Related: Confession day, I messed up our extension

Now then why eight, and counting, extensions? Why not one big one? Why not wait and embed an ISV style extension in AL? Why even bother?

That is a lot of whys. We have been asking ourselves the same thing. Or not actually, we just enjoy building cool stuff and extensions are cool. Yes, they are. Period. So, we just started building. But just building cool stuff is not enough, it needs to be well engineered, well programmed, repeatable, reusable, testable, and maintainable. So, when we started planning the migration of one of our bigger business units to NAV2019 Business Central I set to work working out our design rules for building extensions. And since everyone is talking about migrating to extensions these days, I thought I would share them here.

Before I delve into the tech stuff some housekeeping. This list is by no means complete or accurate. It is a living guide. That means I will continue to update this post as I make lots of mistakes learn more. I you have tips or tricks please let me know.

1, Identify what to move into an extension by looking for the edge cases.

What is to be an extension? We don’t have time to redesign everything in one go. Therefore, we start looking for edge cases. An interface with our parcel delivery company. An interface with our order picking software. A redesign of our method of renting books that was happening anyway. Adding functionality to the job queue. Nothing that really touches on anything vital.

As we move the edge cases to extensions, we will find that the really complicated core modifications start to reveal themselves. There is simply less code for them to hide in. And when they do, we will have learned a ton about moving functionality to extensions.

2, Design to market

Working at an end user it is easy to create extensions for our own use only. But designing and building them for the marketplace will force us to not only design and build them better. Our extension will become a standalone microservice by default.

And because we build for the marketplace, we must do proper setup, testing, translation, and documentation. Remember, Microsoft does not require those things to annoy us. They require it to help us create the best extensions we can build.

3, Design all extensions as standalone microservices

This is the core of all our design principles. Moving to Business Central is just a small part of an overall redesign of our application landscape. For our landscape we decided on a system of standalone microservices connected using an events bus. So for Dynamics we also decided on a microservice extension architecture. Doing so enables us to create vertical and isolated extensions that interface neatly with our other microservices. That means a small multidisciplinary team can work on those isolated features as a single project.

In order to move our NAV2018 add-on to a microservice extension architecture we decided on a few ground rules.

  • An extension is a standalone unit. Either a vertical unit that extends certain functionality or a horizontal unit that extends the technical working of the Business Central platform;
  • All extensions will be built as an extension on the standard Dynamics NAV2018 codebase (our current version, to be upgraded to Business Central on premise soon). We do build new Business Central events into our NAV2018 codebase where needed;
  • No extensions will have dependencies on other extensions except where we extend third party extensions. This is because it is harder to maintain an extension when it has dependencies all over the place, every time you upgrade an extension you need to rework all its dependents. Also it is harder to install updates;
  • Extensions will be as small as possible but not smaller. There is no problem with a 500-object extension if that is the smallest it can be.

Using those rules, we end up with this model. It should be recognizable for those who attended Tech Days or Directions.

Would be Architechture

That is great, but it leaves two fundamental problems

  1. How to connect with the remaining custom C/AL objects;
  2. How to connect different extensions to each other.

We can solve these two problems in several ways.

Interface Extension.

An Interface Extension is a new extension that takes a dependency on the two separate extensions that it wants to connect. It is only used to connect extensions, no business logic will be in the interface extension.

An example of this would be an interface between our order import and our rental model. Both are vertical units but when importing an order with rented items on it the rental model business logic needs to do something.

To do this we use the events in the extensions we want to interface with. Because the interface extension is the one that is dependent our microservice extension remains a stand alone unit. That makes it much easier to maintain the entire application.

Interface extensions are also used to connect our new extensions with what remains of our old C/AL code. For smaller interfaces you might even use RecordRefs instead of dependencies.

You might want to check out James Pearson’s excellent 3 part series on integration.

https://jpearson.blog/2018/11/12/integration-between-extensions-in-dynamics-365-business-central/

https://jpearson.blog/2018/11/19/part-2-integration-between-extensions-in-dynamics-365-business-central/

https://jpearson.blog/2018/11/28/part-3-integration-between-extensions-in-dynamics-365-business-central/

Workflows

Business Central has a workflow module that can be configured by the users. Our extensions will add their events and actions to the workflow module, so our application managers can configure the desired integration between extensions. More on this in this blog post.

ForNAV

ForNAV is the reporting tool we use to create output like invoices and packing slips, or management reports. ForNAV can use tables and fields from all extensions without having to create a dependency.

Web services and API

An extension can have its own Web service or API that other applications or other extensions can interface with without the need for dependencies. You might even subscribe to a Business Central API or web service in an Interface Extension.

Our model then looks something like this:

Final Architechture

Some final remarks on microservice design.

  • We will probably have blocks of code that will be cloned in several extensions. If this is something we absolutely want to avoid then this code should be moved to Azure functions or an API;
  • We might end up duplicating data. That does not matter. A key feature of a microservice is that it keeps on working if all other applications fail. We don’t want our extension to stop working if someone deletes an extension that ours depends on for data;
  • We will probably find there are others that have already created an extension that does what we need. We will use them, so we can focus on what we uniquely need. Don’t forget that we won’t have control over those extensions though. We use them as an API.

Some more information on microservice design:

https://martinfowler.com/articles/microservices.html

4, Everything that can’t be done in an App Source extension gets moved out.

Do we just move everything we have to an extension? No, we don’t. I once said that we can do everything we should do with extensions. Or we will be able to in the very near future. But what is we should and should not do?

The three principles we use to decide if something stays in Business Central.

  1. We do not modify the standard code;
  2. We don’t use Business Central as a development platform;
  3. We don’t use Business Central for supporting functions like string manipulation.

The first principle means that we will lose some functionality. We still have modifications in standard code. They have got to go. Even though we request events where Microsoft has not added them already we will have to live with some loss of functionality. If we cannot live with this then Business Central is not the right platform for us.

There is of course a caveat here. Microsoft is still refactoring the core application to make it easier for us to extend it. But we don’t wait until Microsoft is done with this. We move what we can into microservice extensions. When Microsoft is ready we can move the rest.

The second principle means that for us Business Central is a way to log transactions. A way to reliably track money and goods. And that is it. It is now so ridiculously easy to create a well performing and easy to use database application there is really no point in using Business Central as a development platform anymore. learn C#, .NET Core, JavaScript, HTML and CSS, and especially the Entity Framework. Everything that is either not in the Core Business Central application or too heavily modified to be moved to an extension should be deported from Business Central. Besides, if you build your own application you won’t ever have to worry about merging it into Microsoft’s code. Bonus.

If we must have data or even the UI of our web-based application available in Business Central, we bring functions from multiple applications together in a single interface in Business Central using control add ins and React (or Angular, or jQuery, etc). More on this in a future blog post.

The third principle means that we will do away with the tons of supporting functions we have in codeunits. We use Azure functions or APIs to do our string functions, date manipulation, e-mail sending, weird calculations, or whatever.

Case in point, we just found a function in one of our codeunits to check if an e-mail address is valid. That is something we want to use consistently in all our web shops, ERP systems, and other applications. We will stick it in an API. One less function to migrate to AL.

5, Design as an API

Our extension is someone else’s API. Someone somewhere will extend our extension. Someone will create an interface for it. They don’t want to redo their work every time we release an update. That is a sure way of building an unpopular extension no one will use. Some tips.

  • Use proper API design guides. See this post for details on this;
  • Don’t ever break API (the API of an extension is all tables and events but could be argued to include pages, queries, and external functions). If we must break API, we create a new major version of the extension while keeping the previous version available. Check out how Google publishes their JavaScript add-ins;
  • Be generous with events that others can subscribe to. Add events where needed. Check how Microsoft does that using their AL GitHub. They are doing it right;
  • Add web services and APIs for our extensions. Create many ways that others can interface with.

Rounding up

I hope I gave you some ideas on how we work with extensions. I’m sure there are many other ways to deal with the move to extensions. This one just happens to work for us. I am always open for feedback, let me know what you agree or disagree with.

Finally, to win your Slayer of Behemoths t-shirt all you need to do is share this post and send me evidence of that plus evidence of taking a number of microservice style extensions into production. All judging is done by myself and will be final. MVP’s and Microsoft employees are excluded from participation.

Symbols of madness

As you know we have been happily developing extensions for a while now. Every once in a while I blog about some stuff I found out recently but it is mostly smooth sailing at the moment.

Today I was stumped though. For some reason my symbols were not loading. I make it a point to always develop extensions on a standard NAV2018 database (our current version, move to BC on prem is planned but not yet started) but sometimes I need to add functions or parameters that have been requested through Microsoft’s GitHub. And then of course load symbols. Now my symbols are always created automatically by using the finsql command line switch generatesymbolreference=yes as described here. And it always works. I thought.

So what had happened? I had updated my local NAV2018 to CU 9 a while ago. No problems there, just some bug fixes I needed for another extension. But when I added my C/Side function on my NAV2018 CU6 test server my symbols would not load. After checking my Launch settings about 100 times I finally checked my symbols files. That showed this.

Symbol versions

So my extension had symbols files for both CU6 and CU9. And of course the VS Code compiler uses the latest build. So I binned my .app files, downloaded symbols and published my extension without trouble.

As a bonus another symbols problem we found out a while ago. If you have multiple service tiers running your symbols wont generate with the standard batch script, even though it says it is working just fine.

finsql.exe Command=generatesymbolreference, Database="Demo Database NAV (11-0)", ServerName=.\NAVDEMO

What you need to do is add your service tier information.

finsql.exe Command=generatesymbolreference, Database="Demo Database NAV (11-0)", ServerName=SQLServer, navservername=Servicetier.domain, navserverinstance=DynamicsNAV110, navservermanagementport=7045

That should generate your symbols.

Have I forgotten something? Have tips and trick I don’t know yet? Please let me know.

 

On death and Dynamics NAV

Well that is a pretty depressing title. And what the hell does it mean? Does it mean I want Dynamics NAV to burn in the fires of eternal damnation? Has it’s time come?

Nothing so dark I must admit. This is an article about change, but not about change management. As a onetime victim of this medieval practice I am of the opinion that it should burn in the fires of eternal damnation.

So, on change. Dynamics NAV is changing and is becoming Business Central. For good or ill. I am a simple and optimistic guy so I think it is for the good. But time will tell, for all I know soon we might all work with SAP or some hitherto unthought of piece of technology.

Change seems to be a problem that nearly every organization I know wrestles with these days. That all important question seems to be “how do I get my people to change”?

This is a question that gets asked on a couple of levels. Microsoft wants their partners and users to adopt the latest technology. Their partners want their developers and consultants to start using the latest development tools and cloud offerings. Customers want their users to use the latest technology and newest processes. And all of them turn to change management. Which does not work.

“People don’t resist change. They resist being changed.” Peter Senge

But why? Change management has been around for years. You probably used it yourself right? And your projects succeeded. So what on earth is the problem?

To answer this we must first figure out what it is about change that is so difficult we feel the need to manage it. It is because our brains are wired to try to create a state of permanence. It has been clearly proven that we create our own reality out of the inputs that we get. The world we live in is constantly changing but it changes slowly. We don’t really notice it so everything seems permanent. Think of aging for instance. But every once in a while change happens so quickly that it upsets our image of the world. We need to adjust to that change, create an altered reality. And that is difficult. More difficult for some than others but it is easy for no one.

So what happens if we push the problem of change into the extreme? The ultimate form of change is death. It is the ultimate challenge to our reality. And if you ever went through a process of change you might have noticed that there is a mourning period you went through. I really cannot imagine how mourning can be managed. And that is precisely the problem I have with the term Change Management. It lacks the means to address our deep rooted fear of change that is ultimately fear of death. And it lacks the decisiveness that we need to adopt in modern day organizations.

So, I want to propose a new term here. Change Leadership. When we approach change from a leadership perspective we can start by taking decisive action. And then after getting the whole decision thing out of the way we can help people to adjust to the change. But how do you get people to face their fears and embrace change even though they are scared?

Lets stay with my exaggerated claim that change is death for a while. Inevitably we end up on a battlefield. And not just any battlefield, I want to take you back to 1916 to the battle of The Somme where a young lieutenant was leading his men into battle. That young lieutenant later wrote a prophesy that has got me thinking:

“The hands of the King are the hands of a healer.”

That young lieutenant was professor Tolkien, the author of the twentieth centuries favorite book. The quote is from The Return of the King and I think it says something deeply profound about leaders.

It does not mean only doctors make good leaders. It means that leaders need to be able to heal the damage they created. Think of Aragorn who leads his people into battle. A wise decision I’m sure we all agree. After all, he did help save the world. But when the battle is done there is much death, many are wounded. So even though he is weary he helps whomever he can, he draws forth arrows, disinfects wounds, and comforts the dying and the bereaved.

So when looked at from a business standpoint that means that while we have to be quick and decisive in changing our  business we must also realize that this change creates havoc and injures a lot of loyal employees or customers. So we have to heal the hurt we helped create.

But how to do this. I’m sure you won’t need to draw forth arrows on the battlefield. That is difficult but is at least straightforward. How do you heal the invisible wounds of the umpteenth reorganization?

To answer this I turn to two people who have influenced me deeply, Carl Jung and Carl Rogers.

Carl Rogers (1902-1987) was a humanistic psychologist who agreed with the main assumptions of Abraham Maslow, but added that for a person to “grow”, they need an environment that provides them with genuineness (openness and self-disclosure), acceptance (being seen with unconditional positive regard), and empathy (being listened to and understood). Quote found here.

For me this rings very closely to the words of Jung who states that in order to accept the people we try to help we must first embrace our own darkness.

“I am the oppressor of the person I condemn, not his friend and fellow-sufferer” , Carl Jung.

So, back to our old friend change. How can these venerable gentlemen help us to help others? It is by embracing our own fear of change that we can help others. We all share a common dread. That is death. In order to help others we must face our fear and accept it.

It is a fear that I share as well. What will happen if Business Central fails? I really don’t want to work with SAP. And I know Dynamics NAV so well, what am I going to do without it?

This realization fuels my compassion. So when faced with the next angry or fearful user or fellow developer I am able to listen and understand. And this understanding will bridge that gap between a failed and a successful project. Please try it. Whenever people speak in fear find your own fear. And respond from compassion. Respond to connect.

“When people think you’re trying to influence them, they put their guard up. But when they feel you’re trying to help them, or to muse your way to the right answer, or to be honest about your own imperfections, they open up to you. They hear what you have to say”, Susan Cain.

Authors note: This is an opinion piece. I wrote it on the premise of strong opinions, weakly held. If you don’t agree that is fine, I am open to civil discussion. Through discussion my understanding of the topic will grow and evolve. There is no place here for name calling and mud slinging though.

As always I am deeply indebted to those who wrote before me. Please read more from the fine people I quoted here. I stand on the shoulders of giants.

And many thanks to my dear wife for helping me to straighten out my arguments.

Dimensions of woe

So today I wanted to create a simple text variable with 3 dimensions in my extension. Normally no big deal but how on earth do I define dimensions in my variables?

It turns out they are now called Arrays. So that is an improvement but the documentation is worthless. And again, nothing to be found on the old Google. So after a frustrating time of figuring stuff out here it is:

Arrays

Why I named my company after Papa Smurf.

I have written a fair amount of blog posts so far but I realized today I had been remiss in explaining the name I chose for my business. Red and Bundle, what is that all about people ask me.

When I tell them I named my business after Papa Smurf their bewilderment only grows. To explain all this I have to take you back in time with me. To a place where life was not very different than it is now. But I was younger. About eight or nine years old in fact.

We were visiting my grandparents and my granddad let us watch the Smurfs. The particular episode was called The time capsule. It had druids and magic in it so naturally I thought it was excellent. In the period when everything looked bleak the smurfs had to solve a riddle.

“A twig by itself is too weak; it’s twigs in a bundle you seek.”

And for some reason that single line stuck with me in the back of my head for thirty years. A bit like a time capsule.

Explaining the riddle to the smurfs Papa Smurf said it is about working together. One Smurf alone cannot possibly win this.

To me that is the epitome of a great leader. To recognize that you cannot do this alone, to recognize that the people you work with have their own skills and contributions to make. And to help those people to become the best versions of themselves. To realize that Brainy, Hefty, Handy, and even Clumsy are geniuses in their own right. And to coach and guide people so the sum becomes greater than the parts.

So that is the role I decided I want to fulfill. To work with my clients to make the best of the people, software, and organizations I work with. Red and Bundle means that while we as IT specialists sometimes seem to be practicing magic we cannot save the world all by ourselves. If we want to overcome obstacles we have to work together.

Papa Smurf knows all this. He knows that each and every Smurf has their own nature and their own genius. Even if they drive him bonkers every episode. Because without them he would just be a lonely old Smurf.

Image found here.