The DivConq Framework stems from years of research and prototyping of application frameworks. The resulting design takes a pragmatic view of simplicity and elegance vs. power and flexibility. As much as simplicity and elegance are a goal, to get the most from the framework we adopted a few methodologies that may not be immediately intuitive. The antidote to this added complexity is that these methodologies are used over and over – once you learn one part of DivConq you’ll quickly adopt to the rest.
You define your date types and structures in one place and one time. JSON and YAML are wonderfully simple ways of structuring your data – basically they support objects, lists and scalars. And so does DivConq. Like YAML DivConq has data types associated with the values/fields, however, the point is to think simple like JSON does. DivConq does not generally promote the idea of using Java classes to represent data structures. Instead you may use some of the classes provided by DivConq to work with JSON like structures.
These structures alone do not constitute a huge design benefit, but when coupled with the other features in DivConq they become a huge benefit because DivConq is designed to seamlessly handle these structures. Your schema are defined in one place but they are shared by the database (dcDb), the services (dcServiceBus), the scripting engine (dcScripts), the web rendering engine (dcWeb) and the web services engine (dcRPC). When you create a object or list structure using DivConq’s classes you may pass that data to any of the major DivConq features without any additional serialization, special attributes or any other metadata.
One Data Engine
There are many times when table like data structures are all you need for your application. But there are those times when you need a faster or more complex data storage solution. DivConq provides solutions that should assist with almost all structured persisted data storage needs.
The data engine has additional design features such as auditing, replication and values over-time. Auditing can help track when and who made changes to your data. Replication makes it possible for multiple database servers to be active at the same time, each replicating to the other and with eventual data consistency across all servers. Values over-time may be used as a way to track changes to a single field over the history of a record – for example if the Name field of a record is updated the old name is still present and searchable, but the new name is the default.
SOA + Modularization
The features and components of an application should be modular – it has been said many times. A Service Oriented Architecture (SOA) has a lot of promise to that end. The challenge with SOA is that often it requires additional layers, meta data and complexity. With DivConq your services are defined in the schema, right along with custom data types and database tables. Indeed a service call looks a lot like a stored procedure call, which looks a lot like a method call in dcScripts. SOA is more likely to succeed in a framework where there is no additional effort to make data available from a service.
To succeed SOA must also be easy to deploy. When deployed on just one server there is no additional configuration effort, all service calls will be routed locally. When multiple servers are deployed a file describing the “Fabric” lists server (Hub) addresses and identities along with which services they should provide, a minimal fuss activity. Once the fabric is defined the servers will automatically connect and route service calls to correct remote servers.
Scripting + Task Collaboration
Sometimes data interchanges with external applications, and even exchanges between services within an application, require some sort of mediator. DivConq comes with built-in collaboration features to make it possible to share in-memory data across services/modules, or to share task progress (e.g. “step 2 of 5 complete”) and more. Custom collaboration component are possible.
A convenient and secure scripting language is also available to provide high flexibility and efficiency. The scripting language is inherently non-blocking and handles all commands with an “await” qualifier somewhat like C# so that running scripts take up very little time from the work pool threads.
Further, DivConq collaboration supports a claims feature (think locks) where one server may claim ownership of a task or of data for a period of time, thus preventing other servers from making the same claim.
Flexible, Scalable Deployment
As mentioned above, we want applications to have flexible options – from an all-in-one server to a one-of-many server approach. Although not limited to this, DivConq is well suited to an application with a web module, an application service and a database. With that example the web module and service could live in the same JVM which is in turn on the same server as the database:
Or it could live on separate servers like the following. Note how the app tier connects to the web tier instead of the other way around. This eliminates the need for a hole in the firewall between the DMZ and the LAN.
When many servers are present that provide the same feature (module/service) then requests/tasks are automatically load balanced over the servers. For example, below we see multiple app servers – as web requests come in from users so do requests for data and those requests will be sent back to the application servers in round robin fashion. The distribution of data requests in that way ends up balancing the data processing across multiple application servers.
Session state should be stored in the database, and as such the application and web servers are stateless.
Servers may be grouped into Squads. Each squad is fully active and has a complete and continuously replicated copy of the database. A squad may continue to operate even if it loses contact with all other squads. All members of a squad should be geographically close, but squads (relying only on replication) may be geographically distant. Further, squads may be outside of the company – for example one squad could be in the cloud (IaaS) and another could be on-premises. Indeed, squads could even be located in separate companies.
Going beyond deployment options, DivConq also intends to support easy adjustments to a deployment. If the load for a squad gets heavy then adding additional servers to support the load should be easy. Likewise, if the load is not heavy then removing or consolidating servers should be easy. Furthermore, because DivConq is intended to work with many servers, we feel that the framework should support an organized approach to software updates and patches.
To support those goals Divconq will support a master repository that contains all software updates and core (fixed) project files. Servers running DivConq may automatically self-update by connecting to the repository and downloading the updates, then self-restarting to use those new updates. Even database conversions can be automatically managed using this tool. Administrators can control when updates and patches are rolled out.
Which brings up the topic of projects. A project is one or more squads (each squad being one or more servers) that share a common application install – the same core configuration and business goal.
Any DivConq based application may be deployed as another project and serve a different business goal. In the end all DivConq applications can check back with the master repository to get updates for their specific project – other projects will be invisible.
The following shows two projects, each with two squads, that are deployed in three different data centers. Note how all the squads have connections to the single repository (a backup repository could be setup too). DMZ servers get their updates via the LAN servers, as DivConq DMZ servers are not allowed to connect directly to any server inside the LAN.
DivConq is carefully planed to make it possible for application components from various vendors and their customers to be integrated. The database tables can be shared and extended, the stored procedures have a naming convention, even the error codes have assigned ranges, locale translations can be overwritten, web sites can be extended, web pages can be overridden and much more.
When all the development guidelines are followed DivConq may support a high level of customization within each project. From the example above for master repository, project A and B may each share a good number of software components from various vendors, but they may also contain customization from the customer’s own IT staff which end up making the two projects notably different in use.