The technical details on the current and next release of EdgeX Foundry
Jim White, Dell Technologies IoT Platform Development Team Lead & EdgeX TSC Vice Chair
This week, EdgeX Foundry announces the release of Delhi, our third major release in 12 months. With each release that this growing community kicks out – I get a little nostalgic. EdgeX started its life in my kitchen. Like a lot of open source software efforts – it started humbly. In the case of EdgeX, it started on my laptop, propped open on my kitchen island one summer weekend about 4 years ago with an adult beverage or two.
Today, Edgex is being developed around the globe by a lot of tremendous software engineering talent. I was proud to start EdgeX, but I am even more proud to be part of an exciting group of international engineers building the most flexible, interoperable, open source IoT platform on the planet. This new release, Delhi, contains a new service and several key features that have our community and potential customers excited about the future of EdgeX Foundry.
Delhi Release
System Management
Most importantly, this release contains the first EdgeX system management capability. A new service – the System Management Agent (SMA) – serves as the coordinator for control plane information (status, configuration and metrics around EdgeX services) and control actions on EdgeX services (start, stop, and restart). Cloud or third party systems can call on the API provided by the SMA to trigger the actions or to get the control plane data they need. The SMA can serve as a one-stop shop for managing an instance of EdgeX. Each EdgeX micro service has a corresponding management API that the SMA calls on to help control that service (example – stop the service) or pull back its latest configuration or metrics. The SMA and management API provided by each service will be expanded in future releases of EdgeX and will one day offer control plane data and actions via alternate protocols (like LWM2M or SNMP).
Device Service SDKs
The original platform that started on my kitchen island was a Java platform. Last year, the community embarked on an endeavor to replace the bulky and slow Java services and tools with Go and C services and tools. Device Services – the EdgeX services that connect “things” to the platform are still in Java but not for long. With the Delhi release, two new Device Service SDKs have been created. The Go and C SDKs are allowing the community to create smaller, lighter, faster device/sensor connecting services that will allow EdgeX to operate in very limited resource compute environments – the thin edge in IoT solutions.
Improved Service Resiliency and Decoupling
In the past, the EdgeX microservice dependencies required the services be brought up with wait times between the startup of each service to allow the system time to bring things up in order. The services now are able to automatically check and detect when a dependent is in place and to become fully functional only after a dependent is up. This greatly reduces the start of all of EdgeX from minutes to seconds – around 5-10 seconds on average. The services now detect a dependent has gone down and will keep retrying that service until it comes back up. This gives EdgeX more resiliency than it had in the past.
EdgeX User Interfaces
EdgeX was built to facilitate machine to machine communications. As such, it did not have a user interface. With the Delhi release, user interfaces were created – largely to help visually showcase EdgeX functionality and to help developers – these UI also serve to show how EdgeX can be driven and operated.
Improved Core and Supporting Services
Many of the core and supporting services in EdgeX were improved and more unit testing was added to allow the community to keep an eye on the quality of the overall system and compatibility of future features of EdgeX. The Scheduling Service was also transitioned from Java to Go.
More Secure
Security services were initiated in our last release (California). The Delhi release includes the next wave of security features such as access control to grant access to appropriate services, and improved security service bootstrapping.
Better Tested
As mentioned, more unit tests were added to the EdgeX services. Additionally, automated blackbox tests are in place for each service to make sure the public API of each service performs as documented.
Ecosystem Growth
In addition to all this hard work, our EdgeX Foundry ecosystem continued to expand during this release cycle. Among the new companies to join EdgeX Foundry are Basking Automation, Data Ahead, Intel, Redis Labs, and ZEDEDA. This community also launched the availability of a developer kit and community demonstrator (for showcasing EdgeX during conferences and IoT events). Find out more about these here. Stay tuned to the EdgeX news and announcement pages as some big organizations are preparing to announce their joining early in the New Year.
Edinburgh
OK – no time to rest on our laurels. What’s next EdgeX? I am glad you asked! Indeed, the EdgeX technical steering committee (TSC) along with several community members met in Edinburgh Scotland to plan and scope our next release – code named Edinburgh – which is scheduled for April 2019 (I assure you the release name and TSC meeting place alignment happened a little serendipitously).
As Keith Steele, Chair of the EdgeX Foundry Technical Steering Committee, reported in his blog post, the results of the meeting were “superb” – as the Scots like to say.
The Edinburgh release has been scoped to include a rather significant amount of new features as well as making improvements on the current code base (“cleaning up technical debt” as we like to say). Given the increase in community membership and number of contributors to the project, we feel this scope is within reach. Here are some of the major efforts the membership has planned for the next 5-6 months.
Binary Data Support
The Edinburgh release of EdgeX will support the ingestion, use, and export of binary data (using CBOR format). To date, EdgeX has supported ingesting, using and exporting integer, float, string, and boolean types of discrete data collected from sensors. Many of the edge use cases involve video images, audio data, and other data that is binary data serialized.
Automated Performance and Security Testing
EdgeX has come a long way over the past few releases in increasing and improving its quality through testing. In the Edinburgh release, performance and security testing will be automated.
Cornucopia of new Device Services
The new Go and C Device Service SDKs created with the Delhi release allow the community to create smaller, faster, less-resource consuming replacements for the Java Device Services that now serve to connect “things” to EdgeX. This includes Modbus, BACNet, BLE, MQTT and SNMP Device Services. As there has been a bit of a pent up demand to get new Device Services for EdgeX due to the development of the SDKs, we believe this release will contain even more thing connectors than we could have imagined a year ago.
The paint on the new SDKs isn’t even dry and we already have the community showing off some early prototypes of new Device Services today and the numbers are impressive. In one small example, our old Java Modbus Device Service was 141MB in size and consumed 184MB of memory. The Go version produced with the new SDK is 16MB in size and uses just 8MB of memory.
Application Services
The current EdgeX Export Services, while functional, create scalability issues. The current EdgeX capability relies on the Export Services to know and understand all possible endpoint distribution mechanisms and transformation capability necessary to support all clients – even if the use case requires only a single, simple client (like sending the raw sensor event data to an MQTT pipe in JSON). As the number and type of EdgeX north side clients (cloud, enterprise and on-prem IoT servers) expands, this service will be too big and complex to support the north side needs.
The Edinburgh release will feature a new type of “exporting” service – the Application Service. The initial and simple Application Services in the Edinburgh release will be designed and implemented to support smaller, more tailored exportation needs. These services will contain just the functionality (filtering, transformation, enrichment, etc.) needed for a single endpoint. To address multiple endpoints, multiple Application Services will be created (versus having one massive export service as EdgeX has today). In a way, the Application Services will begin to look more like the south side Device Services – that is functionality dedicated to a particular client protocol and data need. Long term over the course of a number of releases, Application Services will replace EdgeX Export Services as the north side distribution facility.
Improved User On-boarding & Support Plan
EdgeX Foundry has been a developer driven effort. As the platform is and will continue to be used in real world IoT solutions, it becomes imperative that the community improve the quality and volume of documentation, tutorials, examples, videos, etc. to help user communities go from day 0 to production as quickly as possible. It is also imperative that EdgeX institute a long term support plan/strategy to give the user community assurances about what they can expect from the open source effort they put into their products. The TSC Chair (Keith Steele) and TSC Vice-Chair (me) are dedicating our resolve to address these two concerns as part of the Edinburgh release cycle.
Certification Program
A certification program will be outlined in concert with the Edinburgh release. A certification program will enable third parties creating EdgeX services to verify their services as alternative or enhancing capability to those provided by the EdgeX open source effort. This will allow 3rd parties to add value (proprietary or open source) to EdgeX that customers can rely on to meet the EdgeX APIs and work without additional code change (enabling a plug-and-play ecosystem). Various levels of certification are being considered, from micro service replacement certification (validating alternate or commercial implementations of EdgeX micro services satisfy API requirements along with performance metrics and quality checks) to full EdgeX deployments (for commercial versions of EdgeX). Additional certification processes may be developed around particular cross cutting features such as security.
And More…
- Use of Vault namespaces for storing secrets for the micro services.
- Initial EdgeX secrets (needed to start Vault/Kong) will be encrypted on the file system using a secure storage abstraction layer – allowing other implementations to store these in hardware stores (based on hardware root of trust systems)
- Refactor EdgeX database-using services (Core Data, Metadata, Export Client, Logging, Notifications, and Scheduling) to be more loosely coupled to the persistence mechanism (currently MongoDB). This will better facilitate the use of alternative persistent stores and technology in future implementations and even allow the project to select alternate or additional reference implementation databases in future releases.
- A smaller, lighter, faster rules engine service to replace the last of the EdgeX Java micro services.
- System Management will offer service health/status checks and additional metrics
- Updating the versioning and dependency management system for all Go micro services (replacing a deprecated technology)
- Upgrading all 3rd party open source tools included with EdgeX (Consul, Kong, Vault, etc.) to use the latest releases
For a full list of the Edinburgh release scope and roadmap, see the project Wiki.
It’s a large scope, but the community and team of developers is growing. Contact our Developer Advocate, Michael Hall, if you and your team would like to be a part of our effort – it’s an exciting time to be part of an exciting and industry impacting project. I have every confidence the EdgeX Edinburgh release will make some news in April 2019… as long as I can keep the team away from all the Scottish whisky we brought home from our trip!
