eKuiper Issues 1.7.0 Release, Adding Lookup Table, Analytic Functions and More
eKuiper—a lightweight IoT data analytics and streaming software—is now available in its 1.7.0 release. eKuiper, an LF Edge project, migrates real-time cloud streaming analytics frameworks such as Apache Spark, Apache Storm and Apache Flink to the edge. eKuiper references these cloud streaming frameworks, incorporates any special requirements of edge analytics and introduces rule engine, which is based on Source, SQL (business logic) and Sink; rule engine is used for developing streaming applications at the edge.
eKuiper 1.7.0 continues to improve the expressiveness of the rules by providing the lookup table, updatable sink, and more SQL syntax and functions. The 1.7.0 release adds a number of significant new features, among them are
- Stream batch combined computation
- Not all data changes frequently, even in real-time computation. In some cases, you may need to supplement stream data with externally stored static data. For example, user metadata may be stored in a relational database, and the stream data only has data that changes in real time, requiring a connection between the stream data and the batch data in the database to enrich the data. In the new version, eKuiper adds the new Lookup Table concept for binding external static data, which can be joined with stream data in the rules to realize the operation of stream-batch combination. Additionally, with the support of updatable sink, eKuiper can process CDC data and update the batch storage.
- Enhanced analysis capabilities
- Analytic functions refer to functions that perform state-related analysis by saving state, allowing users to complete some of the common stateful calculations without complex time windows or custom functions. In the new version, we have added the Partition By syntax to the analytic functions, allowing users to perform partitioned stateful calculations based on the dimensions defined by Partition By clause. We also added a new analysis function latest to get the latest value of data. It is used to collect unstructured data whose columns are not fixed and can be automatically stitched to calculate the complete data.
- Expanding the connectivity ecosystem
- On one hand, we continue to add built-in and extended source/sink by adding Httppush source, which allows users to push data to eKuiper via HTTP protocol; adding built-in Redis lookup source, which supports using Redis as external lookup table; adding Influx V2 sink, which supports writing to InfluxDB 2.x version. At the same time, Memory, SQL, and Redis source are adapted to support lookup tables; Memory, SQL, and Redis sink are adapted to be updatable.
- Enhance the Ops experience
- The new release refactors the external connection configuration API to enable easier connection resource management. Another major update comes from the bulk import/export and initialization of streams and rules to facilitate the migration of eKuiper instances.
Learn more about these and other features of eKuiper’s 1.7.0 release in the release notes.
What’s next
In the next release, the community will work on enhancing graph API to support more nodes, such as switch node and script node, to provide a more powerful and easy-to-use rule pipeline composing tool. A new function to run tensor flow lite model will be provided to easily integrate with AI models. Check out the evolving Github 1.8.0 milestones and we welcome you to add your wisdom.
