We recently made several exciting announcements about Azure HDInsight, Microsoft’s fully managed, full spectrum, open-source analytics service for enterprises:
1. Effective January 5, 2018, we have cut prices by up to 52 percent on all workloads on HDInsight, and added an additional 80 percent price reduction for Microsoft R Server add-on. As part of this price reduction, we are renaming the “premium tier cluster” to "Enterprise Security Package," which can be added optionally to your Azure HDInsight cluster.
2. General availability (GA) of Apache Kafka on HDInsight. The general availability of Apache Kafka will enable customers to build enterprise-grade, open-source, real-time analytics solutions such as IoT, fraud detection, clickstream analysis, social analytics, and more backed by Azure HDInsight's SLA of 99.9 percent.
3. Integration with Log Analytics—GA. Azure Log Analytics is now integrated with Azure HDInsight to enable enterprise-grade monitoring for mission-critical analytics workloads. Get alerts, monitor, and debug all your Azure HDInsight workloads.
4. Integration with Power BI DirectQuery in preview. Create dynamic reports based on data and metrics you already have on your interactive query clusters. Build visualizations on your entire data set faster with the most recent data.
5. Development tools update for Spark. This release adds support in Visual Studio Code for Spark on Azure HDInsight. Developers can build robust production data pipelines and debug distributed Spark jobs. This also brings in the interactive experience for Python and Spark in Visual Studio Code.
Source: Azure Roadmap