Real-time event processing with Azure Database for PostgreSQL and Event Grid integration
Most modern applications are built using events whether it is reacting to changes coming from IoT devices, responding to a new listing in a marketplace solution, or initiating business processes from customer requests. PostgreSQL is a popular open source database with rich extensibility to meet the event-based notification and distributed design needs of the modern application. PostgreSQL’s Notify functionality allows for sending a notification event as change feed to the listener channel specified in the database. With serverless platforms in Azure such as, Azure Event Grid a fully managed serverless event routing service, Azure Functions a serverless compute engine, and Azure Logic Apps a serverless workflow orchestration engine, it is easy to perform event-based processing and workflows responding to the events in real-time.
Consider a marketplace e-commerce solution where buyers meet sellers. A typical marketplace solution is a collection of microservices providing a seamless buying and selling experience to the end users. The modern microservices design leverages purpose-built app platforms and data stores, which are optimized for scenarios while working in tandem to achieve a unified experience for end users. For example, graph store is better suited for recommendation engine, while a relational datastore like PostgreSQL is suited for relational inventory data and NoSQL document stores needs. This architectural pattern is also referred to as polyglot persistence derived from polyglot programming. One of the challenges with polyglot persistence is synchronization between data stores so that writes into one data store are synced with other data stores. The integration of Azure Database for PostgreSQL service with Azure Event Grid enables this synchronization with real-time event-based processing.
In a marketplace solution, consider a scenario where a seller uploads or lists a new product to sell. It is an event which results in a new entry into the PostgreSQL inventory database and requires triggering of a caches refresh and re-indexing of search catalogs immediately to make it visible for new buyers and visitors looking for the product. In addition, if there are customers subscribed for the availability of that product type or category, a notification needs to be sent to them to alert them of the product availability as soon as the product listing is completed. Finally, the change feed needs to be sent to the data lake for real-time analytics and machine learning scoring so that it shows up in the recommendation engine. The workflow for this scenario in Azure platform with Azure Database for PostgreSQL and Event Grid is depicted below.
The notification from Azure Database for PostgreSQL service to Azure Event Grid simplifies the above requirement, making it easy to notify multiple subscribers of the topic when a new product is listed. The integration is enabled by leveraging Notify functionality which is available in Azure Database for PostgreSQL, and by defining a custom listener channel which publishes the changes to the Event Grid topic making it available to all the subscribers. The Event Grid topic for subscribers can be Azure functions, Azure Logic apps, and/or Webhooks in order to process the change feed and perform respective functions of triggering cache refresh, re-indexing of search catalog, or real-time scoring for recommendation engine.
We have published detailed instructions and a sample app with listener code in our GitHub repository. You can leverage the instructions to implement event based triggers and build scale out architecture with Azure Database for PostgreSQL and Event Grid. This allows you to unlock new scenarios and capabilities on Microsoft Azure. We encourage the developer community to reuse it and raise issues or contribute back by sending a pull request.
Get started and create your PostgreSQL servers today! Learn more about Azure Database for PostgreSQL in the overview and supported extensions documentation.
Please continue to provide UserVoice feedback on the features and functionality that you want to see next. If you need any help or have questions, please check out the Azure Database for PostgreSQL documentation. Follow us on Twitter @AzureDBPostgreSQL for the latest news and announcements.
Acknowledgements
Special thanks to Qingging Yuan, Bassu Hiremath, Rachel Agyemang, Harini Gupta, Sunil Kamath, and Raja Ravipati for their contributions to this posting.
Source: Azure Blog Feed