Local testing with live data means faster development with Azure Stream Analytics

We are excited to announce that live data local testing is now available for public preview in Azure Stream Analytics Visual Studio tools. Have you ever thought of being able to test Azure Stream Analytics queries logic with live data without running in the cloud? Are you excited by the possibility of not having to wait for your queries to deploy and other round-trip delays? Well, this enhancement lets you test your queries locally while using live data streams from cloud sources such as Azure Event Hubs, IoT Hub or Blob storage. You can even use all the Azure Stream Analytics time policies in the local Visual Studio environment. Being able to start and stop queries in seconds and local debugging significantly improves development productivity by saving precious time on the inner loop of query logic testing.

ASALiveTesting2

Live data local testing experience

The new local testing runtime can read live streaming data from the cloud or from a local static file. It works the same as the cloud runtime Azure Stream Analytics and therefore supports the same time policies needed for many testing scenarios. The query runs in a simulated environment suitable for a single server development environment and should only be used for query logic testing purposes. It is not suitable for performance/scalability and availability testing.

Key benefits

Query behavior consistency

For any given input, the queries running locally and in the cloud will output the same results. Live data testing in Visual Studio includes full support for time polices and machine learning functions.

Much shorter testing cycles

Compared to running in the cloud, it takes only seconds to start a round of query testing. It enables you to test your query directly in Visual Studio without having to switch environments. You can understand the shape of the data coming to easily adjust the query and see immediately the result on the live data.

Viewing job metrics and error messages

Job metrics such as input data size, output data size, and more are available during local execution. Error messages will also be displayed in testing and debugging.

Fluid experience

  • The query runs locally but connects to input data cloud sources directly. No need to sample data to local first.
  • Query results are immediately displayed in table format in Visual Studio.

Local testing options

In addition to the current local run capability, with this major improvement, we now support all the local testing options listed below.

Input

Output

Job Type

Local static data

Local static data

Cloud/Edge

Live input data

Local static data

Cloud

Live input data

Live output data

(Power BI and ADLS are not supported at this moment)

Cloud

Local testing vs. cloud run caveats

This local testing feature should only be used for functional testing purposes. It does not replace the performance and scalability tests you would do in the cloud. It should not be used for production purposes since running in a local environment doesn’t guarantee any service SLA.

While on your machine you can only rely on local resources, when you deploy to the cloud we can scale out to multiple nodes. Additionally, cloud deployment ensures checkpointing, upgrades, and more. It also provides all the infrastructure to run your job 24/7.

The following table shows the comparison between running a query in a local environment versus running it in the cloud. Please note, that only the cloud input options support time policies and machine learning functions, the local input doesn’t support these.

 

Query logic

JavaScript UDF/UDA

Machine Learning function

Time policies

Execution Environment

Resource Allocation

Service SLA

Scalability

Local testing with local input

Yes

Yes

No

No

Local simulated,

No CPU/memory

commitment

No

No

Local testing with live input

Yes

Yes

Yes

Yes

Local simulated

No CPU/memory

commitment

No

No

Running in the cloud

Yes

Yes

Yes

Yes

Cloud

By SU

Yes

Yes

Try it now

This feature is amazingly easy to use. Just choose the input/output type you want in the script editor and click “Run locally”. Follow this guide on Testing live data locally using Azure Stream Analytics tools for Visual Studio to try it!

Providing feedback and ideas

The Azure Stream Analytics team is highly committed to listening to your feedback. We welcome you to join the conversation and make your voice heard via our UserVoice. For tools feedback, you can also reach out to ASAToolsFeedback@microsoft.com.

Did you know we have more than ten new features in public preview? Sign up for our preview programs to try them out. Also, follow us @AzureStreaming to stay updated on the latest features.

Source: Big Data

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.