Has Anatella an event driven architecture (such as in Azure Data Factory?)
- Thread starter Phil
- Start date
You can indeed react to "events" with TIMi/Anatella and perform "event driven" actions.
There are basically, four mechanisms to react to events inside Anatella:
1. the "build triggers" inside Jenkins.
Jenkins is one of the most well-known open source scheduler available today. TIMi/Anatella is 100% integrated with Jenkins (e.g. you can create new Jenkins job without even leaving your Anatella window). To start a job inside Jenkins, you use a "trigger". The Jenkins community adds new "trigger" every day. The most common Trigger is the "Build Periodically" trigger: it means that, at regular intervals (e.g. each day, each week, each month, etc.) you run a TIMi/Anatella job. ...But the "Build Periodically" Trigger is not alone: There are many other "triggers" inside Jenkins that "reacts to events": For example:
* when a file arrives on your drive
* when a ftp site gets a new file
* when an http URL is called
* when an email arrives
* etc.
Actually, Jenkins has already many "build triggers": see the list here:
https://plugins.jenkins.io/ui/searc...ries=&labels=trigger&view=Tiles&page=1&query=
If you are interested by "events" or "triggers", it's difficult to do something better than all the triggers available inside Jenkins!
You'll also find more details on the subject of Jenkins "triggers" on the point 22 of this page:
https://timi.eu/docs/anatella/index.html?4_8_2_installing_jenkins.html
I think that, using the "triggers" inside Jenkins is the closest thing to what is described inside the "Azure event driven architecture".
But, really, I don't know if you can call that "an architecture"!
..That's just some events that triggers the execution of some processes.
2. Anatella as a Webservice
Anatella can be used as a webservice to start different actions directly without any delay, as soon as the service is called/triggered.
You'll find more details on this subject here:
https://timi.eu/docs/anatella/index.html?10_8_3__general_purpose_web_service.html
We recently added a new "box" inside Anatella v3.20 that simplifies the task of creating a new webservice inside an Apache Webserver. This box automatically generate for you the code to put inside your Apache Webserver to create a webservice based on Anatella.
You should know that Anatella is coded in C and x86 assembly. So the Anatella startup-time is only a few milliseconds. This is in opposition to all our competitors that are coded either in Java (for example: Talend, Spark, Dataiku, etc.) or C# (for example: Alteryx). All the programs made in Java and C# are slow to start because, they all need to start the "virtual machine" before the real execution can start (and this is very slow). They also typically load ALL the libraries (all the .jar files) used inside the tool, that makes the starting-time even slower (In opposition, Anatella only load the libraries that are required to run the current data-transformation-graph, and not everything all the time!).
This means that all these competitors are only good for "batch processes": They are NOT good to create webservices since their startup time is very long (several seconds). In other words, if you use any of these other solutions, your webservice will be very slow (i.e. it will "freeze" several seconds at each start).
Also, you should be cautious about applications coded in C#: they will never run properly under LINUX (there are reliability problems because of the undocumented "virtual machine" made by Microsoft).
3. Anatella from the command-line
One very simple way to trigger the execution of an "Anatella Data Process" is through the Anatella command-line interface.
More details on the command-line here:
https://timi.eu/docs/anatella/index.html?4_7_batch_execution_anatella_command_line.html
We noticed that many young developers always want to run "data processes" as WebServices. This is a trend amongst young developers: For some reason, WebServices are "hype" ? Unfortunately, WebServices are also very inefficient: it can easily take several hundreds of milliseconds, for a webservice just to start (i.e. to establish the connection ; ...and only when it finished starting, you can start to compute the response). When you have millions of requests these milliseconds "adds up" and you soon have a very slow system. A much better/faster way to start a "data process" is through the Anatella command-line. So, if you want the "snappiest/best" response time, it's usually better to avoid any "API calls" (i.e. webservice) and use the command-line interface.
Anatella is designed so that it's very easy to call it from R,Python,PHP,etc. You can call Anatella and get back the execution log and the final execution status, to check that the call succeeded.
4. Run Anatella as an IOT Subscriber.
Anatella can also receive messages from a "message broker" and instantaneously react to it. More details here:
https://timi.eu/timi/anatella/iot/
https://timi.eu/docs/anatella/index.html?5_2_20_mqtt_subscribe_iot_connector.html
Direct bi-directional connections to common IoT brokers such as Kafka, RabbitMQ, Mosqitto is straightforward and easy.
=================
As you can see, there are plenty of solutions available to you to run your Anatella Data-Transformation-Graphs. Also, the .anatella file format is open and quite easy to understand so that you can generate some custom data transformation (i.e. some .anatella file) "on the fly" using a little bit of Python/VBA/PHP code, for example. We already have a few customers doing that.
To run Anatella in the 4 situation described here above (for event-based execution), you can run Anatella on a Windows server but also on a LINUX server. Typically, you design some new data transformations (i.e. you create a new .anatella file) on a Windows machine and, once you have your finished designing your data transformation (i.e. one you have your .anatella file), then you can execute it on LINUX: More details here: https://www.g2.com/products/timi-suite/reviews/timi-suite-review-4676265
==================
Also, to respond to your question specifically about Azure:
Anatella integrates native supports and direct connection to almost all Azure services.
Here is a few of the boxes inside Anatella that interacts with Azure services:
..and we are adding new AZURE boxes every day (and they run under LINUX too, if necessary).
All these connectors means that, if you are currently an heavy Azure user, Anatella is one of the best data management system for you: you get access to everything that is inside Azure in an easier and cheaper way. The best way to test-drive Anatella is to ask for a "Cloud Mutualized" offer (499€/month or 5000€/year): This way you get a very powerful cloud server (in Germany), that is "always on" to react to any event that might happen.
There are basically, four mechanisms to react to events inside Anatella:
1. the "build triggers" inside Jenkins.
Jenkins is one of the most well-known open source scheduler available today. TIMi/Anatella is 100% integrated with Jenkins (e.g. you can create new Jenkins job without even leaving your Anatella window). To start a job inside Jenkins, you use a "trigger". The Jenkins community adds new "trigger" every day. The most common Trigger is the "Build Periodically" trigger: it means that, at regular intervals (e.g. each day, each week, each month, etc.) you run a TIMi/Anatella job. ...But the "Build Periodically" Trigger is not alone: There are many other "triggers" inside Jenkins that "reacts to events": For example:
* when a file arrives on your drive
* when a ftp site gets a new file
* when an http URL is called
* when an email arrives
* etc.
Actually, Jenkins has already many "build triggers": see the list here:
https://plugins.jenkins.io/ui/searc...ries=&labels=trigger&view=Tiles&page=1&query=
If you are interested by "events" or "triggers", it's difficult to do something better than all the triggers available inside Jenkins!
You'll also find more details on the subject of Jenkins "triggers" on the point 22 of this page:
https://timi.eu/docs/anatella/index.html?4_8_2_installing_jenkins.html
I think that, using the "triggers" inside Jenkins is the closest thing to what is described inside the "Azure event driven architecture".
But, really, I don't know if you can call that "an architecture"!
..That's just some events that triggers the execution of some processes.
2. Anatella as a Webservice
Anatella can be used as a webservice to start different actions directly without any delay, as soon as the service is called/triggered.
You'll find more details on this subject here:
https://timi.eu/docs/anatella/index.html?10_8_3__general_purpose_web_service.html
We recently added a new "box" inside Anatella v3.20 that simplifies the task of creating a new webservice inside an Apache Webserver. This box automatically generate for you the code to put inside your Apache Webserver to create a webservice based on Anatella.
You should know that Anatella is coded in C and x86 assembly. So the Anatella startup-time is only a few milliseconds. This is in opposition to all our competitors that are coded either in Java (for example: Talend, Spark, Dataiku, etc.) or C# (for example: Alteryx). All the programs made in Java and C# are slow to start because, they all need to start the "virtual machine" before the real execution can start (and this is very slow). They also typically load ALL the libraries (all the .jar files) used inside the tool, that makes the starting-time even slower (In opposition, Anatella only load the libraries that are required to run the current data-transformation-graph, and not everything all the time!).
This means that all these competitors are only good for "batch processes": They are NOT good to create webservices since their startup time is very long (several seconds). In other words, if you use any of these other solutions, your webservice will be very slow (i.e. it will "freeze" several seconds at each start).
Also, you should be cautious about applications coded in C#: they will never run properly under LINUX (there are reliability problems because of the undocumented "virtual machine" made by Microsoft).
3. Anatella from the command-line
One very simple way to trigger the execution of an "Anatella Data Process" is through the Anatella command-line interface.
More details on the command-line here:
https://timi.eu/docs/anatella/index.html?4_7_batch_execution_anatella_command_line.html
We noticed that many young developers always want to run "data processes" as WebServices. This is a trend amongst young developers: For some reason, WebServices are "hype" ? Unfortunately, WebServices are also very inefficient: it can easily take several hundreds of milliseconds, for a webservice just to start (i.e. to establish the connection ; ...and only when it finished starting, you can start to compute the response). When you have millions of requests these milliseconds "adds up" and you soon have a very slow system. A much better/faster way to start a "data process" is through the Anatella command-line. So, if you want the "snappiest/best" response time, it's usually better to avoid any "API calls" (i.e. webservice) and use the command-line interface.
Anatella is designed so that it's very easy to call it from R,Python,PHP,etc. You can call Anatella and get back the execution log and the final execution status, to check that the call succeeded.
4. Run Anatella as an IOT Subscriber.
Anatella can also receive messages from a "message broker" and instantaneously react to it. More details here:
https://timi.eu/timi/anatella/iot/
https://timi.eu/docs/anatella/index.html?5_2_20_mqtt_subscribe_iot_connector.html
Direct bi-directional connections to common IoT brokers such as Kafka, RabbitMQ, Mosqitto is straightforward and easy.
=================
As you can see, there are plenty of solutions available to you to run your Anatella Data-Transformation-Graphs. Also, the .anatella file format is open and quite easy to understand so that you can generate some custom data transformation (i.e. some .anatella file) "on the fly" using a little bit of Python/VBA/PHP code, for example. We already have a few customers doing that.
To run Anatella in the 4 situation described here above (for event-based execution), you can run Anatella on a Windows server but also on a LINUX server. Typically, you design some new data transformations (i.e. you create a new .anatella file) on a Windows machine and, once you have your finished designing your data transformation (i.e. one you have your .anatella file), then you can execute it on LINUX: More details here: https://www.g2.com/products/timi-suite/reviews/timi-suite-review-4676265
==================
Also, to respond to your question specifically about Azure:
Anatella integrates native supports and direct connection to almost all Azure services.
Here is a few of the boxes inside Anatella that interacts with Azure services:
..and we are adding new AZURE boxes every day (and they run under LINUX too, if necessary).
All these connectors means that, if you are currently an heavy Azure user, Anatella is one of the best data management system for you: you get access to everything that is inside Azure in an easier and cheaper way. The best way to test-drive Anatella is to ask for a "Cloud Mutualized" offer (499€/month or 5000€/year): This way you get a very powerful cloud server (in Germany), that is "always on" to react to any event that might happen.