Graduate Program KB

Cloud Integration

  • Inevitably, applications will need to communicate with each other when they are deployed

  • Two patterns of application communication

    • Synchronous: Application to application
      • Ex. Buying service --> Shipping service
    • Asynchronous / Event-based: Application to queue to application
      • Ex. Buying service --> Queue --> Shipping service

  • Sudden spikes in traffic can cause problems for the synchronous approach

  • Consider needing to encode a thousand videos even though it's usually 10, it's better to decouple your applications (these services scale independently from the application)

    • SQS queue model
    • SNS pub / sub model
    • Kinesis, real-time data streaming model

Amazon SQS

  • Simple Queue Service is a fully managed service to decouple applications

    • Oldest AWS offering (at least 10 years old)
    • Scales from 1 message per second to 10,000s per second
    • Default retention of messages set to 4 days, with a maximum of 14
    • No limit to amount of messages in queue
    • Messages are deleted after being read by consumers
    • Low-latency (less than 10 ms on publish and receive)
    • Consumers share the work to read messages and scale horizontally

  • The ordering of messages is FIFO (First In First Out)

    • Messages are processed in order by the consumer

Amazon Kinesis

  • A managed service to collect, process and analyse real-time streaming data at any scale
    • Kinesis Data Streams: Low-latency streaming to ingest data at scale from hundred thousands of sources
    • Kinesis Data Firehose: Load streams into S3, Redshift, ElasticSearch, etc.
    • Kinesis Data Analytics: Perform real-time analytics on streams using SQL
    • Kinesis Video Streams: Monitor real-time video streams for analytics or machine learning

Amazon SNS

  • Simple Notification Service is a managed service for allowing messaging between decoupled microservice applications
    • The "event publishers" only sends message to one SNS topic
    • Can have as many "event subscribers" as we want to listen to the SNS topic notifications
    • Each subscriber to the topic will get all the messages
    • Up to 12,500,000 subscriptions per topic, with a 100,000 topics limit

Amazon MQ

  • A managed message broker service for RabbitMQ and ActiveMQ

    • Doesn't scale as much as SQS / SNS (these are "cloud-native" services, proprietary protocols from AWS)
    • Runs on servers, can run in multiple AZ with failover
    • Has both queue feature and topic features similar to SQS / SNS

  • Traditional applications running from on-premises may use open protocols such as MQTT, AMQP, STOMP, Openwire, WSS

  • When migrating to the cloud, instead of re-engineering the application to use SQS and SNS, use Amazon MQ