| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <meta http-equiv="X-UA-Compatible" content="ie=edge"> |
| <title>ActiveMQ</title> |
| <link rel="icon" type="image/png" href="/assets/img/favicon.png"> |
| |
| <link rel="stylesheet" href="/css/main.css"> |
| <script defer src="https://use.fontawesome.com/releases/v5.0.8/js/all.js" integrity="sha384-SlE991lGASHoBfWbelyBPLsUlwY1GwNDJo3jSJO04KZ33K2bwfV9YBauFfnzvynJ" crossorigin="anonymous"></script> |
| <script src="https://code.jquery.com/jquery-3.2.1.slim.min.js" integrity="sha384-KJ3o2DKtIkvYIK3UENzmM7KCkRr/rE9/Qpg6aAZGJwFDMVNA/GpGFF93hXpG5KkN" crossorigin="anonymous"></script> |
| <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.12.9/umd/popper.min.js" integrity="sha384-ApNbgh9B+Y1QKtv3Rn7W3mgPxhU9K/ScQsAP7hUibX39j7fakFPskvXusvfa0b4Q" crossorigin="anonymous"></script> |
| <script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/js/bootstrap.min.js" integrity="sha384-JZR6Spejh4U02d8jOt6vLEHfe/JQGiRRSQQxSfFWpi1MquVdAyjUar5+76PVCmYl" crossorigin="anonymous"></script> |
| </head> |
| |
| <body> |
| <nav class="navbar navbar-expand-lg navbar-light fixed-top"> |
| <div class="container"> |
| <!-- <a class="navbar-brand mr-auto" href="#"><img style="height: 50px" src="assets/img/apache-feather.png" /></a> --> |
| <a class="navbar-brand mr-auto" href="/"><img src="/assets/img/activemq_logo_black_small.png" style="height: 50px"/></a> |
| <button class="navbar-toggler ml-auto" type="button" data-toggle="collapse" data-target="#navbarContent" aria-controls="navbarContent" aria-expanded="false" aria-label="Toggle navigation"> |
| <span class="navbar-toggler-icon"></span> |
| </button> |
| |
| <div class="ml-auto collapse navbar-collapse" id="navbarContent"> |
| <ul class="navbar-nav ml-auto"> |
| <li class="nav-item"> |
| <a class="nav-link active" href="/index.html">Home</a> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link" id="navbarDropdownComponents" data-target="#" href="" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">Components</a> |
| <ul class="dropdown-menu dropdown-menu-center" aria-labelledby="navbarDropdownComponents"> |
| <div class="row"> |
| <div class="col-12"> |
| <ul class="multi-column-dropdown"> |
| <li class="nav-item"><a class="dropdown-item" href="/components/classic">ActiveMQ 5</a></li> |
| <li class="nav-item"><a class="dropdown-item" href="/components/artemis/">ActiveMQ Artemis</a></li> |
| <li class="nav-item"><a class="dropdown-item" href="/components/nms">NMS Clients</a></li> |
| <li class="nav-item"><a class="dropdown-item" href="/components/cms">CMS Client</a></li> |
| </ul> |
| </div> |
| </div> |
| </ul> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link" id="navbarDropdownCommunity" data-target="#" href="" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">Contact</a> |
| <ul class="dropdown-menu dropdown-menu-center multi-column columns-1" aria-labelledby="navbarDropdownCommunity"> |
| <div class="row"> |
| <div class="col-12"> |
| <ul class="multi-column-dropdown"> |
| <li class="nav-item"><a class="dropdown-item" href="/contact#mailing">Mailing Lists</a></li> |
| <li class="nav-item"><a class="dropdown-item" href="/contact#chat">Chat</a></li> |
| <li class="nav-item"><a class="dropdown-item" href="/contact#issues">Report Issues</a></li> |
| <li class="nav-item"><a class="dropdown-item" href="/contact#contributing">Contributing</a></li> |
| <li class="nav-item"><a class="dropdown-item" href="/security-advisories.html">Security</a></li> |
| </ul> |
| </div> |
| </div> |
| </ul> |
| </li> |
| <li class="nav-item dropdown"> |
| <a class="nav-link" id="navbarDropdownTeam" data-target="#" href="" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">Apache</a> |
| <ul class="dropdown-menu dropdown-menu-center multi-column columns-1" aria-labelledby="navbarDropdownTeam"> |
| <div class="row"> |
| <div class="col-sm-12"> |
| <ul class="multi-column-dropdown"> |
| <li class="nav-item"><a class="dropdown-item" href="https://www.apache.org">The Apache Software Foundation</a></li> |
| <li class="nav-item"><a class="dropdown-item" href="https://www.apache.org/licenses/">License</a></li> |
| <li class="nav-item"><a class="dropdown-item" href="https://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li> |
| <li class="nav-item"><a class="dropdown-item" href="https://www.apache.org/foundation/thanks.html">Thanks</a></li> |
| <li class="nav-item"><a class="dropdown-item" href="/security-advisories.html">Security</a></li> |
| <li class="nav-item"><a class="dropdown-item" href="https://www.apache.org/events/current-event">Events</a></li> |
| <li class="nav-item"><a class="dropdown-item" href="https://people.apache.org/phonebook.html?pmc=activemq">PMC & Committers</a></li> |
| <li class="nav-item"><a class="dropdown-item" href="/team/reports">Board Reports</a></li> |
| </ul> |
| </div> |
| </div> |
| </ul> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </nav> |
| |
| <div class="content"> |
| <div class="page-title-activemq5"> |
| <div class="container"> |
| <h1>Supporting IO Streams</h1> |
| </div> |
| </div> |
| <div class="container" > |
| <div class="row" style="margin-top: 30px"> |
| <div class="col-12 activemq5"> |
| <p><a href="developers">Developers</a> > <a href="developer-guide">Developer Guide</a> > <a href="design-documents">Design Documents</a> > <a href="Design Documents/supporting-io-streams">Supporting IO Streams</a></p> |
| |
| <p>It’d be great to offer kick ass support for streaming files over ActiveMQ of any arbitrary size. The basic idea is to fragment the stream into multiple messages and send/receive those over JMS.</p> |
| |
| <p>There are a few issues to consider…</p> |
| |
| <h3 id="use-casess">Use casess</h3> |
| |
| <ol> |
| <li>many producers writing and only 1 consumer. Each consumer ideally should completely process 1 stream at once (IO streams are often blocking IO, so the client thread can’t do much with other streams while its reading 1).</li> |
| <li>if a consumer gets 1Gb through a 10Gb file and dies, we need to re-deliver the messages to another consumer.</li> |
| </ol> |
| |
| <h3 id="goal">Goal</h3> |
| |
| <p>Our goal should be</p> |
| |
| <ul> |
| <li>for each consumer to process a single stream in one go before trying to process another.</li> |
| </ul> |
| |
| <h3 id="possible-failure-conditions">Possible Failure Conditions</h3> |
| |
| <ul> |
| <li>Consumer could die in the middle of reading a stream. Recovery options: |
| <ol> |
| <li>Trash the rest of the stream</li> |
| <li>Restart delivery of the stream to the next consumer</li> |
| <li>Continue delivery of the steam to the next consumer at the point of failure.</li> |
| </ol> |
| </li> |
| <li>Producer could die in the middle of writing a stream. We may need to detect his failure. We could |
| <ol> |
| <li>Send the stream in a transaction. The stream is not sent to a consumer until it’s fully received by the broker. Down side: consumer has high latency before being able to receive the message.</li> |
| <li>Consumer timeout: if a message is not receive soon enough the consumer assumes the producer is dead. (What if he’s not??)</li> |
| </ol> |
| </li> |
| <li>Consumer could start to receive in the middle of a stream. Condition could happen: |
| <ol> |
| <li>if another consumer assumed the broker was dead (but it was not).</li> |
| <li>if a non stream consumer acidentally removed messages, or messages were sent to the DLQ due to consumer rollbacks.</li> |
| </ol> |
| </li> |
| </ul> |
| |
| <h3 id="implementation-issues">Implementation Issues</h3> |
| |
| <ul> |
| <li>we can use message groups to ensure the same consumer processes all messages for a given stream - but unfortunately message groups does not prevent a consumer being overloaded with more than one message group at a time - maybe thats a new feature we can add?</li> |
| <li>avoid the broker running out of RAM - so spool to disk (or throttle the producer) if the only consumer is working on a different stream. |
| <ul> |
| <li>If messages sent using transactions, major chanages need may need to be made to how we do transaction management and message journaling. Currently, all messages in a in progress transaction are held in memory until it commits (Synchronization callbacks are holding on the the messages). The journal currently holds on to all transacted messages in it’s journal + memory until commit, it does now allow the journal to rollover messages that are part of transaction that is in progess.</li> |
| </ul> |
| </li> |
| <li>given that the entire stream must be processed in one go, we can only ACK the entire stream of messages - so we need to disable pre-fetch? |
| <ul> |
| <li>Prefetch disabling is not needed as the consumers send back a special kind of ack which only temporarily expands the prefetch window.</li> |
| </ul> |
| </li> |
| </ul> |
| |
| |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="row sitemap"> |
| <div class="col-sm-12"> |
| <div class="container"> |
| <div class="row"> |
| <div class="col-sm-12"> |
| <div class="row"> |
| <div class="col-sm-3"> |
| <div > |
| <img class="float-left" style="max-height: 100px" src="/assets/img/activemq_logo_white_vertical_small.png"/> |
| </div> |
| </div> |
| <div style="text-align: center; margin-bottom: 0px; margin-top: 30px; font-size: 65%" class="col-sm-6"> |
| <p>Apache ActiveMQ, ActiveMQ, ActiveMQ Artemis, Apache, the Apache feather logo, and the Apache ActiveMQ project logo are trademarks of The Apache Software Foundation. Copyright © 2019, The Apache Software Foundation. Licensed under <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License 2.0</a>.</p> |
| </div> |
| <div class="col-sm-3"> |
| <div > |
| <a href="https://www.apache.org"><img class="float-right" style="margin-top: 10px; max-height: 80px" src="/assets/img/apache-logo-small.png"/></a> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| |
| </body> |
| </html> |
