We like to transfer information and talk to each other in the office. Much like people, computers and systems need to talk to each other too. However, like people, computers have to wait their turn to communicate when things get busy. Enter message queues!
As you may have noticed, we gather information from many sources to build your reports. A single PDF report with 3 parcels might require making 25 “requests” for information and map images. An AQUAOSO portfolio for 100 parcels may generate 20,000 cross tabulated data points. What happens if some of the some of the information fails to be fetched?
A “message queue” plays an important role in handling a failed fetch: to ensure that an operation happens exactly once. If it fails partway through, it will restart again later. This ensures a higher percentage of completed reports!
PDF reports have maps that are pulled from numerous public and private sources, and often these third parties fail to serve up the map layers. This isn’t a big deal for the AQUAOSO research platform because we cache all the successful requests, and if one fails, we pause the report then try again later. The message queue gets a “job” for the report. A “worker” will checkout the job, then if it times out or fails, the job will go back in the queue until another worker can grab it. When third party sources are under a heavy load, a single reports may five attempts before it is successful. The message queue is a critical component in making sure that it gets retried until it succeeds, and that only a single worker will be crunching a report at once.
