Blocking function call with asynchronous content

Question

I am sure that this is a common design pattern, but I seem to have a blind-spot.

I have a requirement that the function call from Application to Service be blocking, but service needs to do something asynchronous.

Well, I can handle that, BUT, there can be multiple applications and there is only one service.

While the service should block the application, it should also be able to handle multiple parallel requests from different applications.

I suppose I need to spawn a new thread for each application’s request, but how then can I return from the thread to application at the end?

I don't mind an assembler insert to store the application’s return address & pop it later, but is that really necessary?

Do I need to do anything special in C++ to mark the service function as re-entrant?

Can someone please lift my blindfold?

---

[Update] There is no correlation between the applications. Also, only the service would be allowed to spawn a thread, not the applications. So, I am thinking of a service main spawning a bunch of service threads, but I don't see how I can can handle the function call blocking & return.

Answer 1

You have to use a blocking construct on a per-request basis.

This is called Futures (programming)

Each request will have its own Future. Once the request processing is started (possibly in a separate pool of threads, as you have described), the caller will be blocked on the Future's fulfillment, or failure.

When the result arrives, the Future is unblocked, and the call will return to the application.

When using this pattern, one must take great care prevent deadlocks and zombies. In other words, the Future needs to be unblocked at some time, whether the result is successful or failure (and in case of failure, throwing an exception back to the application is a fair game) - the call must not just hang there indefinitely.

As to thread-pool design:

• There will be at least one thread per simultaneous requester (i.e. application), which will be blocked during the request.
  • If the applications and the service are inside the same process, these threads are the same as the application threads, which will become blocked.
• There will be a separate thread pool, requiring as many threads as is necessary to do the work between the service and the internet. These threads are in addition to the request-accepting threads.
  • The exact number depends on how the work is done. It is possible to use asynchronous pattern (reactor) here, which might reduce the number of threads needed, but that will not have any effect on the number of request-accepting threads.

---

If the request from Application to Service occurs over network, there is a decoupling between Application threads and Service threads (in other words, a blocking of an Application request does not involve a "thread" at all; just a non-response from a network connection.) In that case the Service can also use reactor pattern, which means you can further reduce the number of threads.

Answer 2

Since the application is blocked while it is in the service call, every application already must have its own thread if there are to be multiple service calls at the same time.

Assuming the service call simply starts the async operation, maybe does something else, and then blocks until the operation finishes, I don't see the problem. Since every call to service is on its own thread, you can simply block that thread.

Answer 3

The short answer to your question is that you don't return from the asynchronous code. Instead you pass a context object to your asynchronous code to send a response back when you finish processing the request. This is done mostly in two forms:

You are using some server framework that already takes care of the async for you:

void init() {
    registerHandler("requestType", [](ReqType req, Response resp) {
        // ..
        resp.send(data);
    });
}

Or if you handle client requests in a blocking event loop you can achieve asynchronous by using multi-thread, like this:

while (true) {
    // ...
    Req req = recv();
    thread t(processReq, req, ctx);
}

void processReq(Req req, Context &ctx) {
    // ...
    ctx.send(response);
}

If you need to perform multiple async operations to serve one request then you use future to chain together the async operations, provided that your async operations return a future/promise type. This is not the same as waiting the async operation to return:

void processReq(Req req, Context &ctx) {
    // ...
    op1().then(op2).then(op3);
}

One last word, if you meant waiting for the async code to finish before exiting the process, then use join method of your thread objects, or use the wait method of your promise objects.