How openstack service like nova-api consume oslo.messaging. Walkthrough to understand how openstack services consume oslo.messaging and the rabbitmq driver and to understand the root cause of the AMQP heartbeat/eventlet issue under apache MPM prefork
The goal of this post is to centralize pieces of my analyze and of the original puzzle, related to an openstack issue with the AMQP heartbeat on nova-api under apache mod_wsgi.
I this post I want to show you:
preforkmodule in use with eventlet is an issue
These info can be useful for other persons so I want to share these with you.
My walkthrough (cf. section bellow) had help me to understand how things works together, so I think it can be useful to also add it there even if it’s a little bit out of the scope of the fix and the real issue.
I’m not a nova expert and maybe I’m wrong on some points, so, if you see errors in this walkthrough do not hesitate to fix it and to submit a pull request.
For developers and engineers who wants to understand why apache with eventlet can be an issue in some situations.
You don’t need to be developer on openstack, if you want to use eventlet through a HTTP restful API this article can be useful for you to help you to understand the dependencies in your stack and the execution model behind your choices.
We facing an issue with the rabbitmq driver heartbeat
under apache MPM
mod_wsgi when nova-api [monkey
patch the stdlib by using eventlet]((https://eventlet.net/doc/patching.html).
nova-api calling eventlet.
monkey_patch() when it runs under mod_wsgi.
This impacts the AMQP heartbeat thread, which is meant to be a native thread. Instead of checking AMQP sockets every 15s, It is now suspended and resumed by eventlet.
However, resuming greenthreads can take a very long time if mod_wsgi isn’t processing traffic regularly, which can cause rabbitmq to close the AMQP connection.
Here is a list of related issues:
The RabbitMQ heartbeat was introduced few years ago to keep connections from various components into RabbitMQ alive. In some situations, by example by placing a stateful firewall between the connection could result in idle connection being terminated without either endpoint being aware.
The oslo.messaging RabbitMQ driver and especially the heartbeat suffer to inherit the execution model of the service which consume him.
In this scenario nova-api need green threads to manage cells and edge features so nova-api monkey patch the stdlib to obtain async features, and the oslo.messaging rabbitmq driver endure these changes.
On openstack The default apache MPM module in use is the
I think the main issue here is that nova-api want async and use eventlet green
threads to obtain it, because, eventlet is based on epoll
or libevent, in an environment based on
prefork module who doesn’t support epoll and recent kernel
prefork apache module is appropriate for sites that
need to avoid threading for compatibility with non-thread-safe
So we suspect that the apache MPM engine (
prefork) in use here is also a part
of the problem.
We have 2 possible solutions to fix the issue:
To avoid similar issue we want to allow user to isolate the heartbeat
execution model from the parent process inherited execution model by passing the
heartbeat_in_pthread option through the driver config.
While we use MPM
prefork we want to avoid to use libevent and epoll.
heartbeat_in_pthread option is given we want to force to use the
python stdlib threading module to run the
rabbitmq heartbeat to avoid issue related to a non “standard”
environment. I mean “standard” because async features isn’t the default
config in mostly case, starting by apache which define
prefork is the
This is an experimental feature, we can help us to ensure to run heartbeat through a classical python thread not build over epoll and libevent to avoid issue with environment who doesn’t support these kernel features.
We will try to switch  from the apache
prefork module to the
module which support non blocking sockets, use modern kernel features
like epoll through APR.
These changes are not fully tested yet with the oslo.messaging rabbitmq heartbeat. I need to realize some tests with a non patched version of oslo.messaging to observe if it work better with eventlet under apache and mod_wsgi.
Possibly if these changes work as expected the oslo.messaging fix (cf. previous section) will become obsolete and we could restore the original behaviour. An inhereted parent execution model would not be an issue for oslo.messaging.
I will update this section when I’ll more info to share with you about this.
To see how things works in openstack we will do a walkthrough to inspect how the nova-api consume and use the oslo.messaging rabbitmq driver.
We will start by analyze how openstack run the nova-api, the different approaches to launch the service (puppet, ansible), in a second time we will observe how the WSGI application for the nova-api work, and to finish we will see how nova-api established the connections with the rabbitmq driver and how the health check heartbeat will be launched.
I’ve initiate this walkthrough to better understand how openstack service works and to try to fix an oslo.messaging issue related to the rabbitmq driver when we run it through nova in an eventlet monkey patched environment under apache mod_wsgi (cf. the previous described issue).
The puppet team provide a puppet project dedicated to nova who define how to run nova under an apache environment. Also it’s important to note that other deployments systems exists and are in use on openstack, like ansible through the openstack ansible project who provide a different approach to deploy nova. There are many others deployment systems in use too (Chef, Salt, etc.).
This puppet project will configure apache to run nova
It is the
script_path is used for the
wsgi_script_dir in configuring the
In the end its all used by https://github.com/openstack/puppet-openstacklib/blob/master/manifests/wsgi/apache.pp to configure the wsgi settings of the vhost
We can observe that this puppet script define the following configuration
api_wsgi_script_source => '/usr/bin/nova-api-wsgi.
Nova will be launched by calling the wsgi script available at
The nova-api will be runned under the apache MPM
prefork module, like described previously.
This entry point will when it’s called while trigger
who correspond to the initialization of the nova wsgi application.
To implement the Python Web Server Gateway Interface (WSGI)(PEP 3333) nova
paste.deploy python module.
Paste deploy was at the origin a submodule of the paste module.
Basically at some point in the past people realized that
one of the main useful parts of paste and they didn’t want all the rest of
paste, so it was extracted.
As paste and paste.deploy both got “old” maintenance sort of diverged. They are still maintained, but as separate packages.
Even if the Paste module seems not used here
I will describe some
paste specific behaviours and especially
concerning requests handling, threads, and the python interpreter life cycle,
which I guess we need to take care to really undestand the eventlet issue and green threads for the heartbeat.
Indeed, to avoid issue with requests management (freeze, memory usage, etc…),
paste manage threads with 3 states:
paste thread initiate the oslo.messaging AMQP heartbeat who will run in
an async way by using green thread maybe in this case the parent thread consider that
it can become idle for some reasons and the main reason is that this thread
(heartbeat) is not a blocking thread.
In parallel, uwsgi (not used here but the heartbeat seems occur within too)
paste facing some issues in multiple process/workers mode
Paste Deployment is a system for finding and configuring WSGI applications and servers. For WSGI application consumers it provides a single, simple function (loadapp) for loading a WSGI application from a configuration file or a Python Egg. For WSGI application providers it only asks for a single, simple entry point to your application, so that application users don’t need to be exposed to the implementation details of your application.
The nova service call the
loadapp method by passing available configurations.
The configuration seems to be designed by following the
nova configuration guide.
A sample configuration example for nova is available online (cf. the wsgi part).
Paste deploy don’t seem to define specific behaviours related to threads management. It only seem to help to define application parts and related url/uri, database url, etc…
Well, now we will continue to follow the nova code.
During the init of the nova-api application, nova will try to setup some services.
These services are setup by using the
Services to setup are defined in a service manager.
Nova manage services by using a dedicated service module.
To continue this inspection of nova we will now become focused on the
This class will instanciate a
This object (
router method who will return
a rpc client.
The instantiated oslo.messaging
The transport layer is defined by nova and it will be retrieved by using the oslo.messaging mechanismes based on the config and the url and the oslo.messaging defined drivers by using stevedore.
In our case we will instantiate a
Also on the oslo.messaging we need to take care about the connection class execution model. Even if rabbit has only one Connection class, this connection can be used for two purposes:
On an over hand the connection pool seems to define the
This listener seems to be called when an:
Idle connection has expired and been closed
The Idle connection here seem to be the connection with the rabbitmq server (by example) who have expired.
Then the “thread safe”
Modelled after the eventlet.pools.Pool interface, but designed to be safe
when using native threads without the GIL, defined the
expire method who
will clean the pool from the expired connections based on a ttl.
I think we need to take care about the previous mechanism due to the fact that the connection and the heartbeat have been invoqued from the connection pool mechanism (cf. previous lines about how the connection pool module start the health check)
There the used executor is an
eventlet executor, so the initiated object will use eventlet and so the heartbeat thread will use a green thread.
The oslo.messaging instantiate RPC server (https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/rpc/server.py#L190)
Some POC are available at: https://github.com/4383/pyamqp-heartbeat/blob/master/POC.md
You can test behaviors between execution models and different versions of the oslo.messaging code base (patched/non-patched).
These POC can be useful to compare the difference between using a native python
thread or an eventlet green thread under apache MPM
These changes don’t reflect the latest changes and especially the
heartbeat_in_pthread option and the possibility to turn on/off the
feature. In other words on these POCs we always force to use pthreads
and these POCs allow you to compare the rabbitmq heartbeat connection
with pthread and green thread.
Congretulation! You are now at the end of this article, you’ve read the most biggest part of this post.
Execution environment can have impactes on services, applications, and libraries, like the nova-api/oslo.messaging use case.
It’s not a trivial thing to debug, I hope my article can help some of you.
Openstack is a little bit complicated to debug, to run a service we need to instantiate many environments and apps like apache, mod_wsgi, etc…
With stack like this it can be difficult to determine which part introduce the issue, and why the error occur.
I hope you appreciated read this article, don’t hesitate to fix errors if you see one. Don’t hesitate to contact me for further discussions and questions.
You can find further resources and useful links bellow.