Hervé Beraud

FOSS Hacker at Red Hat
Python Senior Software Engineer
Science Lover

blog-image

Apache MPM prefork - AMQP heartbeat and eventlet issue

Autored by Hervé Beraud on 15 July 2019

How openstack service like nova-api consume oslo.messaging. Walkthrough to understand how openstack services consume oslo.messaging and the rabbitmq driver and to understand the root cause of the AMQP heartbeat/eventlet issue under apache MPM prefork

Introduction

The goal of this post is to centralize pieces of my analyze and of the original puzzle, related to an openstack issue with the AMQP heartbeat on nova-api under apache mod_wsgi.

I this post I want to show you:

  • a brief description of the original issue and possibly why we facing this issue
  • an analyze of the root cause and why the default apache MPM prefork module in use with eventlet is an issue
  • how openstack services consume oslo.messaging and especially how they use the rabbitmq driver.
  • some POC to test different behaviours

These info can be useful for other persons so I want to share these with you.

My walkthrough (cf. section bellow) had help me to understand how things works together, so I think it can be useful to also add it there even if it’s a little bit out of the scope of the fix and the real issue.

I’m not a nova expert and maybe I’m wrong on some points, so, if you see errors in this walkthrough do not hesitate to fix it and to submit a pull request.

Who this article is for?

For developers and engineers who wants to understand why apache with eventlet can be an issue in some situations.

You don’t need to be developer on openstack, if you want to use eventlet through a HTTP restful API this article can be useful for you to help you to understand the dependencies in your stack and the execution model behind your choices.

The original issue

We facing an issue with the rabbitmq driver heartbeat under apache MPM prefork module and with mod_wsgi when nova-api [monkey patch the stdlib by using eventlet]((https://eventlet.net/doc/patching.html).

nova-api calling eventlet.monkey_patch() when it runs under mod_wsgi.

This impacts the AMQP heartbeat thread, which is meant to be a native thread. Instead of checking AMQP sockets every 15s, It is now suspended and resumed by eventlet.

However, resuming greenthreads can take a very long time if mod_wsgi isn’t processing traffic regularly, which can cause rabbitmq to close the AMQP connection.

Here is a list of related issues:

The Origins of the RabbitMQ heartbeat

The RabbitMQ heartbeat was introduced few years ago to keep connections from various components into RabbitMQ alive. In some situations, by example by placing a stateful firewall between the connection could result in idle connection being terminated without either endpoint being aware.

Root Cause

The oslo.messaging RabbitMQ driver and especially the heartbeat suffer to inherit the execution model of the service which consume him.

In this scenario nova-api need green threads to manage cells and edge features so nova-api monkey patch the stdlib to obtain async features, and the oslo.messaging rabbitmq driver endure these changes.

On openstack The default apache MPM module in use is the prefork module.

I think the main issue here is that nova-api want async and use eventlet green threads to obtain it, because, eventlet is based on epoll or libevent, in an environment based on apache MPM prefork module who doesn’t support epoll and recent kernel features.

The MPM prefork apache module is appropriate for sites that need to avoid threading for compatibility with non-thread-safe libraries.

So we suspect that the apache MPM engine (prefork) in use here is also a part of the problem.

solutions

We have 2 possible solutions to fix the issue:

  • fixing the oslo.messaging RabbitMQ driver by using native python thread for the heartbeat
  • change the apache MPM module in use from prefork to event

The oslo.messaging fix

To avoid similar issue we want to allow user to isolate the heartbeat execution model from the parent process inherited execution model by passing the heartbeat_in_pthread option through the driver config.

While we use MPM prefork we want to avoid to use libevent and epoll.

If the heartbeat_in_pthread option is given we want to force to use the python stdlib threading module to run the rabbitmq heartbeat to avoid issue related to a non “standard” environment. I mean “standard” because async features isn’t the default config in mostly case, starting by apache which define prefork is the default engine.

This is an experimental feature, we can help us to ensure to run heartbeat through a classical python thread not build over epoll and libevent to avoid issue with environment who doesn’t support these kernel features.

Proposed fix:

  • https://review.opendev.org/#/c/663074/

The apache MPM fix

We will try to switch [4][5] from the apache prefork module to the event module which support non blocking sockets, use modern kernel features like epoll through APR.

Proposed fixes:

  • https://review.opendev.org/#/c/668862/
  • https://review.opendev.org/#/c/669178/

These changes are not fully tested yet with the oslo.messaging rabbitmq heartbeat. I need to realize some tests with a non patched version of oslo.messaging to observe if it work better with eventlet under apache and mod_wsgi.

Possibly if these changes work as expected the oslo.messaging fix (cf. previous section) will become obsolete and we could restore the original behaviour. An inhereted parent execution model would not be an issue for oslo.messaging.

I will update this section when I’ll more info to share with you about this.

Walkthrough - how nova-api use oslo.messaging

To see how things works in openstack we will do a walkthrough to inspect how the nova-api consume and use the oslo.messaging rabbitmq driver.

We will start by analyze how openstack run the nova-api, the different approaches to launch the service (puppet, ansible), in a second time we will observe how the WSGI application for the nova-api work, and to finish we will see how nova-api established the connections with the rabbitmq driver and how the health check heartbeat will be launched.

I’ve initiate this walkthrough to better understand how openstack service works and to try to fix an oslo.messaging issue related to the rabbitmq driver when we run it through nova in an eventlet monkey patched environment under apache mod_wsgi (cf. the previous described issue).

The puppet part

The puppet team provide a puppet project dedicated to nova who define how to run nova under an apache environment. Also it’s important to note that other deployments systems exists and are in use on openstack, like ansible through the openstack ansible project who provide a different approach to deploy nova. There are many others deployment systems in use too (Chef, Salt, etc.).

This puppet project will configure apache to run nova

It is the script_path is used for the wsgi_script_dir in configuring the httpd conf. In the end its all used by https://github.com/openstack/puppet-openstacklib/blob/master/manifests/wsgi/apache.pp to configure the wsgi settings of the vhost

The wsgi nova entry_point to use to start the nova wsgi application is defined here and the apache vhost file to use too

We can observe that this puppet script define the following configuration api_wsgi_script_source => '/usr/bin/nova-api-wsgi.

Nova will be launched by calling the wsgi script available at /usr/bin/nova-api-wsgi.

The nova-api will be runned under the apache MPM prefork module, like described previously.

This wsgi scripts is generated by pbr and setuptools during the nova install.

Nova define it in its own setup.cfg file.

This entry point will when it’s called while trigger nova.api.openstack.compute.wsgi:init_application who correspond to the initialization of the nova wsgi application.

To implement the Python Web Server Gateway Interface (WSGI)(PEP 3333) nova use the paste.deploy python module.

Generate the wsgi apps by using Paste Deploy

Paste deploy was at the origin a submodule of the paste module.

Basically at some point in the past people realized that paste.deploy was one of the main useful parts of paste and they didn’t want all the rest of paste, so it was extracted.

As paste and paste.deploy both got “old” maintenance sort of diverged. They are still maintained, but as separate packages.

Even if the Paste module seems not used here I will describe some paste specific behaviours and especially concerning requests handling, threads, and the python interpreter life cycle, which I guess we need to take care to really undestand the eventlet issue and green threads for the heartbeat.

Indeed, to avoid issue with requests management (freeze, memory usage, etc…), paste manage threads with 3 states:

  • idle (waiting for a request)
  • working (on a request)
  • should die (a thread that should die for some reasons)

If a paste thread initiate the oslo.messaging AMQP heartbeat who will run in an async way by using green thread maybe in this case the parent thread consider that it can become idle for some reasons and the main reason is that this thread (heartbeat) is not a blocking thread.

In parallel, uwsgi (not used here but the heartbeat seems occur within too) support for paste facing some issues in multiple process/workers mode

On the paste.deploy side nova call the loadapp method to serve the nova-api.

Paste Deployment is a system for finding and configuring WSGI applications and servers. For WSGI application consumers it provides a single, simple function (loadapp) for loading a WSGI application from a configuration file or a Python Egg. For WSGI application providers it only asks for a single, simple entry point to your application, so that application users don’t need to be exposed to the implementation details of your application.

The nova service call the loadapp method by passing available configurations. The configuration seems to be designed by following the nova configuration guide.

A sample configuration example for nova is available online (cf. the wsgi part).

Paste deploy don’t seem to define specific behaviours related to threads management. It only seem to help to define application parts and related url/uri, database url, etc…

Running the WSGI app and services at start

Well, now we will continue to follow the nova code.

During the init of the nova-api application, nova will try to setup some services. These services are setup by using the _setup_service method.

Services to setup are defined in a service manager.

Nova manage services by using a dedicated service module.

They services seems to be also retrieving by querying the database by using the previous definied service module.

To continue this inspection of nova we will now become focused on the ConsoleAuthManager module. This class will instanciate a compute_rpcapi.ComputeAPI object This object (ComputeAPI) define a router method who will return a rpc client.

The returned RPC client returned by the nova rpc module is an oslo.messaging rpc client

The instantiated oslo.messaging RPCClient

The transport layer is defined by nova and it will be retrieved by using the oslo.messaging mechanismes based on the config and the url and the oslo.messaging defined drivers by using stevedore.

In our case we will instantiate a RabbitDriver.

The used driver will initiate a connection pool by using the Connection class defined in the driver.

The oslo.messaging connection pool module will create the connection and also start the healt check mechanism by triggering the heartbeat in a dedicated thread

Also on the oslo.messaging we need to take care about the connection class execution model. Even if rabbit has only one Connection class, this connection can be used for two purposes:

  • wait and receive amqp messages (only do read stuffs on the socket)
  • send messages to the broker (only do write stuffs on the socket) The code inside a connection class is not concurrency safe. Using one Connection class instance for doing both, will result of eventlet complaining of multiple greenthreads that read/write the same fd concurrently… because ‘send’ and ‘listen’ run in different greenthread. So, a connection cannot be shared between thread/greenthread and this two variables permit to define the purpose of the connection to allow drivers to add special handling if needed (like heatbeat). amqp drivers create 3 kind of connections:
  • driver.listen*(): each call create a new ‘PURPOSE_LISTEN’ connection
  • driver.send*(): a pool of ‘PURPOSE_SEND’ connections is used
  • driver internally have another ‘PURPOSE_LISTEN’ connection dedicated to wait replies of rpc call

On an over hand the connection pool seems to define the _on_expire event listener.

This listener seems to be called when an:

Idle connection has expired and been closed

The Idle connection here seem to be the connection with the rabbitmq server (by example) who have expired. Then the “thread safe” Pool mechanism, Modelled after the eventlet.pools.Pool interface, but designed to be safe when using native threads without the GIL, defined the expire method who will clean the pool from the expired connections based on a ttl.

I think we need to take care about the previous mechanism due to the fact that the connection and the heartbeat have been invoqued from the connection pool mechanism (cf. previous lines about how the connection pool module start the health check)

The nova rpc module also define a RPC server inherited from oslo.messaging

There the used executor is an eventlet executor, so the initiated object will use eventlet and so the heartbeat thread will use a green thread.

The oslo.messaging instantiate RPC server (https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/rpc/server.py#L190)

POC

Some POC are available at: https://github.com/4383/pyamqp-heartbeat/blob/master/POC.md

You can test behaviors between execution models and different versions of the oslo.messaging code base (patched/non-patched).

These POC can be useful to compare the difference between using a native python thread or an eventlet green thread under apache MPM prefork.

These changes don’t reflect the latest changes and especially the heartbeat_in_pthread option and the possibility to turn on/off the feature. In other words on these POCs we always force to use pthreads and these POCs allow you to compare the rabbitmq heartbeat connection with pthread and green thread.

Conclusion

Congretulation! You are now at the end of this article, you’ve read the most biggest part of this post.

Execution environment can have impactes on services, applications, and libraries, like the nova-api/oslo.messaging use case.

It’s not a trivial thing to debug, I hope my article can help some of you.

Openstack is a little bit complicated to debug, to run a service we need to instantiate many environments and apps like apache, mod_wsgi, etc…

With stack like this it can be difficult to determine which part introduce the issue, and why the error occur.

I hope you appreciated read this article, don’t hesitate to fix errors if you see one. Don’t hesitate to contact me for further discussions and questions.

You can find further resources and useful links bellow.

Cheers!

References & further reading