Myujth

I've been running a flask application with a celery worker and redis in three separated docker containers without any issue.

This is how I start it:

celery worker -A app.controller.engine.celery -l info --concurrency=2 --pool eventlet

Celery starts fine:

 -------------- celery@a828bd5b0089 v4.2.1 (windowlicker)
---- **** -----
--- * *** * -- Linux-4.9.93-linuxkit-aufs-x86_64-with 2018-11-15 16:06:59
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: app.controller.engine:0x7f8ba4eb70b8
- ** ---------- .> transport: redis://redis:6379/0
- ** ---------- .> results: redis://redis:6379/1
- *** --- * --- .> concurrency: 2 (eventlet)
-- ******* ---- .> task events: ON
--- ***** -----
 -------------- [queues]
 .> celery exchange=celery(direct) key=celery


[tasks]
 . app.controller.engine.do

INFO:engineio:Server initialized for eventlet.
INFO:engineio:Server initialized for threading.
[2018-11-15 15:44:34,301: INFO/MainProcess] Connected to redis://redis:6379/0
[2018-11-15 15:44:34,321: INFO/MainProcess] mingle: searching for neighbors
[2018-11-15 15:44:35,358: INFO/MainProcess] mingle: all alone
[2018-11-15 15:44:35,396: INFO/MainProcess] pidbox: Connected to redis://redis:6379/0.
[2018-11-15 15:44:35,415: INFO/MainProcess] celery@12af03844cd0 ready.

But when starting a task , it suddenly crashes:

[2018-11-15 15:02:03,224: CRITICAL/MainProcess] Unrecoverable error: AttributeError("'float' object has no attribute 'items'",)
Traceback (most recent call last):
 File "/app/env/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start
 self.blueprint.start(self)
 File "/app/env/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
 step.start(parent)
 File "/app/env/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
 return self.obj.start()
 File "/app/env/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 317, in start
 blueprint.start(self)
 File "/app/env/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
 step.start(parent)
 File "/app/env/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 593, in start
 c.loop(*c.loop_args())
 File "/app/env/lib/python3.6/site-packages/celery/worker/loops.py", line 121, in synloop
 connection.drain_events(timeout=2.0)
 File "/app/env/lib/python3.6/site-packages/kombu/connection.py", line 301, in drain_events
 return self.transport.drain_events(self.connection, **kwargs)
 File "/app/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 963, in drain_events
 get(self._deliver, timeout=timeout)
 File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 359, in get
 ret = self.handle_event(fileno, event)
 File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 341, in handle_event
 return self.on_readable(fileno), self
 File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 337, in on_readable
 chan.handlers[type]()
 File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 724, in _brpop_read
 self.connection._deliver(loads(bytes_to_str(item)), dest)
 File "/app/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 983, in _deliver
 callback(message)
 File "/app/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 632, in _callback
 self.qos.append(message, message.delivery_tag)
 File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 149, in append
 pipe.zadd(self.unacked_index_key, time(), delivery_tag) 
 File "/app/env/lib/python3.6/site-packages/redis/client.py", line 2263, in zadd
 for pair in iteritems(mapping):
 File "/app/env/lib/python3.6/site-packages/redis/_compat.py", line 123, in iteritems
 return iter(x.items())
AttributeError: 'float' object has no attribute 'items'

I don't see any reference to my code in the stacktrace.

This is the task method:

@celery.task()
def do(module_name, json_input):
 logger.info('____ Running _________________________')
 logger.info('testing ***********************')
 #modules.run(module_name, json_input)

I commented the call to my actual code just to verify I didn't mess it up with my modules code.. but it's crashing anyways.

Anybody has an idea what could be wrong or how to debug it properly?

Thanks a lot

The problem was that the my celery container downloaded a new release of the redis package with pip at build time, eventually with a bug or a serious change that crashes when celery tries to use it when connecting to redis. This new redis library is the 3.0.0 and it was released today.

To fix the issue you have to specify the previous version in the requirements.txt (which is 2.10.6).

And actually, specifying package versions in the requirements.txt is a good practice that prevents this type of problems.

I ran into this same issue today. hopefully this gets resolved shortly in an update by Celery. I reverted back to a previous version as well.

Not enough rep yet for a comment:

This issue is tracked by the celery folks in:
https://github.com/celery/celery/issues/5175

Leading to this PR:
https://github.com/celery/celery/pull/5176

Which has been merged and reverted. Because some other fix landed. I'm not sure if this fixed it. It is no longer relevant for me.

For anyone who use the docker-airflow, I used the folk of puckel/docker-airflow
Just need to modify the dockerfile by adding

 && pip install 'redis>=2.10.5,<3'

replace celery[redis] with only celery, by adding celery in apache-airflow built-in module i.e.,

 && pip install apache-airflow[crypto,celery,postgres,hive,jdbc,mysql]==$AIRFLOW_VERSION 

Update:
They also change the docker-airflow dockerfile recently to fix this bug, see here