AWS Lambda with Pandas and NumPy

AWS Lambda does not include Pandas/NumPy Python libraries by default. How use Pandas and NumPy with Lambda functions?

Problem statement

There are no default Pandas in AWS Lambda. You can see AWS Lambda execution environment and available libraries here. Let’s verify and create AWS Lambda Python 3.6 function with code below:

Lambda’s response:

Unable to import module 'lambda_function': No module named 'pandas'

Create new local directory with lambda_function.py file. Install Pandas to local directory with pip:

$ pip install -t . pandas

Remove *.dist-info and __pycache__. Prepare zip.zip archive with lambda_function.py file and Pandas:

$ rm -r *.dist-info __pycache__
$ zip -r zip.zip .

zip.zip archive with lambda_function.py file and Pandas

Create new lambda function (e.g. medium). Go to Function code section and select Upload a .zip file from Code entry type dropdown. Click Upload button. Upload zip.zip file. Finally click Save button:

Upload function package

Let’s check new Lambda function response:

Unable to import module 'lambda_function': Missing required dependencies ['numpy']

What? As you can see below, zip.zip archive includes NumPy.

zip.zip archive with lambda_function.py file, Pandas, and Numpy

Standard method does not work. AWS Lambda need special Pandas/NumPy. Let’s fix it.

Note: Do not forget clean your working environment, first. Remove pandas, numpy, and *.dist-info directories:

$ rm -r pandas numpy *.dist-info

Solution

AWS Lambda use Amazon Linux operating system. Idea is download Pandas and NumPy compatible with Amazon Linux.

Pandas. Navigate to https://pypi.org/project/pandas/#files. Search for and download newest *manylinux1_x86_64.whl package. In my case for Python 3.6 is pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl file.

NumPy. Do the same for NumPy. File is numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl.

Download whl files to directory with lambda_function.py. Unzip whl files.

$ unzip numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl
$ unzip pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl

Note: Lambda with Python 3.7 requires pytz lib: $ pip install -t . pytz

Remove whl files, *.dist-info, and __pycache__. Prepare new zip.zip archive:

$ rm -r *.whl *.dist-info __pycache__
$ zip -r zip.zip .

zip.zip archive for Amazon Linux with lambda_function.py file, Pandas, and Numpy

Navigate to lambda function (e.g. medium). Go to Function code section and select Upload a .zip file from Code entry type dropdown. Click Upload button. Upload zip.zip file. Finally click Save button:

Upload function package

Let’s check new Lambda function response:

Execution result: succeeded

Congrats!