Gunicorn flask application throw mysql database connection timed out

Gunicorn flask application throw mysql database connection timed out


0

I have a python flask application, where we upload a csv file and creating a graphql endpoint based on csv file header names. in order to retrieve the data, we are sending post request to graphql endpoint with a graphql query to retrieve the data. My python flask application is running with gunicorn server along with 4 works. and there is part where we have to retrieve the file header names from the database in order to create the graphql endpoint.

my database is a rds mysql instance and it can handle 146 max connections.
in the application, i am using Flask-SQLAlchemy to connect to my rds instance. so connection script looks like this.

from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, scoped_session
from sqlalchemy.pool import QueuePool
import os

host = '#####'
username = '#####'
password = '#####'
database_name = '#####'

engine = None
session = None

engine_params = {
    'poolclass': QueuePool,
    'pool_size': 5,
    'pool_pre_ping': True,
    'echo': False,
    'pool_recycle': 3600,
    'connect_args': {
        'connect_timeout': 20
        }
}

def get_session():
    global engine, session
    engine_params = {
        'poolclass': QueuePool,
        'pool_size': 5,
        'pool_pre_ping': True,
        'echo': False,
        'pool_recycle': 3600,
        'connect_args': {
            'connect_timeout': 20
            }
    }
    if not engine:
        engine = create_engine('mysql+pymysql://' + username +
                       ':' + password + '@' + host + '/' + database_name , **engine_params)
        session = scoped_session(sessionmaker(autocommit=False,
                                                autoflush=False,
                                                bind=engine))
    return session , engine

def create_connection():
    global engine
    if engine:
        return engine
    engine = create_engine('mysql+pymysql://' + username + ':' + password + '@' + host + '/' + database_name , **engine_params)
    return engine

def create_session(connection):
    global session
    if session:
        return session
    session = sessionmaker(bind=connection)()
    return session
    
def close_session(session):
    session.close()

in order to get the header details from the database, i am using this query class

from connection import *

class CsvSourceSpAdapter:
    session, connection = get_session()
 
 def get_by_id(self, app_id, file_id):
        result = self.session.query(CsvFile).filter(and_(CsvFile.file_id == file_id, CsvFile.app_id == app_id)).first()
        if result:
            output_result = result.to_dict()
            logger.info("graphql stuff")
            logger.info(output_result)
            return {
                'status': True,
                'message': 'Found',
                'data': output_result
            }
        else:
            logger.info("CSV file Not Found*******************************")
            return {
                'status': False,
                'message': 'CSV file not found'
            }

we have deploy this to Kubernetes pods and my docker file looks like this

FROM Test/test-python-3.9

ARG RUNTIME_USER=appuser
USER ${RUNTIME_USER}
ARG HOME=/home/${RUNTIME_USER}
RUN mkdir -p ${HOME}/app/src

ENV PYTHONPATH "${HOME}/app:${HOME}/.local/bin"
ENV PATH "${PATH}:${HOME}/.local/bin"

COPY --chown=${RUNTIME_USER}:${RUNTIME_USER} ./requirements.txt .

RUN python3 -m pip install -r requirements.txt

WORKDIR ${HOME}/app/src

COPY --chown=${RUNTIME_USER}:${RUNTIME_USER} . .

ENTRYPOINT ["gunicorn"]

CMD ["-w", "4", "--bind", "0.0.0.0:8080", "__main__:app"]

after deploy the this application to pod we have sent a post request to graphql endpoint and it worked without any issue. but after one day when i try to send the same request again I’m getting following errors

First error:
Exception on /graphql/csv_source_service/v1/apps/8aa1eed7-9f86-4864-b163-a0b4c1eed8b3/csv_files/MjA2LXJlY29yZHMtZmlsZS0wMS5jc3Y= [POST]
Traceback (most recent call last):
  File "/home/appuser/.local/lib/python3.9/site-packages/pymysql/connections.py", line 756, in _write_bytes
    self._sock.sendall(data)
TimeoutError: [Errno 110] Connection timed out

second error:
Exception on /graphql/csv_source_service/v1/apps/8aa1eed7-9f86-4864-b163-a0b4c1eed8b3/csv_files/MjA2LXJlY29yZHMtZmlsZS0wMS5jc3Y= [POST]
Traceback (most recent call last):
  File "/home/appuser/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1202, in _execute_context
    conn = self._revalidate_connection()
  File "/home/appuser/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 469, in _revalidate_connection
    raise exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Can't reconnect until invalid transaction is rolled back

but when we ran the same application with development server(werkzung server) i’m not getting this issue. so how can I fix this issue. any help would be grateful.

1

  • Not an expert on sqlalchemy, but your usage of an eternal Session looks dangerous to me and could be related to the "transaction" mentioned in the Exception.

    – Matthias Huschle

    58 mins ago


Load 6 more related questions


Show fewer related questions

0



Leave a Reply

Your email address will not be published. Required fields are marked *