March 15, 2023 |News

How to download an RDS log file from the AWS console or CLI

Now and then when working with large clients I face an issue with downloading big MySQL slog-log files from the AWS RDS console. In the middle of the download it will suddenly stop and say its network failure. The simplest way is to go to the AWS console, choose an RDS server, and navigate to a specific database instance from which you want to download a log file. But here is also a guide of how to tackle the issue with large RDS slow logs.

Choose “Logs and events” tab:

Scroll to the very bottom of the page and select a log file you want to download

After that you can click the “Download” button and download the log file.

The main disadvantages of this method are:

Manual work and it could be time consuming if you need to download a lot of files;
A lot of clicking;
Won’t work if you need to download a log file not on your computer but on some running server.

How to download an RDS log file using aws cli

We can tackle all issues described above once if we will start using aws cli.

In order to install aws cli on your computer please follow this link: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

After installation is completed, you can use the following command to download a log file from a specific RDS instance. For example, let’s try to download the current slow log file:

> aws rds download-db-log-file-portion --db-instance-identifier 
<your-instance> --log-file-name slowquery/mysql-slowquery.log --starting-token 0 --output text > /tmp/<your-instance>-slow-log-current

If your files are small you will be able to download it but if not then you can fall into a trap when this command will end successfully with throwing any error but it will download only the part of the file. Here is the good issue opened at github where you can find some workarounds: running custom python or bash scripts to download the whole file: https://github.com/aws/aws-cli/issues/2268

Another issue you can hit is the limit:

A client error (Throttling) occurred when calling the DownloadDBLogFilePortion operation: Rate exceeded

The workaround is to either increase the limit and in this case you will have to contact AWS support or add delay between each command call.

Alternatively if you do not want to stick to python and prefer golang then you can try script provided by https://github.com/jonstacks/aws (I will give an example in the next section)

If you need to get the whole list of files you can download you can use the following command:

> aws rds describe-db-log-files --db-instance-identifier <your-instance>
{                        	 
	"DescribeDBLogFiles": [
    	{
        	"LogFileName": "error/mysql-error.log",      	 
        	"LastWritten": 1621497600000,
        	"Size": 0    	 
    	},
....
    	{
        	"LogFileName": "error/mysql-error-running.log",   
        	"LastWritten": 1621461600000,
        	"Size": 0
    	},
....
    	{
        	"LogFileName": "general/mysql-general.log",
        	"LastWritten": 1621497615000,
        	"Size": 7704625
    	},
....
    	{
        	"LogFileName": "slowquery/mysql-slowquery.log.9",
        	"LastWritten": 1621414800000,
        	"Size": 68054277
    	}
	]
}

How to download an RDS log file via http endpoint

In order to download rds log files without hitting “throttle” issue or you just have a big log file you can scripts from the following repository: https://github.com/jonstacks/aws

We need to get from there “rds-logs-download-url”. Please follow the README.md file to install it.

You can use the following command to download the current log file:

curl -o slow.current $(./rds-logs-download-url <your-instance> slowquery/mysql-slowquery.log)

If you need to download slow log file you can run the following command:

instance_id=<your-instance>
aws rds describe-db-log-files --db-instance-identifier $instance_id --output text | awk '/slowquery/{print $3}' | cut -d/ -f2 | xargs -i -t bash -c "curl -o {} \$(./rds-logs-download-url $instance_id slowquery/{})"

In the same way you can download error and general logs.