Download Large File using Spring REST API

Introduction

Here in this tutorial I am going to show you how to download large file using Spring REST API. Generally when you download a file, your file is stored into file system or you load it into memory as a byte array. This is not a problem when you deal with a small file but when you download a large file (MB or GB) then it may lead to out of memory error.

I will show you here how to download the large file using RestTemplate as well as WebClient from Spring 5 onward. RestTemplate is synchronous HTTP client, so it’s a blocking HTTP client. WebClient is asynchronous and non-blocking HTTP client.

By comparing to the RestTemplate, the WebClient is:

• non-blocking, reactive, and higher concurrency with less hardware resources
• provides a functional API that takes advantage of Java 8 lambdas
• supports both synchronous and asynchronous scenarios
• supports streaming up or down from a server

Prerequisites

Java at least 8, Spring Boot Web/WebFlux 2.5.2

Download Large File

Now let’s see how I can download large file. I have two URLs from where I can download 100MB and 10GB files. 100MB file can be downloaded from https://speed.hetzner.de/100MB.bin and 10GB file can be downloaded from https://speed.hetzner.de/10GB.bin.

Using WebClient

WebClient is non-blocking client. The file will be downloaded from the URL which is passed to the input.getDownloadUrl(). I am getting the user input of the URL from where the file will be downloaded and the location where the file will be saved after download.

@PostMapping("/download/large/file")
public ResponseEntity<String> downloadFile(@RequestBody Input input) throws IOException {
	String filename = input.getDownloadUrl().substring(input.getDownloadUrl().lastIndexOf("/") + 1);
	Path path = Paths
			.get(input.getDownloadPath() == null ? "/" + filename : input.getDownloadPath() + "/" + filename);

	WebClient webClient = WebClient.builder().baseUrl(input.getDownloadUrl()).build();

	// Get file data
	Flux<DataBuffer> dataBufferFlux = webClient.get().accept(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL)
			.retrieve().bodyToFlux(DataBuffer.class);

	// Streams the dataBufferFlux from response instead of loading it all in memory
	DataBufferUtils.write(dataBufferFlux, path, StandardOpenOption.CREATE).block();

	return new ResponseEntity<String>("File Downloaded Successfully", HttpStatus.OK);
}@PostMapping("/download/large/file")
public ResponseEntity<String> downloadFile(@RequestBody Input input) throws IOException {
	String filename = input.getDownloadUrl().substring(input.getDownloadUrl().lastIndexOf("/") + 1);
	Path path = Paths
			.get(input.getDownloadPath() == null ? "/" + filename : input.getDownloadPath() + "/" + filename);

	WebClient webClient = WebClient.builder().baseUrl(input.getDownloadUrl()).build();

	// Get file data
	Flux<DataBuffer> dataBufferFlux = webClient.get().accept(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL)
			.retrieve().bodyToFlux(DataBuffer.class);

	// Streams the stream from response instead of loading it all in memory
	DataBufferUtils.write(dataBufferFlux, path, StandardOpenOption.CREATE).block();

	return new ResponseEntity<String>("File Downloaded Successfully", HttpStatus.OK);
}

Using RestTemplate

The getForObject() and getForEntity() methods of RestTemplate load the entire response in memory. This is not suitable for downloading large files since it can cause out of memory exceptions. This example shows how to stream the response of a GET request.

The file will be downloaded from the URL which is passed to the input.getDownloadUrl(). I am getting the user input of the URL from where the file will be downloaded and the location where the file will be saved after download.

@PostMapping("/download/large/file")
public ResponseEntity<String> downloadFile(@RequestBody Input input) {
	// Optional Accept header
	RequestCallback requestCallback = request -> request.getHeaders()
			.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));

	// Streams the response instead of loading it all in memory
	ResponseExtractor<Void> responseExtractor = response -> {
		String filename = input.getDownloadUrl().substring(input.getDownloadUrl().lastIndexOf("/") + 1);
		System.out.println("filename: " + filename);

		Path path = Paths
				.get(input.getDownloadPath() == null ? "/" + filename : input.getDownloadPath() + "/" + filename);

		Files.copy(response.getBody(), path);

		return null;
	};

	// URL - https://speed.hetzner.de/100MB.bin, https://speed.hetzner.de/10GB.bin
	restTemplate.execute(URI.create(input.getDownloadUrl()), HttpMethod.GET, requestCallback, responseExtractor);

	return new ResponseEntity<String>("File Downloaded Successfully", HttpStatus.OK);
}

Note that simply returning the InputStream from the extractor, because by the time the execute method returns, the underlying connection and stream are already closed.

Another point is RestTemplate is going to be deprecated and removed from future version of Spring.

The response from the REST API will be sent once your file gets downloaded. If you want to make the response asynchronous from your REST API then you can use Async or DefferedResult.

The whole source code can be downloaded later from the section Source Code.

Testing the Application

I am using Postman tool to test the file download application. make sure your Spring Boot application is up and running.

Look at the download URL and download path in the above image. So your file download URL could be anything from where you want to download the file. Download path is the path where you want to save your downloaded file.

If you continuously refresh the download path location then you will see that the file size increases periodically instead of downloading all the bytes at once.

Source Code

Download