Recovering From a Full Linux Disk on a Rented Dedicated Server

Uploads stop, a database rejects writes, and a deployment reports “No space left on device.” A Linux disk full incident on a rented dedicated server calls for one immediate decision: what resource is exhausted, and which recovery action can restore writes without sacrificing required data?

The rental specification cannot answer that question. Available capacity depends on the filesystem receiving the write, its inode availability, and the process using it. Deleting a large file may help, accomplish nothing, or remove information the service needs.

Use the sequence below to identify the constraint, select a controlled intervention, and verify recovery. The diagnostic commands are read-only; cleanup and service changes require decisions specific to your workload.

Contain the growth before choosing files

Record the affected service, failing path, exact error, and timestamp. Keep the incident record outside the affected filesystem. If a known export, backup staging job, or runaway application task is consuming the remaining capacity, consider pausing that specific producer through its supported control path.

Avoid stopping a database or terminating unknown processes simply because they are writing. Identify the owner and recovery requirements first. A controlled maintenance response may be appropriate when continued requests create failed transactions or unbounded retries.

Preserve the current administrative session. Before a disruptive intervention, establish a second management path using the available remote console and recovery access. A reboot should have a defined purpose and a way to diagnose a failed startup.

Identify the filesystem behind a Linux disk full error

These examples assume Linux with GNU coreutils and util-linux. Use SSH or a console with authorized administrative access. Run the checks against an existing directory where the application actually writes; /var is an example, not a universal application location.

findmnt --target /var --output TARGET,SOURCE,FSTYPE,OPTIONS
df -h /var
df -i /var

findmnt identifies the mount associated with a path. Check that it matches the expected layout. A missing application-data mount can leave writes going to the underlying root filesystem instead. Do not mount another filesystem over those files without planning how to inspect and reconcile them.

df reports filesystem capacity; -i switches the report to inode usage. Inspect available space and available inodes separately. Free capacity on another mount does not establish that the failing destination has room.

Do not treat every failed write as capacity exhaustion. Linux distinguishes errors such as ENOSPC, EDQUOT, and EIO in the write system-call documentation. Application logs may expose the underlying error more clearly than a dashboard alert.

For a containerized service, repeat the investigation in the relevant container or mount namespace. Host-level free space alone does not establish the capacity available to the application’s writable layer or mounted volume.

Locate the consumer without starting a cleanup sweep

Begin with the affected directory tree. Avoid scanning every mounted filesystem during an incident when a narrower path will answer the question.

sudo du -x -h --max-depth=1 /var

With GNU du, -x avoids descending into directories on different filesystems. The depth limit reduces the displayed detail, but the command still traverses the selected tree to calculate usage. It can create substantial metadata I/O.

Inspect the largest relevant child directory next. Do not suppress permission errors: an incomplete scan can make a large consumer disappear from the investigation. Stop a scan that materially worsens service latency and narrow its scope.

If inode availability is the constraint, inspect file counts instead of bytes:

sudo du -x --inodes --max-depth=1 /var

This helps distinguish a few large archives from a directory containing many small cache, session, or queue files. Both can interrupt writes, but their cleanup rules differ. A queue directory may contain unprocessed business work, even if each individual file looks disposable.

Check for deleted files that remain open

When a file’s last directory entry is removed, its space is not released while a process still has it open. The Linux unlink documentation describes this behavior. It explains why deleting an active log can leave filesystem usage largely unchanged.

sudo lsof +L1

The lsof manual documents +L1 as selecting open files that have been unlinked. Administrative privileges are normally needed for a useful system-wide view; process visibility restrictions can still limit results.

Match the process, file, and device to the affected filesystem. Do not add displayed sizes indiscriminately: several descriptors or processes may refer to the same file.

If an application holds a deleted log, use its documented log-reopen procedure when available. A controlled service restart may be an alternative after checking interruption, dependencies, and startup capacity. Do not send an assumed signal to an unfamiliar process or write into its file descriptors as a shortcut.

A difference between filesystem usage and visible file totals is a clue, not proof of this condition. Scan permissions, filesystem accounting, snapshots, and the selected paths can also affect the comparison.

Choose a recovery action by ownership and recoverability

Before removing anything, answer four questions: who owns it, what recreates it, which retention rules apply, and how would you recover it if the decision were wrong?

• Regenerable caches: use the application’s supported cleanup procedure. Confirm that rebuilding the cache will not immediately exhaust capacity again.
• Old exports or staging artifacts: verify that no active job needs them and that required copies exist elsewhere before removal.
• Logs: preserve incident evidence and required retention, then correct rotation or retention through the logging system.
• Database files and recovery logs: follow the database’s own retention and maintenance procedures. Do not delete files directly from its data directory to create emergency headroom.
• Container storage: use runtime-supported inspection and cleanup with an explicit understanding of volumes, images, and stopped workloads. Broad pruning can remove recovery assets.

On a system using systemd-journald, this inspection shows journal storage use:

sudo journalctl --disk-usage

The journalctl documentation distinguishes active and archived journals: vacuum operations remove archived files, while the usage report includes both. Any retention reduction must account for evidence requirements and the irreversible loss of removed logs.

If required data is the dominant consumer, expansion or migration may be safer than deletion. On rented hardware, ask whether expansion needs additional drives, a storage-layout change, downtime, or a replacement server. Adding a drive does not by itself enlarge the filesystem receiving application writes.

Set a recovery margin that lasts until the next intervention

Returning a small amount of capacity is only a temporary result if the producer continues at the same rate. As a planning estimate, divide currently usable free capacity by recent net growth, using consistent units.

Illustrative calculation: 24 GiB free divided by steady net growth of 2 GiB per hour gives approximately 12 hours to exhaustion. A bursty export, deployment, or retention change can invalidate that estimate. Use it to assess urgency, not to promise a safe operating window.

Reserve capacity for the next known operation as well as routine growth. Ask whether a backup, application update, database maintenance task, or cache rebuild needs temporary working space. Set the margin around those requirements and the time needed to respond.

Verify the service and involve the right owner

1. Repeat block and inode checks against the original failing path.
2. Confirm that the growth source is stopped, bounded, or redirected.
3. Run an application-approved write test and verify the resulting data.
4. Check failed jobs, partial uploads, retries, and database health using workload-specific procedures.
5. Resume paused work in stages while watching capacity and errors.

On an unmanaged rental, application retention and filesystem administration commonly remain customer tasks, but the actual agreement governs responsibility. Hardware symptoms, unexpected device loss, or provider-controlled expansion require provider involvement. Send the server ID, timestamps, affected mount, relevant errors, and requested action using a focused incident report.

If recovery now depends on restoring data, use the established restore-testing procedure to validate the recovery path. Do not overwrite the only remaining copy while testing an unproven backup.