Ubuntu OS Operations

Skills and command 101

Hardware

Hardware

Check disk I/O speed

우분투에서 디스크의 읽기 및 쓰기 성능을 확인하려면 dd 명령어, hdparm 명령어, 또는 fio 도구를 사용할 수 있습니다. 가장 쉽고 빠른 방법은 기본 도구인 dd와 hdparm을 활용하는 것입니다. [1, 2, 3, 4, 5]
1. dd 명령어로 쓰기 및 읽기 속도 측정하기
dd 명령어는 시스템에 파일을 직접 쓰고 읽어보며 간단하게 성능을 측정합니다. [1, 2]
쓰기 속도 테스트:
bash
sudo dd if=/dev/zero of=/testfile bs=1G count=1 oflag=direct
Use code with caution.

(1GB 크기의 더미 파일을 만들어 디스크에 직접 기록하는 속도를 측정합니다.)
읽기 속도 테스트:
bash
sudo dd if=/testfile of=/dev/null bs=1G count=1 iflag=direct
Use code with caution.

(앞서 생성한 테스트 파일을 다시 읽어들이는 속도를 측정합니다.)
테스트 완료 후 파일 삭제:
bash
sudo rm /testfile
Use code with caution.

[1]
    2. hdparm 명령어로 읽기 속도 측정하기
    hdparm은 하드디스크와 SSD의 캐시 및 버퍼 읽기 성능을 빠르게 확인하는 전용 도구입니다. [1, 2]
    설치 방법:
    bash
    sudo apt update && sudo apt install hdparm
    Use code with caution.

    속도 측정 실행:
    bash
    sudo hdparm -Tt /dev/sda
    Use code with caution.

    (/dev/sda 부분에는 확인하려는 실제 디스크 장치명을 입력합니다.)
      3. 실시간 디스크 사용량 확인 (iostat)
      현재 시스템에서 발생하는 실시간 입출력(I/O) 부하를 확인하려면 sysstat 패키지를 이용합니다. [1, 2]
      설치 방법:
      bash
      sudo apt install sysstat
      Use code with caution.

      실시간 모니터링 실행:
      bash
      iostat -xz 2
      Use code with caution.
      Hardware

      Clone damaged disk

      Reallocated Sectors: 90 suddenly increase and gives errors I/O input error. I wish to backup data to another disk any change to lost data?

      Yes, there is a strong chance that some data is already lost or at immediate risk.

      A sudden spike in reallocated sectors accompanied by I/O errors indicates active physical degradation on the disk media. Standard file copiers (like Windows Explorer or basic cp) will hang, freeze, or fail completely when encountering these sectors, worsening drive health in the process.

      Physical bad sectors prevent head read operations. Source: Adroit Data Recovery Centre

       

      What Is Happening to Your Disk

      • Reallocated Sectors (90): The drive controller identified damaged blocks, retired them, and remapped them to spare space. Files stored in those specific sectors prior to reallocation may already be corrupted.

      • I/O Errors: The drive's read/write head is repeatedly trying to read sectors that have failed but haven't been remapped yet (pending sectors). Every read attempt risks head contact and further mechanical wear.

      Critical Next Steps (Do Not Use Standard Copying)

      Do not run chkdsk, fsck, or drive diagnostic tools like S.M.A.R.T. extended self-tests. These perform heavy read/write operations that can destroy a failing drive.

      1. Prioritize Crucial Files First If the drive is still mounted, manually copy your absolute highest-value files (documents, personal records) individually to a healthy drive before doing anything else.

      2. Clone the Drive with a Fault-Tolerant Tool To rescue the maximum amount of data without hanging your system on bad blocks, clone the entire drive to a healthy disk of equal or larger size using a tool designed to skip bad sectors:

      • Linux / Command Line: Use ddrescue. It reads the healthy parts of the drive first, notes the bad sectors, and loops back later to attempt non-destructive retries.

        Bash 

        sudo ddrescue -d -r1 /dev/sdX /dev/sdY mapfile.log

        (Replace sdX with your dying drive and sdY with your target drive).

      • Windows (GUI): Use tools like HDD Raw Copy Tool or R-Studio to create an image file while ignoring I/O errors.

      3. Unmount and Replace Once the clone completes, disconnect the failing drive immediately. Perform any file system repair or data recovery operations on the cloned copy, never the original failing drive.

      If the drive begins making clicking noises or ceases to mount entirely during this process, power it off immediately; at that point, only a professional data recovery lab with cleanroom facilities can safely extract the remaining magnetic data.

       

      Depending on your Operating System and setup, we can tailor the exact recovery command:
      Show exact ddrescue commands for Linux rescue
      Recommend Windows tools for rescuing dying drives

       





      oh my os is ubunut and Can I clone my data to another machine using tailscale?

      Yes, you can clone the drive over Tailscale, but because of the failing state of the drive (I/O errors), you must use a tool that streams block-level data while skipping bad sectors over an SSH tunnel. Standard network copy tools (scp, rsync) will freeze or fail when they hit bad blocks.

      Step-by-Step Recovery over Tailscale

      1. Install GNU ddrescue on the Local Machine

      Install the fault-tolerant recovery tool on the Ubuntu machine with the dying drive:

      Bash 

      sudo apt update && sudo apt install gddrescue -y

      2. Identify Drive Device Names

      Find the device node for your failing drive (e.g., /dev/sdb or /dev/nvme0n1):

      Bash 

      lsblk

      (Verify carefully—do not mistake your system root drive for the dying drive).

      3. Stream ddrescue to the Remote Machine over Tailscale

      Run ddrescue on the local machine and pipe its output through SSH to write directly into an image file on the remote machine across your Tailnet:

      Bash 

      sudo ddrescue -d -r1 /dev/sdX - remote_user@REMOTE-TAILSCALE-IP:/path/to/backup/drive_rescue.img mapfile.log

      • /dev/sdX: Replace with your dying drive identifier.

      • -: Outputs the rescued raw data directly to stdout instead of a local file.

      • remote_user@REMOTE-TAILSCALE-IP: Use the Tailscale IP address or MagicDNS hostname of the destination machine.

      • /path/to/backup/drive_rescue.img: The destination file path on the receiving machine (ensure the receiving disk has enough free space to hold the full capacity of the source drive).

      Important Considerations for Network Recovery

      • Bandwidth & Speed limits: Rescuing a drive over a VPN will be bottlenecked by your upload speed. If the drive has hundreds of gigabytes, this will take significant time. If both machines are on the same local network, use the receiving machine's LAN IP inside the command to maximize speed while keeping Tailscale active.

      • Resume Capability: Piping ddrescue over SSH output directly into a raw file makes mapfile tracking remote-side trickier if the network drops. If network dropouts are a concern, mount the target machine's destination folder locally using SSHFS over Tailscale first:

        Bash 

        sudo apt install sshfs -y
mkdir -p /mnt/remote_backup
sshfs remote_user@REMOTE-TAILSCALE-IP:/path/to/backup /mnt/remote_backup
sudo ddrescue -d -r1 /dev/sdX /mnt/remote_backup/drive_rescue.img rescue.log

        This allows ddrescue to save a local mapfile, meaning you can pause or resume the recovery seamlessly if the connection breaks.

       

       





      ssh port is 2010

      To use port 2010, pass the -P parameter (capital P) inside the sshfs mount command or the -p parameter (lowercase p) if piping directly through ssh.

      Using the SSHFS method is strongly recommended here—it keeps your rescue.log mapfile local so you can safely stop and resume the clone if your Tailscale connection drops.

      Method 1: Recommended (SSHFS Mount on Port 2010)

      1. Create a local mount point and mount the target machine:

      Bash 

      mkdir -p /mnt/remote_backup
sshfs -p 2010 remote_user@REMOTE-TAILSCALE-IP:/path/to/backup /mnt/remote_backup

      2. Start ddrescue writing directly into the mount:

      Bash 

      sudo ddrescue -d -r1 /dev/sdX /mnt/remote_backup/drive_rescue.img /var/log/rescue.log

      (Replace /dev/sdX with your failing drive, and update remote_user and REMOTE-TAILSCALE-IP accordingly).

      Method 2: Direct SSH Stream on Port 2010

      If you prefer streaming directly without mounting SSHFS, pass -p 2010 to the ssh command:

      Bash 

      sudo ddrescue -d -r1 /dev/sdX - | ssh -p 2010 remote_user@REMOTE-TAILSCALE-IP "cat > /path/to/backup/drive_rescue.img"

      Unmounting When Finished

      When the cloning process completes, clean up the mount with:

      Bash 

      sudo umount /mnt/remote_backup




      img is how big ?

      The .img file will be the exact total capacity of the dying source disk, not just the size of the files currently stored on it.

      Because ddrescue creates a raw, sector-by-sector clone of the underlying physical storage, it copies every byte on the drive—including empty space and corrupted sectors.

      Disk Capacity vs. File Size

      • 1 TB Hard Drive → .img file will be 1 TB

      • 500 GB SSD/HDD → .img file will be 500 GB

      Example: If you have a 1 TB hard drive that only contains 50 GB of personal photos and documents, the resulting drive_rescue.img file will still be 1 TB.

      Crucial Pre-Flight Check

      Before starting the transfer, verify that the storage drive on your remote machine has more free space available than the total size of your failing disk.

      1. Check source drive size (local machine):

      Bash 

      lsblk -b /dev/sdX | grep disk

      (Look at the total size in bytes/gigabytes).

      2. Check destination free space (remote machine over SSH port 2010):

      Bash 

      ssh -p 2010 remote_user@REMOTE-TAILSCALE-IP "df -h /path/to/backup"

      (Ensure the Avail column is larger than the total size of your source drive).

      What to Do After the .img File Is Created

      Once the clone finishes, do not try to open the .img file like a standard archive. You can mount it directly on the remote Linux machine to browse and copy your healthy files out:

      Bash 

      sudo loopsetup=$(sudo losetup -fP --show /path/to/backup/drive_rescue.img)
sudo mount ${loopsetup}p1 /mnt/restored_files

      (Replace p1 with the specific partition number you want to access).





      Can I just clone one drive to another drive?

      Yes, absolutely. Cloning directly from disk to disk (e.g., from /dev/sdX to /dev/sdY) is actually simpler and much faster because it avoids network bottlenecks, SSH overhead, and intermediate image files.

      Prerequisites

      1. Target Drive Size: The receiving drive must be equal to or larger in capacity than the dying source drive.

      2. Drive Identifiers: You must strictly double-check your drive identifiers using lsblk before running the command. Overwriting the wrong drive will permanently wipe it.

      Step-by-Step Disk-to-Disk Rescue

      1. Identify your drives:

      Bash 

      lsblk -o NAME,SIZE,MODEL,SERIAL

      • Identify your dying source drive (e.g., /dev/sdb)

      • Identify your healthy destination drive (e.g., /dev/sdc)

      2. Run ddrescue directly between the drives:

      Bash 

      sudo ddrescue -d -r1 /dev/sdX /dev/sdY /var/log/rescue.log

      • Replace /dev/sdX with your dying source drive.

      • Replace /dev/sdY with your healthy destination drive.

      • /var/log/rescue.log saves your progress locally on your root OS so you can pause or resume if needed.

      What Happens Next?

      Once ddrescue finishes:

      1. Disconnect the dying drive immediately.

      2. The destination drive (/dev/sdY) is now a 1:1 identical twin of your original disk, including partitions and file systems.

      3. You can safely mount the destination drive and run filesystem checks (like e2fsck or ntfsfix) on it to repair any corruption caused by the bad sectors.

       

       





      Target disk sde    931.5G Tech           K94PT9325BHV

      Source disk sda          931.5G D HTS541010A9E63  000000004BA8

      Your drive sizes match (both 931.5 GB), so a direct disk-to-disk clone will work.

      • Source (Dying): /dev/sda

      • Target (Destination): /dev/sde

      Step 1: Run ddrescue

      Execute the following command to begin copying data directly from sda to sde:

      Bash 

      sudo ddrescue -d -r1 /dev/sda /dev/sde /var/log/sda_rescue.log

      • -d: Direct disk access (bypasses OS kernel cache to read straight from hardware).

      • -r1: Retries unreadable sectors once before skipping ahead, prioritizing maximum data recovery without destroying the drive.

      • /var/log/sda_rescue.log: Saves the mapfile to your root OS so you can safely interrupt (Ctrl+C) and resume later if necessary.

      Step 2: What to Expect During the Run

      1. Pass 1 (Copying): It will quickly copy all the undamaged sectors first.

      2. Pass 2 (Trimming/Scraping): It will return to the bad areas and attempt to extract data around the 90 bad sectors you encountered.

      3. Keep an eye on the pct rescued and errsize lines in the output.

      Step 3: Post-Rescue Cleanup

      Once ddrescue reports finished:

      1. Unplug /dev/sda immediately to prevent further mechanical wear.

      2. Re-scan your partitions or reboot your system:

        Bash 

        sudo partprobe /dev/sde

      3. Your /dev/sde drive will now have the exact partitions and file structure that /dev/sda had. You can safely mount /dev/sde and check your files.

       

       





      IS this work can be done via tailscale?

      Yes, absolutely. You can execute this disk-to-disk clone across machines connected via Tailscale.

      However, because you are cloning a whole 931.5 GB drive directly over the network, there are critical speed and command adjustments to keep in mind.

      Step-by-Step: Disk-to-Disk Clone Over Tailscale

      Assuming /dev/sda is on your Local Machine (with the dying drive) and /dev/sde is physically attached to the Remote Machine on your Tailnet.

      1. Command to Run on Local Machine (Port 2010)

      Pipe ddrescue directly through SSH to write onto /dev/sde on the remote side:

      Bash 

      sudo ddrescue -d -r1 /dev/sda - | ssh -p 2010 remote_user@REMOTE-TAILSCALE-IP "sudo dd of=/dev/sde status=progress bs=64k"

      • /dev/sda -: Tells ddrescue to read the local drive and output the raw stream to standard output (stdout).

      • ssh -p 2010: Connects through your non-standard SSH port over Tailscale.

      • sudo dd of=/dev/sde: Takes the incoming network stream on the remote machine and writes it directly to the physical target disk (sde).

      • bs=64k: Sets block size to optimize network throughput.

      Critical Considerations Before You Start

      1. Transfer Time (Network Bottleneck)

      Over a local disk bus (SATA/USB 3), cloning 931.5 GB takes roughly 2 to 3 hours. Over a Tailscale network connection, your speed will be limited by your upload speed:

      Network Upload Speed Estimated Time to Transfer 931.5 GB
      100 Mbps (~12.5 MB/s) ∼ 21 hours
      500 Mbps (~62.5 MB/s) ∼ 4.2 hours
      1 Gbps (~125 MB/s) ∼ 2.1 hours

      If both machines are on the same physical local network (e.g., same router/switch), use the destination machine's local LAN IP inside the SSH command instead of the Tailscale IP to get full gigabit speeds.

      2. Risk of Network Dropouts

      Piping ddrescue directly over SSH standard output does not allow saving a detailed bad-sector mapfile across the network connection. If your network hiccups or SSH drops mid-transfer, you cannot resume from where you left off; you would have to start from the beginning.

      Recommendation: Run the command inside a tmux or screen session on your local terminal so that a closed SSH/terminal session won't kill the running process:

      Bash 

      tmux new -s rescue

      (If disconnected, re-attach later using tmux a -t rescue)

       

       





      need to format for target disk ?

      No, you do not need to format the target disk (/dev/sde).

      In fact, formatting it beforehand would be a wasted step. ddrescue performs a low-level, sector-by-sector copy—meaning it will overwrite every single byte on the target disk, including its partition table, file system headers, and formatting data.

      What Will Happen to the Target Disk (/dev/sde)

      • Any existing partitions or files on /dev/sde will be completely wiped and replaced.

      • Once the transfer finishes, /dev/sde will become an exact duplicate of /dev/sda (same partition structure, same file system type, same layout).

      One Important Prerequisite

      While you don't need to format /dev/sde, you must ensure that none of its partitions are mounted before running the recovery command:

      Bash 

      sudo umount /dev/sde* 2>/dev/null

      This prevents the Linux kernel on the receiving machine from writing to /dev/sde while ddrescue is actively overwriting the raw drive blocks over your network stream.





      41250 bytes (41 kB, 40 KiB) copied, 76 s, 0.5 kB/sddrescue: /dev/sda: Fatal error reading the input file: No such device

      41720 bytes (42 kB, 41 KiB) copied, 77 s, 0.5 kB/s

      0+78 records in

      0+78 records out

      41720 bytes (42 kB, 41 KiB) copied, 76.9141 s, 0.5 kB/s

      Fatal error reading the input file: No such device means your dying hard drive (/dev/sda) has physically disconnected from the machine.

      When a drive with severe bad sectors suffers repeated read failures, its internal controller crashes, or the Linux kernel resets the SATA/USB bus to prevent system lockup. As a result, the hardware completely disappears from /dev/sda. Notice how it only managed to copy ~41 kB before disconnecting.

      Step 1: Check if the Drive Is Still Detected

      Run this command to see if the system can still see the hardware:

      Bash 

      lsblk

      Or check kernel logs to see the drop event:

      Bash 

      dmesg -T | grep -i "ata\|sd\|error" | tail -n 20

      Step 2: How to Re-identify or Reset the Drive

      If lsblk no longer lists /dev/sda:

      1. Power Cycle / Physical Reset:

        • If it is an external USB drive: Unplug the USB cable and power source, wait 10 seconds, and plug it back in.

        • If it is an internal SATA drive: Shut down the machine completely, power off the PSU, wait 30 seconds, and boot back up.

      2. Check the New Drive Letter: When the drive reconnects, Linux may assign it a different letter (e.g., /dev/sdb or /dev/sdc instead of /dev/sda). Always verify with lsblk before running the next command.

      Step 3: Crucial Strategy Adjustment for ddrescue

      Your drive crashed because ddrescue attempted direct access (-d) on a severely degraded sector right at the beginning of the disk.

      When you restart ddrescue, remove the -d (direct I/O) flag initially and add --scrape-members / --no-scrape options to skip damaged areas instantly without hanging the drive controller:

      Bash 

      sudo ddrescue -n -b 4096 /dev/sdX - | ssh -p 2010 remote_user@REMOTE-TAILSCALE-IP "sudo dd of=/dev/sde status=progress bs=64k"

      • -n (--no-scrape): Tells ddrescue to skip bad areas immediately instead of retrying them. This prevents the drive from hanging and disconnecting.

      • -b 4096: Force a 4KiB block size (many modern drives use 4k physical sectors; attempting 512-byte reads can trigger controller panics on failing drives).

      • /dev/sdX: Replace with the new drive letter assigned after reconnecting.

      Warning Signs of Permanent Hardware Failure

      If the drive disconnects again almost immediately after starting (within seconds or after copying just a few kilobytes):

      • The drive's read heads or PCB controller are failing permanently.

      • Continued attempts will degrade the platters further. At that stage, software cloning over a network won't work, and recovery requires specialized hardware tools or a professional recovery service.





      disk is still connected and


      [Mon Aug  3 13:08:38 2026] I/O error, dev sda, sector 0 op 0x0:(READ) flags 0x800 phys_seg 16 prio class 2

      [Mon Aug  3 13:21:05 2026] sd 1:0:0:0: [sda] tag#1 uas_zap_pending 0 uas-tag 1 inflight: CMD 

      [Mon Aug  3 13:21:05 2026] sd 1:0:0:0: [sda] tag#1 CDB: Read(10) 28 00 00 b8 70 80 00 00 80 00

      [Mon Aug  3 13:21:05 2026] sd 1:0:0:0: [sda] tag#1 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s

      [Mon Aug  3 13:21:05 2026] sd 1:0:0:0: [sda] tag#1 CDB: Read(10) 28 00 00 b8 70 80 00 00 80 00

      [Mon Aug  3 13:21:05 2026] I/O error, dev sda, sector 12087424 op 0x0:(READ) flags 0x800 phys_seg 16 prio class 2

      [Mon Aug  3 13:21:05 2026] device offline error, dev sda, sector 12107136 op 0x0:(READ) flags 0x800 phys_seg 16 prio class 2

      [Mon Aug  3 13:21:05 2026] sd 1:0:0:0: [sda] Synchronizing SCSI cache

      [Mon Aug  3 13:21:05 2026] sd 1:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK

      [Mon Aug  3 13:21:06 2026] EXT4-fs (sda): unmounting filesystem dc434a0a-8500-41a2-9fd3-37b3adef0cc5.

      [Mon Aug  3 13:21:09 2026] sd 1:0:0:0: Attached scsi generic sg0 type 0

      [Mon Aug  3 13:21:09 2026] sd 1:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)

      [Mon Aug  3 13:21:09 2026] sd 1:0:0:0: [sda] 4096-byte physical blocks

      [Mon Aug  3 13:21:09 2026] sd 1:0:0:0: [sda] Write Protect is off

      [Mon Aug  3 13:21:09 2026] sd 1:0:0:0: [sda] Mode Sense: 53 00 00 08

      [Mon Aug  3 13:21:09 2026] sd 1:0:0:0: [sda] Disabling FUA

      [Mon Aug  3 13:21:09 2026] sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

      [Mon Aug  3 13:21:09 2026] sd 1:0:0:0: [sda] Preferred minimum I/O size 4096 bytes

      [Mon Aug  3 13:21:09 2026] sd 1:0:0:0: [sda] Optimal transfer size 33553920 bytes not a multiple of preferred minimum block size (4096 bytes)

      [Mon Aug  3 13:21:09 2026] sd 1:0:0:0: [sda] Attached SCSI disk

      The logs show two very important things:

      1. UAS Protocol Crash (uas_zap_pending, DID_NO_CONNECT): You are using a USB enclosure that relies on the UAS (USB Attached SCSI) protocol. UAS is fast, but it panics and completely drops the drive connection when it encounters disk read timeouts or bad sectors.

      2. Drive Auto-Reattached: At 13:21:09, the kernel reset the USB bus and re-attached /dev/sda. However, as long as UAS remains active, any attempt to read bad sectors will crash the USB controller again.

      To fix this, you must disable UAS for this drive so it falls back to the legacy, rock-solid usb-storage protocol. The standard usb-storage driver handles read errors gracefully without dropping the entire device.

      Step 1: Disable UAS and Force usb-storage

      1. Find your USB enclosure's Vendor and Product ID:

      Bash 

      lsusb | grep -i "terramaster\|orico\|sabrent\|enclosure\|drive"

      (Or simply run lsusb to see all USB devices). Look for a string like ID 152d:0578 (where 152d is Vendor ID and 0578 is Product ID).

      2. Unbind UAS temporarily using modprobe: Run this command (replace 152d:0578 with your actual IDs):

      Bash 

      sudo modprobe -r uas
sudo modprobe usb-storage quirks=152d:0578:u

      (The :u flag forces Linux to ignore UAS and use standard usb-storage for this device).

      3. Unplug and replug the USB cable. Verify it dropped UAS by checking dmesg:

      Bash 

      dmesg | tail -n 15

      You should see USB Mass Storage device detected instead of UAS driver.

      Step 2: Disable Automatic Mounting

      Notice line 13:21:06 in your log showing EXT4-fs (sda): unmounting. Ubuntu tried to automount the file system, which triggers immediate background reads on corrupted sectors.

      Prevent Ubuntu from automounting or touching the partition structure while cloning:

      Bash 

      gsettings set org.gnome.desktop.media-handling automount false

      Step 3: Run ddrescue with Gentle Flags

      Now that UAS is disabled, retry cloning over Tailscale. Use the -n (--no-scrape) option and a 4096-byte block size so ddrescue quickly skips bad blocks instead of getting stuck:

      Bash 

      sudo ddrescue -n -b 4096 /dev/sda - | ssh -p 2010 remote_user@REMOTE-TAILSCALE-IP "sudo dd of=/dev/sde status=progress bs=64k"

      • Because UAS is disabled, when ddrescue hits sector 12087424 or sector 0, the kernel will simply return an I/O error to ddrescue.

      • ddrescue will mark that block as bad, skip ahead, and keep copying the rest of your healthy data without crashing the USB bus.





      us 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

      Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

      Bus 002 Device 002: ID 2109:2812 VIA Labs, Inc. VL812 Hub

      Bus 002 Device 003: ID 13d3:5741 IMC Networks Lenovo EasyCamera

      Bus 002 Device 004: ID 105b:e065 Foxconn International, Inc. BCM43142A0 Bluetooth module

      Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

      Bus 003 Device 002: ID 2109:0812 VIA Labs, Inc. VL812 Hub

      Bus 003 Device 004: ID 2537:1068 Norelsys NS1068/NS1068X SATA Bridge Controller

      Bus 003 Device 005: ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter

      Bus 003 Device 006: ID 152d:0578 JMicron Technology Corp. / JMicron USA Technology Corp. JMS578 SATA 6Gb/s

      You have two external SATA adapters connected. The one causing the UAS crash is the JMicron JMS578 (152d:0578) controller on Bus 003.

      Step 1: Force usb-storage Driver for JMicron (152d:0578)

      Unload the crashing uas module and reload usb-storage with the quirk flag for 152d:0578:

      Bash 

      sudo modprobe -r uas
sudo modprobe usb-storage quirks=152d:0578:u

      Step 2: Cycle the Connection

      1. Unplug the USB cable for the JMicron enclosure, wait 5 seconds, and plug it back in.

      2. Verify that UAS is disabled by checking the kernel log:

        Bash 

        dmesg | tail -n 20

        Look for UAS is ignored for this device or USB Mass Storage device detected instead of uas.

      Step 3: Check Drive Assignment & Disable Auto-Mount

      1. Confirm the drive letter hasn't shifted:

        Bash 

        lsblk

      2. Stop GNOME from auto-mounting corrupted partitions when attached:

        Bash 

        gsettings set org.gnome.desktop.media-handling automount false

      Step 4: Resume ddrescue

      Run ddrescue with --no-scrape (-n) and a 4KiB block size (-b 4096). It will now report read errors to ddrescue instantly and skip ahead without crashing the USB adapter:

      Bash 

      sudo ddrescue -n -b 4096 /dev/sda - | ssh -p 2010 remote_user@REMOTE-TAILSCALE-IP "sudo dd of=/dev/sde status=progress bs=64k"

      (Replace /dev/sda if lsblk shows a different letter after re-plugging).





      after dd is finished, how to know ?which files are damaged

      To identify which specific files are damaged, you need a map of where the bad sectors are located on the disk, and then you cross-reference those bad sector addresses with your filesystem's inode table using testdisk or debugfs.

      Because you piped ddrescue directly into dd over SSH, no local rescue.log mapfile was generated. However, you can still pinpoint corrupt files directly on the cloned target disk (/dev/sde).

      Step 1: Scan the Target Disk for Bad Sectors

      Run badblocks on the cloned target drive /dev/sde. Since /dev/sde now contains an exact sector-by-sector copy of /dev/sda, reading /dev/sde will reveal where unreadable/skipped zeros were written:

      Bash 

      sudo badblocks -v -b 4096 /dev/sde > bad_sectors.txt

      (Save the list of bad sector block addresses to bad_sectors.txt).

      Step 2: Map Bad Sectors to Corrupted Files

      Depending on your file system type (e.g., ext4 or NTFS), use the appropriate tool below to match those sector numbers to actual file paths.

      Option A: For EXT4 Filesystems (Using debugfs)

      1. Find the partition containing your files (e.g., /dev/sde1):

        Bash 

        sudo lsblk /dev/sde

      2. Open the partition in debugfs:

        Bash 

        sudo debugfs /dev/sde1

      3. Inside the debugfs: prompt, test a block number from your bad_sectors.txt file to find its corresponding inode:

        Plaintext 

        debugfs: icheck <BLOCK_NUMBER>

        (This outputs an inode number, e.g., Block 12087424 has inode 524289).

      4. Get the exact file path for that inode:

        Plaintext 

        debugfs: ncheck 524289

        (This outputs the exact file path, e.g., /home/user/Documents/report.pdf).

      Option B: Automated GUI Recovery (Using TestDisk)

      TestDisk is much easier if you prefer an automated tool to scan and list damaged files across ext4, NTFS, or FAT32 filesystems:

      1. Install testdisk:

        Bash 

        sudo apt install testdisk -y

      2. Run testdisk on the target disk:

        Bash 

        sudo testdisk /dev/sde

      3. Select Proceed → Choose your partition table style (usually EFI GPT) → Analyse → Quick Search.

      4. Highlight your partition and press P to list files. TestDisk will display files in red if their metadata or data blocks are unreadable or damaged.

      Step 3: Run Filesystem Repair (e2fsck)

      Before mounting /dev/sde for normal use, repair any filesystem metadata inconsistencies caused by bad sectors:

      Bash 

      sudo e2fsck -f -y /dev/sde1

      • Important: Running e2fsck will fix the file system structure so the drive can mount safely. Any unrecoverable blocks within files will be zeroed out or placed into a /lost+found directory on the drive.





      Basically , target disk has no bad sector physically, then badsector information from source disk affects on target disk ?

      No, the target disk will not gain physical bad sectors.

      Because /dev/sde is a healthy physical drive, its physical platters or NAND flash cells remain 100% fine. However, logical/file system corruption will be copied over.

      Physical vs. Logical Bad Sectors

      Feature Source Disk (/dev/sda) Target Disk (/dev/sde)
      Physical Media Damaged (hardware read failures) Healthy (0 physical bad sectors)
      Data in Bad Sectors Unreadable Filled with zeros by ddrescue
      File System Status Corrupted Corrupted (reads zeros where data used to be)

      What Actually Happens to Target Disk (/dev/sde)

      1. Zeros written to bad spots: When ddrescue encounters a bad sector on /dev/sda that it cannot read, it fills those exact block spaces on /dev/sde with zeros (empty data) so the clone can continue.

      2. Missing file pieces: The physical sectors on /dev/sde are perfectly capable of being read and written to, but the content inside those specific files is now corrupted or truncated.

      3. File system structure: If bad sectors occurred inside file system metadata (e.g., the superblock or inode table), the target disk may refuse to mount initially until you run a repair command (e2fsck).

      How to Fix the File System on Target Disk

      Once ddrescue completes, run a file system check on /dev/sde. This cleans up the logical errors so the operating system can safely use the drive:

      Bash 

      sudo e2fsck -f -y /dev/sde1

      • This fixes any corrupted folder structures.

      • The healthy target disk will function normally going forward, though any specific files that were sitting on sda's bad sectors will have missing/zeroed-out data or be moved to /lost+found.





      Can I know the list of damaged files?

      Yes, you can generate a complete list of damaged files directly from the target disk (/dev/sde), even though it has no physical bad sectors of its own.

      Because ddrescue writes all zeros to the target disk whenever it encounters an unreadable sector on the source drive, any file containing those zeroed-out blocks is corrupted.

      Method 1: The Automated Way (Using ddrescue-tools)

      If you want a direct list of damaged file paths, use ddrescue-view or ddrescue-tools to parse the cloned partition:

      1. Install ddrescue-tools:

        Bash 

        sudo apt update && sudo apt install ddrescue-tools -y

      2. Run ddrescue-findfiles against the target partition:

        Bash 

        sudo ddrescue-findfiles /dev/sde1 /var/log/sda_rescue.log

        (This utility maps the log file's bad sector offsets directly against the filesystem on /dev/sde1 and prints out every impacted path).

      Method 2: Finding Corrupted Files manually on EXT4 (debugfs)

      If you don't have a mapfile from ddrescue, you can scan /dev/sde1 for filesystem blocks that contain unreadable metadata or lost inodes:

      1. Run a thorough filesystem check on the cloned partition:

        Bash 

        sudo e2fsck -f -v -y /dev/sde1 | tee filesystem_repair.log

        • As e2fsck runs, it will print messages whenever it encounters corrupted inodes or unreadable file blocks (e.g., Inconsistency found in inode 12345).

        • Any files that are severely damaged and disconnected from their parent directory will be safely moved to a folder named /lost+found on /dev/sde1.

      2. Mount /dev/sde1 to inspect /lost+found:

        Bash 

        sudo mkdir -p /mnt/target
sudo mount /dev/sde1 /mnt/target
ls -la /mnt/target/lost+found

      Method 3: Finding Zeroed / Damaged Files by Checksum Verification

      If you have a previous backup or known checksums (like .md5 or .sha256 files) for your media, run a verification scan over your mounted files:

      Bash 

      find /mnt/target -type f -exec md5sum {} + > /tmp/checksums.txt 2> /tmp/corrupted_files.txt

      Any file that fails to read or produces an error will be logged into /tmp/corrupted_files.txt.

      Security

      Security

      Brute attack and /var/log/btmp

      **/var/log/btmp` is a system security log that records all failed login attempts to your Linux server.**
      Every time someone tries to log in via SSH, the terminal, or local console and enters an incorrect password or username, the system logs it here. Because it captures malicious activity, it is a vital tool for checking if your server is under a brute-force attack.
      Key Characteristics of /var/log/btmp
      • It is a binary file: You cannot read it using normal text commands like cat, nano, or less. If you try, your screen will fill with unreadable scrambled code.


      • Strict Permissions: Because it logs failed login details (which sometimes accidentally contain real passwords typed into the username field), only the root user can read it.
      How to View It Safely
      To read this binary file, Linux includes a specific tool called last with the -f flag. Run this command with sudo:
      bash
      sudo last -f /var/log/btmp
      Use code with caution.

      What the output looks like:
      text
      root     ssh:notty    192.168.1.50   Thu Aug 27 17:45 - 17:45  (00:00)
admin    ssh:notty    45.142.120.14  Thu Aug 27 17:42 - 17:42  (00:00)
invalid  ssh:notty    185.220.101.5  Thu Aug 27 17:40 - 17:40  (00:00)
      Use code with caution.

      • Column 1: The username attempted (e.g., root, admin, or completely fake usernames like invalid).
      • Column 3: The IP address of the machine or bot trying to break into your server.
      Common Issues: Why is it growing large?
      If your server has a public IP address and port 22 (SSH) is open to the internet, automated botnets will target it 24/7 trying thousands of common passwords. This can cause /var/log/btmp to swell to hundreds of megabytes or gigabytes very quickly.
      To check how many failed login attempts have hit your server, you can count the lines in the parsed file:
      bash
      sudo last -f /var/log/btmp | wc -l
      Use code with caution.










      Even with SSH moved to a custom port like 2010, sophisticated automated bots will eventually scan your server, find the open port, and begin brute-forcing it.
      You can use a few quick terminal commands to analyze your /var/log/btmp file and find out exactly who is attacking you and if Fail2ban is doing its job.
      1. Find the Top Attacking IP Addresses
      To see a ranked list of the specific IP addresses hitting your server with failed login attempts, parse the btmp file using this command:
      bash
      sudo last -f /var/log/btmp | awk '{print $3}' | sort | uniq -c | sort -nr | head -n 10
      Use code with caution.

      • What this does: It extracts the IP column, counts the duplicate entries, and displays the top 10 most aggressive IP addresses alongside the exact number of failed attempts they made.
      2. Find the Most Targeted Usernames
      Attackers usually guess common system accounts. Run this to see what usernames they are trying to use to break in:
      bash
      sudo last -f /var/log/btmp | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 10
      Use code with caution.

      • What to look for: You will likely see root, admin, user, or ubuntu. If you see actual personal usernames that exist on your system, it means the attack is more targeted.
      3. Verify if Fail2ban is Actually Blocking Them
      Moving SSH to port 2010 means you need to ensure Fail2ban is actually listening to that custom port. Otherwise, it will ignore the attacks.
      First, check the status of your SSH jail:
      bash
      sudo fail2ban-client status sshd
      Use code with caution.

      • Look at the "Banned IP list" at the bottom of the output. If it shows active IP addresses, Fail2ban is successfully protecting your new port.
      • If the list is completely empty despite having thousands of entries in your btmp log, Fail2ban might still be looking for port 22.
      4. Ensure Fail2ban Knows Your Custom Port
      If Fail2ban isn't catching them, open your Fail2ban configuration file (usually /etc/fail2ban/jail.local):
      ini
      [sshd]
enabled = true
port    = 2010
logpath = %(sshd_log)s
backend = %(sshd_backend)s
      Use code with caution.

      Ensure port = 2010 is explicitly set. If you make changes, restart the service:
      bash
      sudo systemctl restart fail2ban
      Use code with caution.









      Your terminal output provides highly valuable security data.
      Yes, you are definitely being attacked by automated brute-force bots. The IP address 103.137.184.170 alone has hammered your custom SSH port over 216,000 times.
      The good news is that Fail2ban is working. It actively monitors your custom port (as shown by Journal matches: _SYSTEMD_UNIT=sshd.service), has banned 427 IPs total, and currently has 7 IPs blocked—including 211.253.8.115, which is number 10 on your top attackers list.
      However, because the attack numbers are so massive (hundreds of thousands of attempts), Fail2ban is letting too many retries through before trigger-banning, or your ban times are too short.
      Recommended Hardening Steps
      1. Make Fail2ban More Aggressive
      Currently, bots are allowed to try thousands of times because your ban duration might be too short, or they reset after a few failures. Let's make it strict. Open your Fail2ban local configuration file:
      bash
      sudo nano /etc/fail2ban/jail.local
      Use code with caution.

      Find or create your [sshd] section and adjust these parameters:
      ini
      [sshd]
enabled = true
port    = 2010
maxretry = 3          ; Ban them after just 3 failed attempts
findtime = 1d         ; Look for 3 failures within a 24-hour window
bantime  = 1w         ; Ban them for a full week (or use -1 for permanent)
      Use code with caution.

      Save the file (Ctrl+O, Enter, Ctrl+X) and restart Fail2ban:
      bash
      sudo systemctl restart fail2ban
      Use code with caution.

      2. Manually Ban the #1 Threat Right Now
      The IP 103.137.184.170 has broken the threshold completely. You can force Fail2ban to ban it instantly without waiting for the next trigger:
      bash
      sudo fail2ban-client set sshd banip 103.137.184.170
      Use code with caution.

      3. Disable Root Logins Over SSH
      Your log shows 82,029 attempts trying to guess the password for the root user. If you haven't already, completely disable logging directly into the root account via SSH.
      Open your SSH daemon configuration file:
      bash
      sudo nano /etc/ssh/sshd_config
      Use code with caution.

      Find the line PermitRootLogin and change it to no:
      text
      PermitRootLogin no
      Use code with caution.

      Save the file and restart your SSH service to apply:
      bash
      sudo systemctl restart ssh
      Use code with caution.

      (Note: Ensure you have your normal meson user password or SSH key working before doing this, so you don't lock yourself out).
      4. Clean up /var/log/btmp
      Now that you have analyzed the threat, you can safely clear out this binary log file to reclaim disk space, just like you did with the MariaDB log:
      bash
      sudo truncate -s 0 /var/log/btmp
      Use code with caution.

      To wrap up your server security checklist, let me know:
      • Do you log in using a password or an SSH Cryptographic Key? (Switching to keys stops brute forces completely).
      • Would you like help configuring UFW (the firewall) to restrict SSH port 2010 access to only your specific home/office IP?
      AI responses may include mistakes. Learn more


      Using an SSH key is the absolute best security choice you could make. Even with 216,000 brute-force password guesses, it is mathematically impossible for those bots to break in because your server completely rejects password authentication for your key-based account.
      Since you are already using UFW, you can easily block these relentless bots at the firewall layer. This is much more efficient because it drops their network traffic completely before it can even touch or slow down your SSH service.
      Here is how to finalize and lock down your setup based on your configuration:
      1. Close the Root Exploit Completely
      In Ubuntu, if #PermitRootLogin prohibit-password is commented out, the system defaults to allowing root login attempts under certain conditions. Since you are using SSH keys for your user account, you should explicitly deny the root user from ever logging in.
      Open your configuration file:
      bash
      sudo nano /etc/ssh/sshd_config
      Use code with caution.

      Add or modify this exact line (make sure there is no # symbol in front of it):
      text
      PermitRootLogin no
      Use code with caution.

      Save the file (Ctrl+O, Enter, Ctrl+X) and restart SSH:
      bash
      sudo systemctl restart ssh
      Use code with caution.

      2. Block the Worst Attacker in UFW Immediately
      Fail2ban works dynamically, but for an IP address as aggressive as 103.137.184.170, you should block it permanently at the firewall level so it cannot access your Nextcloud instance or any other port:
      bash
      sudo ufw insert 1 deny from 103.137.184.170 to any
      Use code with caution.

      (Using insert 1 ensures this rule sits at the very top of your firewall checklist, blocking the packet instantly).
      3. Ultimate Security: Restrict Port 2010 to Your IP Only
      If you always connect to your server from the same location (like your home, office, or a static VPN IP), you can tell UFW to block the entire world from port 2010 except for you.
      First, delete your generic open SSH rule:
      bash
      sudo ufw delete allow 2010/tcp
      Use code with caution.

      Next, allow only your specific IP address to see port 2010:
      bash
      sudo ufw allow from YOUR_HOME_IP to any port 2010 proto tcp
      Use code with caution.

      (Replace YOUR_HOME_IP with your actual public IP address, which you can find by searching "what is my IP" on your computer).
      Once you run this, the 216,000 automated bot attempts will instantly drop to zero because their network packets will be silently ignored by UFW.
      Security

      Security against Brute attack

      This volume of failed logins is normal background noise for any server with port 22 open to the internet — automated bots scan all IPv4 addresses and start hammering common usernames within minutes of your server going online. Your fail2ban and UFW are doing exactly what they're supposed to: stopping them from actually getting in.

      That said, you can eliminate most of this traffic entirely. If you only have SSH keys enabled, the thousands of password guesses are harmless and simply never succeed.

      Is this normal?

      • 1564 failed logins / 192 bans / 7329 firewall blocks is a typical-to-calm level for a public server. Many freshly-unprotected boxes see tens of thousands of attempts daily.

      • The important test: do any of the failed logins succeed? If they don't (key-based auth only), this is nuisance traffic, not a breach.

      • One thing worth checking: if most failures come from a single IP, fail2ban may be misconfigured or its filter may not match your auth log format — otherwise it should be banning that IP. Verify with sudo fail2ban-client status sshd.

      What you can do to cut it down

      The single most effective change is to make password login impossible, so the brute-force attempts can never succeed:

      bash

      • Use SSH keys only — this alone eliminates the overwhelming majority of brute-force risk.

      • Disable root login (PermitRootLogin no) so attackers can't target root directly.

      • Rate-limit at the firewall: sudo ufw limit OpenSSH (allows ~6 connections per 30s per IP, dropping brute-force spikes at the kernel level).

      • Use a non-standard port (e.g., 2222) — reduces scanner traffic dramatically, though it's security-by-obscurity.

      • Restrict access to known admin IPs or a VPN (Tailscale/WireGuard) if your IP is static — then port 22 isn't exposed at all.

      Before you change anything

      • Confirm key-based login already works from a fresh session before disabling passwords, or you'll lock yourself out.

      • Keep one SSH session open, back up the config (cp, then sshd -t to validate syntax), then sudo systemctl reload ssh — not restart, so existing sessions survive.

      • Check that your cloud provider has a web/serial console as a break-glass fallback.

      Bottom line: there's nothing special or alarming in your numbers — it's the standard pre-scan traffic every internet-exposed server receives. If you want to reduce it (and the log noise), the fixes are key-only auth + disable root + firewall rate limiting. Once that's in place, the 1564 failures are wasted effort against a door that's already locked.

      Want me to check whether any of the failed logins actually succeeded on a real account, or walk through a specific hardening step?

      Used ssh keys only and disabled root login and I am using custom port 2010 for ssh. Is there anything if I changed ssh port for fail2ban ?


      Yes — you should update fail2ban so its ban action targets your port 2010. The good news: fail2ban detects attacks by parsing your auth log, not by watching ports, so it still sees the failed logins on any port. The problem is only the ban itself — if fail2ban's action still blocks port 22 (or the OpenSSH app profile that points to 22), the banned IPs aren't actually being blocked from connecting to 2010.

      What to change

      On Ubuntu with UFW, the cleanest way is a dedicated UFW application profile for port 2010, then point the sshd jail at it.

      1. Add an app profile for your port in /etc/ufw/applications.d/openssh-server:

      text

      2. Point the sshd jail at it in /etc/fail2ban/jail.local:

      text

      3. Reload and verify:

      bash

      The port = 2010 line tells fail2ban the port to report/ban, and the action = ufw[application="OpenSSH-2010"] makes UFW insert a deny from <ip> to any app OpenSSH-2010 rule that actually drops traffic to 2010.

      Important caveat

      There's a known failure mode: if you leave fail2ban's default iptables/ufw action pointing at 22 or forget to update it, fail2ban-client status sshd will show bans, but ufw status will have no matching deny rules and the IPs keep reaching 2010 — exactly the "fails to block ssh on custom port" problem.

      Two checks to confirm it's working after changing the port:

      • sudo fail2ban-client status sshd — shows the banned IP count.

      • sudo ufw status numbered — verify a matching DENY ... to 2010/tcp (or to ... app OpenSSH-2010) rule exists for recently banned IPs.

      If you'd rather not maintain a custom profile, the alternative is the simpler stock line action = ufw with banaction = ufw under [DEFAULT] — but that still binds to the port = 2010 you set, so make sure that's updated.

      One extra tip since you moved to 2010: add a second jail watching for activity on port 22 too — legacy scanners may still probe it, and banning them at 22 keeps noise out of your UFW logs.

      [OpenSSH-2010] title=Secure shell server (custom) description=OpenSSH on custom port ports=2010/tcpJust append this?

      Yes — just append that block to the end of /etc/ufw/applications.d/openssh-server. Adding a second [OpenSSH-2010] section alongside the existing [OpenSSH] (port 22) profile is fine; UFW supports multiple profiles in one file and each is referenced by its own bracketed name.

      bash

      Append at the bottom:

      text

      Two quick notes:

      • Don't add a second [OpenSSH] with the same name, or UFW will complain about a duplicate app — a uniquely-named block is the correct approach.

      • After saving, re-read the profile list and confirm it appears:

      bash

      Then you can use it in boththe firewall rule and the fail2ban action:

      bash

      If ufw app list doesn't show the new profile after saving, just re-run sudo ufw app update OpenSSH-2010 (or reload UFW) to refresh its parsed application definitions.

      Performance

      Performance

      Mariadb Slow log

      /var/log/mysql/mariadb-slow.log is the standard file path used to store the slow query log in MariaDB and MySQL databases on Linux systems. This log records any SQL queries that exceed a defined execution time threshold, helping database administrators diagnose and troubleshoot performance bottlenecks. [1, 2, 3, 4]
      Here is a quick guide on how to verify, enable, and analyze this log file.


      1. View Current Settings (MySQL Console)
      Log into your MariaDB/MySQL console to see if the slow query log is active and where it is pointing: [1]
      sql

      SHOW VARIABLES LIKE 'slow_query_log%';
SHOW VARIABLES LIKE 'long_query_time';

      Use code with caution.

      • slow_query_log: Should be ON to record queries.
      • slow_query_log_file: Path to the log file (e.g., /var/log/mysql/mariadb-slow.log).
      • long_query_time: The threshold in seconds. Queries taking longer than this value are logged. [1, 2, 3, 4]
      2. Enable it Temporarily (Runtime)
      If the log is disabled, you can turn it on immediately without restarting the database service: [1, 2]
      sql

      SET GLOBAL slow_query_log = 'ON';
SET GLOBAL slow_query_log_file = '/var/log/mysql/mariadb-slow.log';
SET GLOBAL long_query_time = 2; -- Logs queries taking over 2 seconds
SET GLOBAL log_queries_not_using_indexes = 'ON'; -- Optional: logs unindexed queries

      Use code with caution.

      3. Enable it Permanently (Configuration File)
      Runtime changes disappear if the database restarts. To make them permanent, edit your server's configuration file (usually /etc/my.cnf, /etc/mysql/my.cnf, or /etc/mysql/mariadb.conf.d/50-server.cnf): [1, 2, 3]
      ini

      [mysqld]
slow_query_log = 1
slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 2
log_queries_not_using_indexes = 1

      Use code with caution.

      4. Set Linux Permissions
      Because the mysql system user must write to this file, the directory and file permissions must be configured correctly in Linux: [1, 2]
      bash

      sudo mkdir -p /var/log/mysql
sudo touch /var/log/mysql/mariadb-slow.log
sudo chown -R mysql:mysql /var/log/mysql
sudo chmod 750 /var/log/mysql

      Use code with caution.

      After updating the configuration file and permissions, restart the service: [1, 2]
      bash

      sudo systemctl restart mariadb
# Or on some systems: sudo systemctl restart mysql

      Use code with caution.

      5. How to Analyze the Log
      Slow query logs can get incredibly large and difficult to read manually. Instead of using cat or vi, use native parsing tools: [1, 2]
      • Using mysqldumpslow: A built-in command-line tool that groups similar queries together and sorts them by execution time.
        bash

        mysqldumpslow -s t /var/log/mysql/mariadb-slow.log

        Use code with caution.

      • Using pt-query-digest: Part of the advanced Percona Toolkit, this creates highly detailed and easily readable performance summaries.
        bash

        pt-query-digest /var/log/mysql/mariadb-slow.log

        Use code with caution.

        [1, 2]
      If you are experiencing issues with this log, please let me know:
      • Is your database currently experiencing slow downs or high CPU usage?
      • Do you need help optimizing a specific slow query found in this log?
      • Are you getting a permission denied error when MariaDB tries to create the file?
      AI responses may include mistakes. Learn more




      • mariadb.com

        Slow Query Log Overview | Server | MariaDB Documentation
        Slow Query Log Overview. Complete Slow Query Log Overview gu...

      • Plesk

        How to enable the MySQL/MariaDB slow query log and analyze it on ...
        Answer * Connect to your Plesk server via SSH. * Enable the ...

      • cPanel

        How to Enable the Slow Query Log in MySQL® or MariaDB
        To enable the Slow Query Log for MySQL or MariaDB: * Log in ...
      Show all

      what it could be useful?








      The slow query log is one of the most powerful tools available for database optimization. It functions like a traffic camera for your database, catching the exact queries that are slowing down your applications.
      Here is exactly how it is useful:
      1. Finding Performance Bottlenecks
      • Identifies slow pages: If a website page takes 5 seconds to load, this log reveals the exact SQL query causing the delay.
      • Pins down high CPU/Memory: Queries that scan millions of rows without using indexes spike server resources. The log flags them immediately.
      2. Discovering Missing Indexes
      • Spotting unindexed tables: By enabling log_queries_not_using_indexes, the log captures queries performing full table scans.
      • Easy fixes: Adding a single index to a column flagged in this log can often drop a query's execution time from 3 seconds to 3 milliseconds.
      3. Catching Code Problems (Application Bugs)
      • Poorly written ORM queries: Modern web frameworks (like Laravel, Hibernate, or Django) sometimes generate massive, inefficient SQL joins behind the scenes.
      • N+1 query problems: It reveals loops where your application accidentally runs hundreds of tiny, redundant queries instead of one efficient query.
      4. Database Capacity Planning
      • Tracking growth trends: By comparing log sizes and query counts month-over-month, you can see if your database performance is degrading as your user base grows.
      • Postponing costly upgrades: Optimizing the top 3 slow queries in this log can often slash server resource usage by 50%, saving you from paying for a larger, more expensive server.
      To help you get the most out of this, let me know:
      • Is your database currently running slow, or are you just setting this up proactively?
      • What application or framework (e.g., WordPress, Node.js, Python) is connecting to this database?
      • Would you like an example of how to read a raw entry from this log file?
      AI responses may include mistakes. Learn more































      mariadb-slow.log.1 is about 3.0G is it normal ?










      No, a 3.0 GB mariadb-slow.log.1 is not normal for a healthy, optimized production database.
      The .1 extension indicates this is a rotated log file (an older log preserved by Linux logrotate), meaning your database generated 3 GB of slow query data in a single log rotation period—typically just one week or one day.
      A file this large means your database server is wasting massive amounts of CPU, RAM, and disk I/O processing inefficient queries.
      Why is it so large?
      1. Missing Indexes (Most Common): If a query searches a table with 1 million rows without an index, MariaDB must read all 1 million rows from the disk every single time. This is called a "Full Table Scan."
      2. log_queries_not_using_indexes is ON: If this setting is enabled, MariaDB logs every single query that doesn't use an index, even if it executes in 0.001 seconds. On a busy site, this will inflate the log file to gigabytes in hours.
      3. long_query_time is set too low: If this is set to 0 or a very low fraction (like 0.01 seconds) on a high-traffic application, almost every single query gets logged.
      4. An Application Loop Bug: Your application code might be trapped in a loop, running the same slow query thousands of times per minute.
      Step-by-Step Action Plan
      1. Safely inspect it without crashing your terminal
      Do not try to open a 3 GB file with cat, nano, or vi, as it can freeze your server's memory. Instead, look at just the first few lines:
      bash

      head -n 50 /var/log/mysql/mariadb-slow.log.1

      Use code with caution.

      2. Check your configuration
      Run this command inside the MariaDB console to check if it's logging non-indexed queries:
      sql

      SHOW VARIABLES LIKE 'log_queries_not_using_indexes';
SHOW VARIABLES LIKE 'long_query_time';

      Use code with caution.

      • If log_queries_not_using_indexes is ON and you have a busy database, turn it off to stop the log explosion:
        sql

        SET GLOBAL log_queries_not_using_indexes = 'OFF';

        Use code with caution.

      3. Summarize the 3 GB file instantly
      Do not read it manually. Use mysqldumpslow to extract the top 5 worst offending queries that are taking up all that space:
      bash

      mysqldumpslow -s c -t 5 /var/log/mysql/mariadb-slow.log.1

      Use code with caution.

      (This sorts the file by count (-s c) and shows the top 5 (-t 5) query patterns causing the bulk of the log size).
      4. Reclaim your disk space safely
      Once you have reviewed or backed up the file, you can empty it to reclaim 3 GB of disk space instantly without restarting MariaDB:
      bash

      sudo truncate -s 0 /var/log/mysql/mariadb-slow.log.1

      Use code with caution.

      If you'd like to fix this, let me know:
      • What is the output of the mysqldumpslow command above? (You can paste a small snippet of the top query pattern).
      • What are your current long_query_time and log_queries_not_using_indexes values?
      • Is your server currently running out of disk space because of this?
      AI responses may include mistakes. Learn more




















































      meson@CiCADA: 17:56:36 /var/log/mysql$ head -n 50 /var/log/mysql/mariadb-slow.log.1 /usr/sbin/mariadbd, Version: 10.11.14-MariaDB-0ubuntu0.24.04.1-log (Ubuntu 24.04). started with: Tcp port: 0 Unix socket: /run/mysqld/mysqld.sock Time Id Command Argument # Time: 260725 22:46:02 # User@Host: cloud_user[cloud_user] @ localhost [] # Thread_id: 162 Schema: cloud QC_hit: No # Query_time: 0.089309 Lock_time: 0.000092 Rows_sent: 1 Rows_examined: 191623 # Rows_affected: 0 Bytes_sent: 68 # Full_scan: Yes Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: No Filesort_on_disk: No Merge_passes: 0 Priority_queue: No # # explain: id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra # explain: 1 SIMPLE oc_filecache ALL NULL NULL NULL NULL 186499 191623.00 100.00 79.09 Using where # use `cloud`; SET timestamp=1784987162; SELECT COUNT(*) FROM `oc_filecache` WHERE `path` LIKE 'appdata_ocnu72fp1mua%'; # Time: 260726 1:46:03 # User@Host: cloud_user[cloud_user] @ localhost [] # Thread_id: 536 Schema: cloud QC_hit: No # Query_time: 0.067386 Lock_time: 0.000065 Rows_sent: 1 Rows_examined: 191623 # Rows_affected: 0 Bytes_sent: 68 # Full_scan: Yes Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: No Filesort_on_disk: No Merge_passes: 0 Priority_queue: No # # explain: id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra # explain: 1 SIMPLE oc_filecache ALL NULL NULL NULL NULL 186499 191623.00 100.00 79.09 Using where # SET timestamp=1784997963; SELECT COUNT(*) FROM `oc_filecache` WHERE `path` LIKE 'appdata_ocnu72fp1mua%'; # Time: 260726 4:51:03 # User@Host: cloud_user[cloud_user] @ localhost [] # Thread_id: 975 Schema: cloud QC_hit: No # Query_time: 0.079616 Lock_time: 0.000082 Rows_sent: 1 Rows_examined: 191623 # Rows_affected: 0 Bytes_sent: 68 # Full_scan: Yes Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: No Filesort_on_disk: No Merge_passes: 0 Priority_queue: No # # explain: id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra # explain: 1 SIMPLE oc_filecache ALL NULL NULL NULL NULL 186499 191623.00 100.00 79.09 Using where # SET timestamp=1785009063; SELECT COUNT(*) FROM `oc_filecache` WHERE `path` LIKE 'appdata_ocnu72fp1mua%'; # Time: 260726 7:56:02 # User@Host: cloud_user[cloud_user] @ localhost [] # Thread_id: 1316 Schema: cloud QC_hit: No # Query_time: 0.083674 Lock_time: 0.000084 Rows_sent: 1 Rows_examined: 191623 # Rows_affected: 0 Bytes_sent: 68 # Full_scan: Yes Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: No Filesort_on_disk: No Merge_passes: 0 Priority_queue: No MariaDB [(none)]> SHOW VARIABLES LIKE 'log_queries_not_using_indexes'; +-------------------------------+-------+ | Variable_name | Value | +-------------------------------+-------+ | log_queries_not_using_indexes | ON | +-------------------------------+-------+ 1 row in set (0.003 sec) MariaDB [(none)]> SHOW VARIABLES LIKE 'long_query_time'; +-----------------+----------+ | Variable_name | Value | +-----------------+----------+ | long_query_time | 2.000000 | +-----------------+----------+ 1 row in set (0.003 sec)









      This output explains exactly why your log file reached 3 GB.
      Your database belongs to a Nextcloud or OwnCloud instance (indicated by the oc_filecache table). The log is exploding because log_queries_not_using_indexes is turned ON, combined with Nextcloud constantly checking its app data folder.
      The Root Cause
      Look closely at this specific part of your log entry:
      • Query_time: 0.089309: The query is actually very fast, taking only 0.08 seconds (well below your 2.0-second long_query_time threshold).
      • Full_scan: Yes: The query is doing a full table scan, checking all 191,623 rows in the oc_filecache table every single time it runs.
      • Why it's logged: Because log_queries_not_using_indexes = ON, MariaDB is forced to write this fast query to your log file every few hours. On a busy server with multiple users or cron jobs, this fills up gigabytes of text very quickly.
      How to Fix It
      Step 1: Turn off the unindexed query logging (Instant Relief)
      This stops MariaDB from writing fast, unindexed queries to your log, immediately freezing the growth of the file. Run this in your MariaDB console:
      sql

      SET GLOBAL log_queries_not_using_indexes = 'OFF';

      Use code with caution.

      To make this change permanent so it survives a server reboot, open your MariaDB configuration file (e.g., /etc/mysql/mariadb.conf.d/50-server.cnf or /etc/my.cnf) and change or add this line under the [mysqld] section:
      ini

      log_queries_not_using_indexes = 0

      Use code with caution.

      Step 2: Reclaim your 3 GB disk space
      Now that the logging behavior is changed, you can safely wipe out the bloated backup log file to free up your hard drive:
      bash

      sudo truncate -s 0 /var/log/mysql/mariadb-slow.log.1

      Use code with caution.

      Step 3: Address the Nextcloud Index Issue (Optional but Recommended)
      The query SELECT COUNT(*) FROM oc_filecache WHERE path LIKE 'appdata_...%' uses a wildcard (%) at the end of the string. Nextcloud should be using its built-in index on the path column, but sometimes these indexes get corrupted or missing during major Nextcloud upgrades.
      You can tell Nextcloud to check and repair its database indexes automatically. Run this command in your terminal from your Nextcloud installation directory (usually /var/www/nextcloud or /var/www/html):
      bash

      sudo -u www-data php occ db:add-missing-indices

      Use code with caution.

      (Replace www-data with your web server user if you are using a different one like nginx or apache).
      If you run into any issues, let me know:
      • Did turning off log_queries_not_using_indexes successfully stop the log from growing?
      • Do you know the exact directory path where your Nextcloud/OwnCloud instance is installed?
      • Are you running this instance inside a Docker container, or directly on the host system?
      Performance

      Journal log size Opimization

      /var/log/journal is too high


      You can quickly clear space by running sudo journalctl --vacuum-size=100M to limit the size of your system logs. [1]
      Immediate Cleanup
      • Check current size: Run journalctl --disk-usage to see how much space the logs take.
      • Reduce by size: Run sudo journalctl --vacuum-size=500M to shrink the logs down to 500 megabytes (or another size like 1G).
      • Reduce by time: Run sudo journalctl --vacuum-time=2weeks to delete logs older than a specific timeframe. [1, 2, 3]
      Permanent Fix
      • Open config file: Edit /etc/systemd/journald.conf using a text editor with root privileges (like sudo nano /etc/systemd/journald.conf).
      • Set maximum limits: Find and uncomment (remove the # at the start of) lines like SystemMaxUse=500M or SystemMaxFileSize=50M to
      • Linux Mint

        SOLVED: var/log is large and keeps growing
        ... /var/log/lightdm 4.0K /var/log/mintsystem.log 4.0K /var/

      • Debian forums

        [Solved] How to manage the size of /var/log/journal
        My system runs headless and I have only 782 packages install...

      • SiteLint

        How do I clear a big /var/log/journal folder? - SiteLint
        Let's then delete old log files from the /var/log/journal fo...
      Show all

      How big is reasonable size for journal ?








      For most personal computers and standard servers, a reasonable size for the systemd journal is between 500MB and 4GB.
      Recommended Settings By System Type
      • Desktop / Personal PC (500MB – 1GB): Enough for daily troubleshooting without wasting SSD space.
      • Standard Web/App Server (1GB – 2GB): Retains weeks of operational history for debugging.
      • Production / High-Traffic Server (4GB – 10GB): Necessary if you log heavy application traffic locally.
      • Containers / Microservices (50MB – 100MB): Keep it minimal since logs should stream to an external aggregator.
      Factors to Consider
      • Disk Capacity: A safe rule of thumb is to limit your journal to 10% of your total partition size (systemd's default behavior) or caps at 4GB, whichever is smaller.
      • Log Rotation Frequency: If your system generates 500MB of logs a day, a 1GB limit only gives you 2 days of history.
      • External Forwarding: If you send logs to a central server (like Elasticsearch or Datadog), you can safely set the local journal size much lower.
      If you want to fine-tune this, let me know:
      • Is this for a personal PC or a server?
      • What is the total storage capacity of your drive?
      I can help you calculate the exact configuration values to use.
      AI responses may include mistakes. Learn more


      For a personal server with an i5 processor and 8GB of RAM, a reasonable journal size is 1GB to 2GB. This provides months of troubleshooting history without wasting valuable disk space.
      Recommended Configuration
      To set this up permanently, open your configuration file:
      sudo nano /etc/systemd/journald.conf
      Uncomment and update the following lines to these recommended values:
      • SystemMaxUse=2G (Caps total disk space used by logs)
      • SystemMaxFileSize=200M (Splits logs into smaller, easier-to-read files)
      • RuntimeMaxUse=200M (Caps log memory usage to protect your 8GB RAM)
      Save the file and apply the changes instantly by running:
      sudo systemctl restart systemd-journald
      Performance

      Essential logs to watch

      For an Ubuntu server administrator, I recommend monitoring logs in six major areas: kernel/hardware, storage, system services, security, networking, and applications.

      1. System and Kernel Logs — highest priority

      Kernel messages

      journalctl -k
dmesg -T

      Watch for:

      For live monitoring:

      journalctl -kf

      2. Storage and Disk Health ⚠️

      Especially important for servers running databases, Docker, Nextcloud, Immich, etc.

      Filesystem errors

      journalctl -k | grep -iE "error|ext4|xfs|btrfs|i/o"

      Critical messages include:

      EXT4-fs error
Buffer I/O error
blk_update_request
I/O error
device offline
reset

      Disk SMART health

      sudo smartctl -a /dev/sda

      Monitor:

      SMART Attribute Concern
      Reallocated_Sector_Ct Growing
      Current_Pending_Sector > 0
      Offline_Uncorrectable > 0
      UDMA_CRC_Error_Count Increasing
      Temperature High
      Power_On_Hours Aging disks

      Run long tests periodically:

      sudo smartctl -t long /dev/sda

      3. Memory and OOM Events

      OOM killer

      journalctl | grep -iE "out of memory|oom|killed process"

      Live:

      journalctl -f | grep -iE "oom|memory"

      Check memory:

      free -h

      Swap:

      swapon --show

      For servers, unexpected OOM events are high priority because they may silently kill:

      4. Failed System Services

      One of the most useful administrator commands:

      systemctl --failed

      Check system errors:

      journalctl -p err..alert

      Recent boot errors:

      journalctl -b -p warning

      Useful services to watch:

      systemctl status nginx
systemctl status php8.3-fpm
systemctl status mariadb
systemctl status redis-server
systemctl status docker

      Depending on your PHP version:

      systemctl list-units --type=service

      5. Nginx Web Server

      For web servers:

      tail -f /var/log/nginx/error.log

      Access log:

      tail -f /var/log/nginx/access.log

      Important errors:

      502 Bad Gateway
504 Gateway Timeout
connect() failed
upstream timed out
permission denied
too many open files

      For error frequency:

      grep -i "error" /var/log/nginx/error.log

      6. PHP-FPM

      Check:

      journalctl -u php8.3-fpm

      Live:

      journalctl -fu php8.3-fpm

      Watch for:

      A particularly important message:

      server reached pm.max_children

      This usually means PHP-FPM is saturated.

      7. MariaDB / MySQL Database

      Check logs:

      journalctl -u mariadb

      Live:

      journalctl -fu mariadb

      Watch for:

      Useful checks:

      mysqladmin status

      Connections:

      mysqladmin processlist

      8. Docker

      For your type of server setup, Docker should be monitored continuously.

      Docker daemon

      journalctl -u docker

      Live:

      journalctl -fu docker

      Container status

      docker ps

      Stopped containers:

      docker ps -a

      Container logs:

      docker logs CONTAINER_NAME

      Live:

      docker logs -f CONTAINER_NAME

      Check Docker disk usage:

      docker system df

      This is important because Docker logs and images can unexpectedly fill the filesystem.

      9. Disk Space 🚨

      Run:

      df -h

      Also check inode exhaustion:

      df -ih

      Administrators should generally investigate:

      Usage Status
      < 70% Normal
      70–85% Monitor
      85–95% Warning
      >95% Critical

      Find large directories:

      sudo du -xh / --max-depth=1 2>/dev/null | sort -h

      For logs:

      journalctl --disk-usage

      Reduce old journals if necessary:

      sudo journalctl --vacuum-time=14d

      10. Network Problems

      Check networking services:

      journalctl -u NetworkManager

      For Ubuntu Server using systemd-networkd:

      journalctl -u systemd-networkd

      Current network connections:

      ss -tulpn

      Network errors:

      ip -s link

      Watch for:

      11. SSH and Security 🔐

      Ubuntu authentication log:

      sudo tail -f /var/log/auth.log

      Failed SSH attempts:

      grep "Failed password" /var/log/auth.log

      Successful logins:

      grep "Accepted" /var/log/auth.log

      Current users:

      who

      Recent logins:

      last

      Failed login database:

      lastb

      12. System Reboots and Unexpected Shutdowns

      Check reboot history:

      last reboot

      Previous boot:

      journalctl -b -1

      Check for unexpected shutdown:

      journalctl | grep -iE "shutdown|reboot|panic|crash"

      Kernel panic:

      journalctl -k | grep -i panic

      13. Cron and Scheduled Jobs

      Depending on Ubuntu configuration:

      journalctl -u cron

      Or:

      grep CRON /var/log/syslog

      Important for:

      14. Nextcloud-specific Monitoring

      For your Nextcloud servers, I would specifically watch:

      sudo -u www-data php /var/www/nextcloud/occ status

      Cron:

      sudo -u www-data php /var/www/nextcloud/occ background:cron

      Logs:

      tail -f /path/to/nextcloud/data/nextcloud.log

      Watch for:

      ⭐ My Recommended Administrator Dashboard

      For a self-hosted Ubuntu server, these are the most important things to continuously monitor:

      ┌───────────────────────────────────────┐
│ Ubuntu Server Monitoring              │
├───────────────────────────────────────┤
│ 🔴 Disk I/O / Filesystem errors       │
│ 🔴 SMART disk health                  │
│ 🔴 Disk space / inode usage           │
│ 🔴 OOM / memory exhaustion            │
│ 🔴 Failed systemd services            │
│ 🔴 Docker container failures          │
│ 🟠 Nginx 502/504 errors               │
│ 🟠 PHP-FPM saturation                 │
│ 🟠 MariaDB errors                     │
│ 🟠 Network interface errors           │
│ 🟠 SSH/security attacks               │
│ 🟡 Backup failures                    │
│ 🟡 SSL certificate expiration         │
│ 🟡 Temperature / hardware issues      │
└───────────────────────────────────────┘

      # Critical system errors
journalctl -p err..alert --since today

# Failed services
systemctl --failed

# Disk usage
df -h

# Memory
free -h

# Docker containers
docker ps -a

# Kernel/hardware errors
journalctl -k -p err..alert

# Recent authentication failures
grep "Failed password" /var/log/auth.log | tail

      For your Ubuntu servers specifically

      Given that you operate multiple Ubuntu servers with Nextcloud, Nginx, PHP-FPM, MariaDB, Redis, Docker, Immich, and other self-hosted services, I would prioritize:

      1. Disk I/O errors → 2. SMART health → 3. OOM events → 4. disk capacity → 5. failed services → 6. Docker container health → 7. Nginx/PHP/MariaDB errors.