We help IT Professionals succeed at work.

Cannot re-start RedHat 7 on Azure,

Y I
Y I asked
on
I added a disk to RedHat 7 on Azure. After re-boot server, I cannot ssh to the server .
The following is the steps I did referring https://docs.microsoft.com/ja-jp/azure/virtual-machines/linux/attach-disk-portal .
Could you help me to resolve this issue?

1. I created a disk on Azure Portal.

2. Confirmed file types of existing disks.
[servername]$ df -T
Filesystem     Type     1K-blocks     Used Available Use% Mounted on
devtmpfs       devtmpfs    970560        0    970560   0% /dev
tmpfs          tmpfs       980972        4    980968   1% /dev/shm
tmpfs          tmpfs       980972    37856    943116   4% /run
tmpfs          tmpfs       980972        0    980972   0% /sys/fs/cgroup
/dev/sda2      xfs       30929148 30877300     51848 100% /
/dev/sda1      xfs         508580   116672    391908  23% /boot
/dev/sdb1      ext4       4061824    16380   3819400   1% /mnt/resource
tmpfs          tmpfs       196196        0    196196   0% /run/user/1008
tmpfs          tmpfs       196196        0    196196   0% /run/user/995

3. Confirmd a disk added by dmeg. sdc was added.
[servername]$ dmesg | grep SCSI
[    0.374123] SCSI subsystem initialized
[    1.075264] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)
[    3.073908] sd 3:0:1:0: [sdb] Attached SCSI disk
[    3.078645] sd 2:0:0:0: [sda] Attached SCSI disk
[12159.133527] sd 5:0:0:0: [sdc] Attached SCSI disk

4. Executed fdisk
[servername]$ sudo fdisk /dev/sdc
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x215d20ca.

The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-1073741823, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-1073741823, default 1073741823):
Using default value 1073741823
Partition 1 of type Linux and of size 512 GiB is set

Command (m for help): p

Disk /dev/sdc: 549.8 GB, 549755813888 bytes, 1073741824 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0x215d20ca

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048  1073741823   536869888   83  Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

5. Created a directory and sdc1 is mounted on directory
sudo mkdir /datadrive
sudo mount /dev/sdc1 /datadrive

6. Confirmed UUID by blkid
[servername]$ sudo -i blkid
/dev/sdb1: UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" TYPE="ext4"
/dev/sda1: UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" TYPE="xfs"
/dev/sda2: UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" TYPE="xfs"
/dev/sdc1: UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" TYPE="xfs"

7. Added sdc1 UUID to /fstab and rebooted
[servername]$ sudo vi /etc/fstab
Comment
Watch Question

Fractional CTO
CERTIFIED EXPERT
Distinguished Expert 2019
Commented:
Note: The most common problem with a failed reboot after disk work, is the disk fails to come up + blocks the reboot process.

Fix: Fix may require rebooting server in single user mode, then reviewing boot logs.

First let's take a look at some data...

Comment about this...

1) Mention whether ping works to the server.

2) Post output of...

nmap -p 22 -Pn $your-server-ip

Open in new window


3) Mention how you're running the commands you posted above. Maybe you ran these before you rebooted.
Y I

Author

Commented:
Thank you for your comment.  Regarding 1), I pinged to the server by using psping.exe but ping couldn't reach the server. Before executing the command, I could access to the server by using SSH. Those commands were run accessing the server by putty. But, after attached a disk, I cannot log in to the server ...  If there are any way to boot Linux server from Azure Portal, I will try.
Y I

Author

Commented:
I checked boot diagnostic. I found the following errors.

[1;33mDEPEND [0m] Dependency failed for /datadrive.
[1;33mDEPEND [0m] Dependency failed for Local File Systems.
[1;33mDEPEND [0m] Dependency failed for Migrate local... structure to the new structure.

According to the below article, it looks it's fstab errors.
https://docs.microsoft.com/ja-jp/azure/virtual-machines/troubleshooting/linux-virtual-machine-cannot-start-fstab-errors

It's appreciate if I can know how to use Azure serial console.  I read the below article but I'm still not sure how to connect...
https://docs.microsoft.com/ja-jp/azure/virtual-machines/troubleshooting/serial-console-linux
CERTIFIED EXPERT
Distinguished Expert 2019
Commented:
You need to go through the azure console access path to get the thing booted in single user mode, where you can exclude the mounting of the /datadrive partition.
Not sure why you did not simply mount based on the device /dev/sdc1 /data drive....

Once you get it

/dev/sdc1 does not seem to have a filesystem, I do not see where you initialized it  to xfs which according to fdisk is how you defined the partition, but you never run to initialize the xfs filesystem of /dev/sdc1
CERTIFIED EXPERT
Distinguished Expert 2019
Commented:
CERTIFIED EXPERT
Distinguished Expert 2019
Commented:
Y I

Author

Commented:
Hi arnold, thank you for the information and comment. I should mount the drive carefully... I understand what is happening.., Thank you,
I'm now trying to log in to the server as emergency mode. The console asks root password as the attached image. However I don't know the root password of this Azure VM. In this case, are there any way to login the VM with single user mode and edit fstab?
serial_console.JPG
CERTIFIED EXPERT
Distinguished Expert 2019
Commented:
During initial boot you have to edit the grub boot directive singleuser on the line


Another option is the standard password use ISO to boot from, then mount the drive and edit /etc/shadow, /etc/passwd make sure to have an encrypted version of the password you want.
Y I

Author

Commented:
Apologies for late reply. Thank you very much for the information. Throughout this issue, I learned the importance of taking regular backup, setting root password and enabling serial console. It was good experience to rescue Linux server from Azure.  I tried two approaches. The first is to connect as single user mode and edited grub. The other is duplicated the miss-configured disk and attached to newly created rescue VM to edit fstab. Thank you very much your patient advise and help!

Explore More ContentExplore courses, solutions, and other research materials related to this topic.